=Paper=
{{Paper
|id=Vol-2387/20190274
|storemode=property
|title=Design of Mobile Augmented Reality System for Early Literacy
|pdfUrl=https://ceur-ws.org/Vol-2387/20190274.pdf
|volume=Vol-2387
|authors=Marlen Ablyaev,Afife Abliakimova,Zarema Seidametova
|dblpUrl=https://dblp.org/rec/conf/icteri/AblyaevAS19
}}
==Design of Mobile Augmented Reality System for Early Literacy==
https://ceur-ws.org/Vol-2387/20190274.pdf
Design of Mobile Augmented Reality System
for Early Literacy
Marlen Ablyaev, Afife Abliakimova, Zarema Seidametova
Crimean Engineering-Pedagogical University, Simferopol
{ablyaev.marlen, ablyakimova.afife, z.seydametova}@gmail.com
Abstract. Mobile applications allow connecting the world around us with digital
information. Augmented Reality (AR) applications are commonly used in
museums, education and for research. AR applications can attract users with
contextual and locally specific information. We present a concept, development
of augmented reality (AR) application “Tilsimli arifler” in which we create
literate environment for motivating of children in practicing reading and writing.
In order to enhance the kids reading ability we propose to develop a learning
mobile application that allows using modern mobile technologies, in particular,
using the technology of augmented reality, to study the alphabet and learn how
to make words. In this case, a prerequisite was the use of special (drawn) cards
and a camera of a mobile device. When scanning a card with a camera on a user's
smartphone, a three-dimensional image of the object deposited on the card should
appear and which can be rotated in different directions (“spinning”) in real time.
Keywords: augmented reality (AR), virtual reality (VR), software development
kit (SDK) software development, augmented marks, vuforia, quick response
(QR) code.
1 Introduction
The rapid growth in information and communication technologies and the large-scale
process of informatization society are the leading factors that identify the needs and
necessities for changes in education. Mobile applications allow connecting the world
around us and digital information. Therefore such applications are widely used in
museums, research and educational institutions. Augmented Reality (AR) applications
offer intuitive ways to provide users with contextual and locally specific information.
One of the widely used approaches in the development of augmented reality mobile
applications is the use of visual markers. Visual markers used in AR applications allow
users to “animate” information associated with markers. Applications that use markers
allow user to read information about a specific object or its location using mobile
devices.
The modern period of the society evolution is characterized by a strong influence on
it of computer technologies that penetrate into all spheres of human activity, ensure the
dissemination of information flows in society, forming a global information space. An
integral and important part of these processes is the computerization of education. At
present, a new education system is being formed, oriented towards entering the world
information and educational space. This process is accompanied by significant changes
in the pedagogical theory and practice of the educational process associated with
making adjustments to the content of learning technologies that should be adequate to
modern technical capabilities, and contribute to the harmonious entry of the child into
the information society.
One of the first stages of the child’s entry into the educational environment is the
study of the alphabet and the acquisition of word-constructing skills. Considering that
modern technologies surround us in everyday life, children also periodically encounter
them. But, to make the process of interaction between the child and computer
technologies more productive and expedient, it is necessary to prepare appropriate
software that will allow users to provide educational content in a game form.
Thus, there is a need to create a mobile application for learning the alphabet and
constructing words using special cards, which depict a specific letter and subject,
starting with this letter. To attract the attention of the child uses augmented reality.
In order to enhance the kids reading ability we propose to develop a learning mobile
application that allows using modern mobile technologies, in particular, using the
technology of augmented reality, to study the alphabet and learn how to make words.
In this case, a prerequisite was the use of special (drawn) cards and a camera of a mobile
device. When scanning a card with a camera on a user's smartphone, a three-
dimensional image of the object deposited on the card should appear and which can be
rotated in different directions (“spinning”) in real time.
In the paper [1] authors provided an overview of Augmented Paper Systems use in
education. Studies related to the effect of augmented reality on learning efficiency
presented in papers [2], [3], [4], [5], in which also discussed questions on how to
combine the capabilities of augmented reality and students’ learning. An experience of
the successful use of AR mobile applications in learning and teaching process described
in [6]. In [7] – [14] authors presented the results of their research on the development
and implementation of AR apps for children. In [15] – [20] researchers discussed the
effects of marketing, usability, and design on augmented reality technologies and ways
of the development of AR apps. Comparative analysis of augmented reality frameworks
that are used to develop educational applications presented in [21].
2 Tools for Augmented Reality Development
One of the most common environments for visualizing and interacting of the three-
dimensional graphics as development environment is Unity [22]. Unity is a tool for
developing two- and three-dimensional applications and games. Applications created
in Unity are supported by following operating systems: Windows, OS X, Windows
Phone, Android, Apple iOS, Linux. It is also possible to use the Unity Web Player plug-
in, which enables to launch and use applications in different web browsers.
Applications created using Unity also support DirectX [23] and OpenGL [24], [25].
One of the main advantages of Unity is cross-platform that means the ability to use the
same code with minimal changes to create applications for multiple platforms at once.
It is possible to expand the functionality with the help of Asset Store. There are a large
numbers of different plug-ins and extensions in Asset Store.
Another popular engine is the Unreal Engine [26]. The main difference from Unity
Unreal Engine is to use a different programming language. Unreal Engine uses C ++,
and Unity uses C # and Java. Unreal Engine has a modular system of components that
are dependent on each other, which simplifies the transfer of the project from one
system to another. Supported systems are: Microsoft Windows, Linux, Mac OS and
Mac OS X; Xbox, Xbox 360, PlayStation 2, PlayStation 3, PSP, PS Vita, Wii,
Dreamcast, GameCube, etc., as well as on various portable devices, such as Apple
(iPad, iPhone) devices controlled by iOS [27] and others.
To create augmented reality, developers use systems such as Vuforia [28] and Kudan
[29]. Vuforia is a platform that allows an application installed on a mobile device with
a camera to “see” and recognize various objects, images, texts and markers located on
them. Object recognition is used for the so-called “revitalization” of them in the form
of 3D models on the device screen. For example, with the help of this platform, it is
possible to “revive” some mechanisms, having previously read them with the help of
the special utility Vuforia Object Scanner. Using it, a special file that loads markers
into the manager is created, and then these markers are used to recognize the original
object. The same principle applies when you create markers from the images and other
texts. In particular, there is an opportunity to create a variety of effects, such as video
playback when you move the marker, the use of night vision effects or creating controls
on the markers in the real world, read the camera device. Vuforia is supported by mobile
devices running Android 4.0.3 and iOS 7 and above. It is also supported by augmented
reality glasses, for example, such as Epson Moverio [30].
Kudan is another augmented reality developing system. Among the key features of
this technology are the enhanced tracking capabilities of the markers, such as steeper
viewing angles, accelerated guidance to the marker, long distance to the marker, the
ability to work in difficult lighting conditions. Another feature is the provision of
various methods of augmented reality, such as marker and marker-less. Marker
technology provides reliable tracking of the object, the ability to create custom markers
and their simultaneous tracking. Marker-less technology allows you to track objects
without installing special labels on them. The advantage of this technology is that real-
world objects can serve as markers themselves and the creation of special visual
identifiers for them are not required [29]. Both of these technologies work using the
OpenCV [25] specialized computer vision library. This library is able to work with
virtually any operating system, has C ++, C, Java and Python interfaces. The library
has more than 2500 optimized algorithms, which includes a full range of both classical
and computer vision and machine learning algorithms.
EasyAR [31] is a free and easy-to-use alternative to Vuforia. EasyAR supported
platforms are Android, iOS, UWP, Windows, OS X and Unity. It is known that the
applied development augmented reality technology tools depend on the type of tasks
and the available equipment for implementation. Development tools such as Daqri,
MixAR and ZooBrust are fairly simple and do not require high programming skills.
Other tools that include SDKs, such as ARToolKit, Unifeye Mobile SDK, and
Wikitude, were created for advanced developers [32].
Wikitude is a set of software for developers that allow you to create your own AR-
applications from scratch, and integrate AR-functionality into ready-made applications.
You can see a comparison of most popular tools for the development of augmented
reality applications in the table 1.
Table 1. Comparison of the most popular AR tools
Type Cross-platform Programming Documentatio
languages n / tutorials
Unity Boost Windows, OS X, Windows С#, JavaScript +
Software Phone, Android, Apple iOS,
License Linux
Unreal Boost Microsoft Windows, Linux, C++ +
Engine Software Mac OS и Mac OS X;
License консолей Xbox, Xbox 360,
PlayStation 2, PlayStation 3,
PSP, PS Vita, Wii,
Dreamcast, GameCube
Vuforia Free + iOS, Android, Unity C + +, Java, +
Commercial Objective-C,
SDK option .Net
Kudan Free + Microsoft Windows, Linux, C++, C, Java и +
Commercial Mac OS, Mac OS X; Python
SDK option PlayStation 2, PlayStation 3,
PSP, PS Vita, Wii,
Dreamcast, GameCube
OpenCV Boost iOS, PC, Android, Linux С/C + +, +
Software Python, Java,
Ruby, Matlab,
Lua
Wikitud Free + Android, iOS, smart glasses С#, Java –
e Commercial
SDK option
Because of the accessibility, availability of the necessary functionality, the possibility
of connecting additional sets of development tools and the availability of peripheral
(additional) services the Unity platform and Vuforia SDK were selected for
development of the “Tilsimli arifler” application.
3 Project Design with Vuforia
Vuforia has several recognition methods (data entry) for visual information ([33], [34],
[35], [36]):
1) Markers (Image Targets) - these are real flat images that applied to any favorable
flat surface. Markers can be represented as QR or matrix codes. The marker does not
require special black and white areas in order for it to be tracked in the Vuforia
Engine. It is convenient to use markers anywhere, but in the near future they will
most likely be used for prompts, information references, for creating interactive
books and manuals, visualizing any concepts, and so on.
2) Multi-Targets are analogous to ordinary markers, but with their use, more than
one marker is tracked on one real object, for example, on a cube. This is especially
useful for extra-large physical objects that are overlaid with augmented reality and
need to be captured from all sides, for example: houses, monuments, museums, etc.
3) Cylinder Targets are real flat images applied to cylindrical surfaces. They allow
you to track markers in the form of twisted cones and cylinders. Such markers are
used on cups, bottles, wide footsteps, etc.
4) Virtual buttons work thanks to the obstacle detection technology, which allows
the user to perform any action or function visually resembling a real button when it
overlaps a certain predefined zone.
5) Recognition of visual text (Text Recognition) is recognition of images of a
predetermined list of text characters. The default SDK includes 100,000 Latin
characters of various formats that can be recognized by the application. It is also
possible to create personal character sets (maximum 10,000 additional characters).
6) Recognition of physical objects in 3D (Object Scanner) recognizes 3D objects
and makes of them simplified geometric skeletons-primitives (label-objects). In the
future this allows identifying a specific physical object in the physical space.
To create a new object marker first program scans an existing object (Object
Scanner), after which the Object Data File is automatically generated. The generated
file is uploaded to the Vuforia Target Manager (it is possible to load up to a maximum
of 20 Object Data File).
For the application being developed, targets were used Image Targets, therefore
Single Image objects were loaded into the database.
It should be noted that not every picture is suitable for creating a target. Good targets
are those in which there are many contrasting details. It is on these details that the
reference matrix is built for their subsequent recognition. In Figure 1 in the table of
loaded targets the Rating field shows how much the picture is suitable for recognition.
Fig. 1. Downloaded objects into the database.
In the “Tilsimli arifler” application, targets were created for loading into the database,
the design of which took into account the peculiarities of Vuforia recognition. Samples
of targets are presented in Figure 2.
Fig. 2. Samples of targets for recognition.
You can add Vuforia targets to your scene by selecting the corresponding game objects
in the GameObject–>Vuforia menu.
The target game object will be added to the scene hierarchy and appear on the scene.
Each target object can be configured in the Inspector. The configuration window
indicates the database and the name of the target to be used. Now you can add digital
content to complement the target. To do this, add objects that will complement the
target as child elements of the target in the scene hierarchy (Figure 3).
Fig. 3. Screen view of the object and targets.
Displaying the contents of augmented reality is done by customizing the rendering and
physical behavior.
Vuforia provides a simulator in game mode, which can be activated by pressing the
play button. You can use this feature to evaluate and quickly prototype your scene (s)
without having to deploy it on the device. Playback mode is configured in the Vuforia
WebCam section.
4 Implementation of “Tilsimli Arifler”
The implementation of this application was made in the Unity development
environment, which by its means and capabilities allows processing three-dimensional
objects and images obtained from the camera of a mobile device using the Vuforia
Augmented Reality SDK [30].
Vuforia Augmented Reality SDK allows you to track one or more images with the
camera, and Unity allows you to place a three-dimensional object on the user's screen
in a certain position. Thus, the Vuforia Augmented Reality SDK determines the image
coordinates in which Unity should place the corresponding object. Thus, in the
application there is a process of finding letters and displaying the necessary models.
To enable the user to compose words from letters, a special algorithm was
developed, a fragment of which is given below (Listing 1):
Listing 1. Camera image processing and word recognition algorithm.
public class SetRay : MonoBehaviour {
public int NumberWord = -1;
[SerializeField] private GameObject cameraGO;
private int tmp = 200;
private RaycastHit hit;
private RaycastHit hit1;
private Ray rayRight;
private Ray rayLeft;
private bool firstChar = false;
void Update(){
rayRight = new Ray(transform.position, transform.right);
rayLeft = new Ray(transform.position, -transform.right);
if (Physics.Raycast (rayRight, out hit)) {
if (hit.transform.tag == "Char" && NumberWord !=0) {
if (!Physics.Raycast (rayLeft, out hit1)) {
firstChar = true;
cameraGO.GetComponent ().Word = "";
NumberWord = 0;
cameraGO.GetComponent ().Word = "" + transform.name;
tmp = NumberWord;
}
}
}
if (Physics.Raycast (rayRight, out hit)) {
if (hit.transform.tag == "Char" && SetRay().NumberWord !=
NumberWord + 1) {
int tmpInt = NumberWord + 1;
hit.transform.GetComponent ().NumberWord = tmpInt;
MainWord().Word = MainWord().Word+ hit.transform.name;
}
}
}
}
This algorithm processes the image from the camera and recognizes the word
composed with the help of appropriate cards (Fig. 4).
Fig. 4. Word recognition algorithm.
The Left () function in this algorithm is necessary to determine the first element, that is
the first letter of the word. If there is no other card (letter) to the left of the card (letter),
then this element is the first, therefore it will be first element written into the variable
that defines the word. Then the Right () function is executed defining the next element.
If during the execution of the Right () function it turns out that another card (letter) is
on the right, a new element is added to the variable, thereby forming a word. And if
there is no other element to the right, then this element is the last one, it starts the Search
() function. The Search () function searches a local database of words, comparing the
resulting word with those already in the program. If the constructed word is found in a
local database, a three-dimensional model that corresponds to the received word will
be displayed on the user's screen (Fig. 5).
If the constructed word is not found in the local database, the application will notify
the user about a possibly incorrectly composed word or the absence of the
corresponding model.
To visualize all the letters of the alphabet and the words existing in the local
database, it was necessary to use three-dimensional models that correspond to one or
another word or image on the card. Source objects were obtained from the official Unity
public store.
The resulting objects were imported into Unity with their subsequent configuration.
Setting up each model meant changing the size of the object, which should be
proportional to the size of the card; this allows avoiding displaying an object beyond
the visibility of the camera. The important step in setting up objects was the stage of
texture optimization, which should be clearly visible under different light conditions
and spend as little as possible on the memory of the mobile device. Next, animations
were added to each model for a smoother appearance of the object on the scene and
improvement of the visual perception of the final image on the screen of the mobile
device. In addition, all three-dimensional models were optimized in order to reduce the
size of each object; this reduced the size of the source file of the application and
increased the overall performance.
Fig. 5. A running example of the “Tilsimli arifler” application: recognized word image
appearance.
The application development process is shown in video tutorials and published on
YouTube [37], [38]. The videos describe step by step the entire development process
from creating a project to compiling it. The course is designed for students and novice
developers who want to use the technology of augmented reality (Augmented Reality).
The development of the project “Tilsimli arifler” was used as a demonstration example
in the videos. You can see screenshot of videos playlist in Fig. 6.
Fig. 6. YouTube Playlist of the “Unity AR for beginners” videos created in Applied Informatics
Department of CEPU [ ].
5 Conclusions
Nowadays the augmented reality is one of the most innovative technologies in the
developing mobile applications that create a platform for the emergence of new types,
genres and activities of similar applications. The AR technology opens a new horizon
for the development of mobile applications, including in the educational activities.
The AR project “Tilsimli arifler” developed by us is given the opportunity to study
English and Crimean Tatar alphabets, and learn how to compose words in the
appropriate languages. In this case, the process of learning is accompanied by three-
dimensional visualization and animation of each letter and each word. This contributes
to the quick memorization, retention of the learner's attention, visibility, dynamism and
susceptibility of the new information. As a result, learning becomes exciting and more
productive.
In addition, the experience gained from the development of the “Tilsimli arifler”
application opens up a new look at the development of mobile learning applications. In
the future we plan to introduce a larger number of three-dimensional objects for letters
and words, expand the local database of words, audio explanations to accompany each
word, expand the application in order to add new databases for learning as many
languages as possible.
References
1. Prieto, L.P., Wen, Y., Caballero, D. and Dillenbourg, P.: Review of Augmented Paper
Systems in Education: An Orchestration Perspective. Journal of Educational Technology &
Society 17(4), 169-185 (2014).
2. Lee, K.: Augmented reality in education and training. TechTrends 56(2), 13-21 (2012).
3. Liarokapis, F. and Anderson, E.F.: Using augmented reality as a medium to assist teaching
in higher education. In: Proceedings of the 31st annual conference of the european
association for computer graphics (Eurographics 2010), pp. 9-16. Norrkoping, Sweden
(2010).
4. Bower, M., Howe, C., McCredie, N., Robinson, A. and Grover, D.: Augmented Reality in
education–cases, places and potentials. Educational Media International 51(1), 1-15 (2014).
5. Akçayır, M. and Akçayır, G.: Advantages and challenges associated with augmented reality
for education: A systematic review of the literature. Educational Research Review 20, 1-
11(2017).
6. Santos, M.E.C., Chen, A., Taketomi, T., Yamamoto, G., Miyazaki, J. and Kato, H.:
Augmented reality learning experiences: Survey of prototype design and evaluation. IEEE
Transactions on learning technologies 7(1), 38-56 (2014).
7. Hornecker, E. and Dünser, A.: Supporting Early Literacy with Augmented Books-
Experiences with an Exploratory Study. GI Jahrestagung (1), 555-559 (2007).
8. Bhadra, A., Brown, J., Ke, H., Liu, C., Shin, E.J., Wang, X. and Kobsa, A.: March.
ABC3D—Using an augmented reality mobile game to enhance literacy in early childhood.
In: Pervasive Computing and Communication Workshops (PerCom Workshops) IEEE
International Conference on, pp. 1-4(2016).
9. Alakärppä, I., Jaakkola, E., Väyrynen, J. and Häkkilä, J.: Using nature elements in mobile
AR for education with children. In: Proceedings of the 19th International Conference on
Human-Computer Interaction with Mobile Devices and Services, pp 41 (2017).
10. Barkhaya, N.M.M., Halim, N.D.A. and Yahaya, N.: The Importance of Augmented Reality
Application for Children’s Development During Preschool Years. Advanced Science Letters
24(11), 7935-7938 (2018).
11. ChanLin, L.: Bridging Children’s Reading with an Augmented Reality Story Library. Libri
68(3), 219-229 (2018).
12. Baloch, S., Qadeer, S. and Memon, K.: Augmented Reality, a Tool to Enhance Conceptual
Understanding for Engineering Students. In: International Journal of Electrical Engineering
& Emerging Technology, Vol. 01, No. 01, DEC 2018, pp 41-48 (2018).
13. Huang, Y., Li, H. and Fong, R.: Using Augmented Reality in early art education: a case
study in Hong Kong kindergarten. Early Child Development and Care 186(6), 879-894
(2016).
14. With Help from Augmented Reality, ELL's are Making Strides in Literacy,
https://bit.ly/2TGKYQr, last accessed 2019/02/05.
15. Liao, T.: Augmented or admented reality? The influence of marketing on augmented reality
technologies. Information, Communication & Society 18(3), 310-326 (2015).
16. Santos, M.E.C., Taketomi, T., Sandor, C., Polvi, J., Yamamoto, G. and Kato, H.: A usability
scale for handheld augmented reality. In: Proceedings of the 20th ACM Symposium on
Virtual Reality Software and Technology, pp. 167-176 (2014).
17. Sánchez, A., Redondo, E. and Fonseca, D.: Developing an augmented reality application in
the framework of architecture degree. In: Proceedings of the 2012 ACM workshop on User
experience in e-learning and augmented technologies in education, pp. 37-42. (2012).
18. Santos, M.E.C., Taketomi, T., Yamamoto, G., Rodrigo, M.M.T., Sandor, C. and Kato, H.:
Augmented reality as multimedia: the case for situated vocabulary learning. Research and
Practice in Technology Enhanced Learning 11(1), 1-23 (2016).
19. Chenga, L.K., Selamatb, A., Putehc, F. and Mohamedb, F.: AReview Of recent
methodologies, technologies and usability in English language content delivery. Jurnal
Teknologi 78(12-3), 1-11(2016).
20. Huang, Z., Hui, P., Peylo, C. and Chatzopoulos, D.: Mobile augmented reality survey: a
bottom-up approach. arXiv preprint arXiv:1309.4413, 112-126 (2013).
21. Herpich, F., Guarese, R.L.M. and Tarouco, L.M.R.: A comparative analysis of augmented
reality frameworks aimed at the development of educational applications. Creative
Education 8(09), 1433-1451(2017).
22. nity for Mobile AR, https://unity.com/solutions/mobile-ar, last accessed 2019/02/08.
23. DirectX at Curlie, https://curlie.org/Computers/Programming/Libraries/DirectX/, last
accessed 2019/02/06.
24. OpenGL – The Industry Standard for High Performance Graphics, https://www.opengl.org/,
last accessed 2019/02/06.
25. OpenCV library, https://opencv.org/, last accessed 2019/02/06.
26. Unreal Engine, https://www.unrealengine.com/en-US/what-is-unreal-engine-4, last
accessed 2019/02/06.
27. Xcode - Apple developer, https://developer.apple.com/xcode/, last accessed 2019/02/06.
28. Vuforia. Augmented Reality for the Industrial Enterprise, https://www.vuforia.com, , last
accessed 2019/02/09.
29. Kudan. Artificial Perception technologies, https://www.kudan.eu/, last accessed 2019/02/06.
30. Epson Moverio / Smart Glasses, https://moverio.epson.com/, last accessed 2019/02/06.
31. EasyAR-Best engine for developing Augmented Reality, https://www.easyar.com/, last
accessed 2019/02/06.
32. Amin, D. and Govilkar, S.: Comparative study of augmented reality SDKs. International
Journal on Computational Science & Applications 5(1), 11-26 (2015).
33. Wang, J.T., Shyi, C.N., Hou, T.W. and Fong, C.P.: December. Design and implementation
of augmented reality system collaborating with QR code. In: Computer Symposium (ICS),
2010 International, pp. 414-418. IEEE (2010).
34. Alcantara, M.D., Hounsell, M.S. and Silva, A.G.: Alternative position, orientation and data
recognition algorithms for augmented reality markers. In: International Conference Applied
Computing (https://bit.ly/2DugAli). Rio de Janeiro:[sn](2011).
35. AR tracking with different markers, http://it-jim.com/2016/09/19/augmented-reality-
tracking-with-different-markers/, last accessed 2019/02/08.
36. Theory and applications of marker-based augmented reality,
https://www.vtt.fi/inf/pdf/science/2012/S3.pdf, last accessed 2019/02/08.
37. Seidametova, Z.: Design and Development of MOOCs. CEUR Workshop Proceedings
(CEUR-WS.org). Proceedings of the 14th International Conference on ICT in Education,
Research and Industrial Applications. Integration, Harmonization and Knowledge Transfer.
Volume 2104, issue 2, pp. 462-471 (2018).
38. CP4B. Applied Informatics of CEPU. YouTube Playlist, https://bit.ly/2th4jvP, last accessed
2019/02/11.