Augmented reality (AR) is an emerging technology with many applications in education. Web-based augmented reality (WebAR) provides a cross-platform approach to deliver immersive learning experiences on mobile devices. Integrating machine learning models into WebAR applications can enable advanced interactive efects by responding to user actions. However, little research exists on efective methodologies to teach students WebAR development with integrated machine learning. This paper proposes a methodology with three main steps: (1) Integrating standard TensorFlow.js models like handpose into WebAR scenes for gestures and interactions; (2) Developing custom image classification models with Teachable Machine and exporting to TensorFlow.js; (3) Modifying WebAR applications to load and use exported custom models, displaying model outputs as augmented reality content. The methodology is designed to incrementally introduce machine learning integration, build an understanding of model training and usage, and spark ideas for using machine learning to augment educational content. The methodology provides a starting point for further research into pedagogical frameworks, assessments, and empirical studies on teaching WebAR development with embedded intelligence.
Web-based Augmented Reality (WebAR) is one of the most common ways to combine the real and the
virtual on mobile Internet devices [
Currently, the world’s most famous non-profit library for WebAR development is AR.js [
AR.js and MindAR are built on the classic ARToolKit and OpenCV engines, respectively, which are
currently the industry standard. At the same time, while AR.js is focused on processing primarily simple
markers up to 16 × 16, MindAR is focused on natural images of complex structures. Another feature
of MindAR that makes it an appropriate learning tool is the inclusion of the well-known TensorFlow
machine learning library [
The aim of the study is to develop the methodology for teaching the development of augmented reality for the Web with integrated machine learning models.
The main objectives of the study are as follows: 1. Perform a bibliometric analysis of sources from educational applications of WebAR. 2. Choose tools for developing augmented reality for the Web. 3. Develop and test a methodology for developing WebAR applications for face tracking. 4. Develop and test a methodology for integrating machine learning models into WebAR applications.
To perform a systematic bibliometric analysis for the queries “WebAR” and “Web-based augmented
reality for education”, VOSviewer version 1.6.18 [
As a data source for the first query, Crossref was selected with a search by document titles, which made it possible to select 19 documents from 2017-2022 (date of request: 26.11.2022). The selected documents were analysed by the times they were co-cited with other documents.
Out of 92 sources cited in 19 documents, 26 are cited together more than once, forming only 1 cluster
(figure 1), which includes works [
Scopus was chosen as the data source for the second query, with a search by titles, abstracts and
keywords, which made it possible to select 93 documents from 2001-2023 (figure 2), 66 of which are from
the last five years. The majority of them are journal articles (58 [
Out of 301 authors of 93 documents, 27 were cited twice or more times and 9 were cited three or more times. figure 3 shows the semantic network of keywords in the documents for the query “Web-based augmented reality for education”. The distribution of keywords by clusters (figure 4) is shown in table 1.
The first cluster (highlighted in red in figure 4 and table 1) connects the basic concepts of augmented reality in education: augmented and virtual reality with education (including medical education) and human learning, including the use of smartphones.
Augmented reality is a systemic element – it connects all the clusters and is itself connected to all other concepts.
In the analysed documents, virtual reality is not linked to traditional, mobile, and Internet/web-based learning. It is essential to distinguish virtual reality from virtual learning environments, which include these concepts.
The concept of education is also almost universal – it is not only associated with user interfaces and AR applications.
The links of medical education with other clusters are quite revealing: in the second cluster – with the concepts of curricula, computer-aided instruction, education computing, e-learning and students; in the third – with websites and pedagogical augmented reality technology, in the fourth – with distance education.
Learning (in the sense of studying) is related in the second cluster to teaching, students, education computing, computer-aided instruction, e-learning and user interfaces, and in the third cluster to websites and motivation. This concept has no direct links to distance education.
The concept of a human(s) (person(s)) outside their cluster is linked to students and e-learning in the second cluster and websites in the third.
Outside of its cluster, Internet/web-based learning is only associated with traditional teaching in the second cluster.
Finally, smartphones are linked in the second cluster to teaching, students, education computing, e-learning and engineering education, and in the third cluster to websites and augmented reality applications.
The second cluster (highlighted in green in figure 4 and table 1) connects the concepts of learning environment design: teaching, engineering education, computer-aided instruction, e-learning, students, mobile learning, learning environments, education computing, and curricula.
Central to the second cluster are the concepts of “e-learning” and “students”, which are also almost universal – formally, they are not associated only with Internet/web-based learning due to their synonymity with e-learning.
Computer-aided instruction is related to the concepts of the first (augmented and virtual reality, education (including medical) and learning) and third (motivation, websites, learning systems, interactive learning environments, augmented reality applications, augmented reality technology) clusters. 15 year 4 16 2 17 6 18
The concept of teaching is linked in the first cluster to augmented reality, education and learning, smartphones and Internet/web-based learning, and in the third cluster to websites, augmented reality applications and augmented reality technology.
Engineering education is related in the first cluster to augmented and virtual reality, education and smartphones, and all concepts of the third and fourth clusters.
Education computing is related in the first cluster to augmented and virtual reality, education (including medical) and learning, smartphones, and in the third cluster to motivation, learning systems and websites, and the fourth – with distance education.
Outside their cluster, learning environments are only related to augmented and virtual reality education from the first cluster and websites from the third.
Similarly, mobile learning is related to education and augmented reality in the first cluster and motivation, websites and learning systems in the third.
User interfaces have links to the concepts of the first (learning, augmented and virtual reality) and third (motivation, websites) clusters.
The Curricula are related to education (including medical education), augmented and virtual reality in the first cluster, websites in the third cluster, and distance education in the fourth cluster.
The third cluster (highlighted in blue in figure 4 and table 1) connects the concepts of immersive learning environment implementation: websites, motivation, learning systems, interactive learning environments, augmented reality applications and augmented reality technology.
Central to the third cluster are websites, which are almost universal concepts – formally, they are not associated only with Internet/web-based learning due to the overlap of the relevant concepts.
The concept of motivation is related in the first cluster to augmented and virtual reality, education and learning, and in the third cluster to e-learning and mobile learning, education computing, user interfaces, computer-aided instruction, students and engineering education.
Learning systems are related in the first cluster to augmented and virtual reality and education and in the third cluster to e-learning and mobile learning, education computing, computer-aided instruction, student-centred teaching and engineering education.
Interactive learning environments also have similar links: in the first cluster, with augmented and virtual reality and education, and in the third cluster, with e-learning, computer-aided instruction, students and engineering education.
Naturally, augmented reality applications are related to augmented reality and smartphones in the ifrst cluster and to e-learning, computer-aided instruction, teaching, students and engineering education in the second.
Augmented reality technology are related in the first cluster to augmented and virtual reality and education (including medical education) and in the second cluster to e-learning, computer-aided instruction, teaching, students and engineering education.
The fourth cluster (highlighted in yellow in figure 4 and table 1) contains the concept of distance education, which is linked in the first cluster to the concepts of augmented and virtual reality and the concept of education (including medical education), in the second cluster to the concepts of student, engineering education, education computing, e-learning and curricula, and in the third cluster to the concept of website.
The analysis of the distribution of concepts by the density of links (figure 5) and time makes it possible to determine that the oldest (before 2015) studies focused on user interfaces and their application in education. In 2016, the focus shifted to studying the impact of teaching in learning environments on students. In 2017, the research actualised the concepts of virtual reality, interactive learning environments, curricula, and computer-aided instruction in engineering education. The focus of research in 2018 was on education computing, the use of smartphones, augmented reality applications and pedagogical augmented reality technology.
WebAR is the focus of research in 2019, with studies addressing the use of smartphones, online/webbased learning and augmented reality. In 2020, the impact of the COVID-19 pandemic added to the issues of learning motivation and medical education. A new element of recent research is human augmentation.
The main development tools to develop in HTML and JavaScript are a simple text editor and a web browser, where you can open a regular HTML web page saved locally.
However, this may not work for applications that require a camera. In addition, you may want to test applications on your own mobile devices from time to time, so it is best to install a local web server like Simple Web Server.
It may be helpful to select the HTTPS protocol in the advanced settings – without it, the mobile device may not be able to access the camera.
Doing all the development and testing work directly on a desktop browser is possible, but sometimes, it is still worth trying on a mobile phone.
If the devices are connected to the same local area network that does not have a firewall, there is no problem accessing the web server. However, if the network access point is behind a firewall, you can use ngrok to redirect trafic from the restricted port.
After installing ngrok and creating an account on the website [
Once started, ngrok provides a global HTTPS Internet link – but only while the local web server and the ngrok redirect are running.
Traditionally, debugging web applications involves viewing the web browser console, which displays notifications related to debugging the application.
However, it may be challenging on a mobile device. Here, RemoteJS [
After that, all debug messages will be sent to the web page at https://remotejs.com/viewer/agent_code, where agent_code is the value of the data-consolejs-channel variable.
WebGL [
Three.js [
To understand how Three.js works at a high level, it is useful to draw an analogy with the work of a photographer or film director who: 1) customises the scene by placing objects on it; 2) moves the camera to capture footage from diferent positions and angles.
Three.js is not a specialised library for augmented reality – it contains much more functionality, including that which is more suitable for web VR (lighting, cameras, etc.) (figure 6).
As shown in figure 6, the basis is a scene where objects are created in three steps: 1) determination of object geometry – position vectors, colours, etc.: e.g., BoxGeometry is responsible for the rectangular parallelepiped; 2) definition of the material – the way the object is rendered (its optical properties – colour, texture, gloss, etc.): for example, MeshBasicMaterial corresponds to a material that has its colour and does not reflect rays; 3) geometry and material composition is performed using Mesh.
The renderer will display the 3D model on the canvas, considering the material, texture and lighting. For WebAR applications to work, the scene needs to be transparent so that the video stream from the camera can be overlaid. This is achieved by setting the alpha parameter to true in the WebGLRenderer class constructor.
Rendering itself is performed by the render method, which displays the projection of the scene onto the canvas (canvas element) from the camera’s point of view.
Connect the video stream before linking a canvas to an HTML page for WebAR applications. ifgure 7 shows the first implementation of WebAR, in which a real object from the camera is supplemented with a virtual object.
Placing a canvas over the video is the basis of WebAR. The only thing that needs to be added is displaying the object in a more appropriate location and updating its position according to the camera signal, i.e. object tracking.
You can change the position of the image by moving the virtual camera, changing its position (coordinates), and tilting it. Appropriate changes require tracking objects, so it is expected to classify augmented reality into marker-based, markerless, location-based, etc.
HiuKim Yuen ofers a classification of augmented reality by the type of tracking.
The first type is image tracking: in this type, virtual objects appear on top of target images, which can be barcode-like, which have a predefined structure, and natural, which can be anything.
The images do not have to be printed or on-screen – there can even be augmented reality T-shirts
[
The second type of augmented reality is face tracking, where objects are attached to the human face. Examples include Instagram filters, Google Meet, social media campaigns, apps for trying on virtual accessories, etc.
The third type of augmented reality is world tracking, also known as markerless augmented reality. With this type of tracking, augmented reality objects can be placed anywhere, not limited to a specific image, face, or physical object.
World tracking applications continuously capture and track the environment and estimate the physical position of the application user. Augmented reality objects are often attached to a specific surface, such as the ground.
Location-based augmented reality, known for Pokémon GO, Ingress etc., involves linking content to a specific geographical location – latitude and longitude. Usually, these apps track the environment, as the augmented content is usually attached to the ground, and the location-based part is rather an additional condition that triggers the tracking of the environment (or a face) in a specific location.
Other types of tracking can be defined, such as 3D object tracking, hand tracking, etc.
Despite the variety of libraries for augmented reality, their main task is to determine the position of the virtual camera following the tracked object, as illustrated by the following pseudocode: const ar_engine = new SOME_AR_ENGINE(); while(true) { await nextVideoFrameReady(); const {position, rotation} = ar_engine.computeCameraPose(video); camera.position = position; camera.rotation = rotation; }
First, you need to initiate a library – a specific AR engine – and get a link to it. Then, in a continuous loop, wait for a frame from the video stream of the real camera, determine its position (tilt coordinates), and move the virtual camera on the canvas to the same position.
Often, however, it is not the virtual camera that moves but the objects on the scene. In this case, the position of the tracked object is determined, rather than the real camera, and then the virtual reality object is moved to the same position as the tracked object: const ar_engine = new SOME_AR_ENGINE(); while(true) { await nextVideoFrameReady(); const {position, rotation} = ar_engine.computeObjectPose(video); some_object.position = position; some_object.rotation = rotation;
The tracked image can be of any origin, but it must be prepared: if it contains unnecessary elements, they should be removed.
To recognise an image using the MindAR library, you need to select landmarks on the image – the elements that will be used for recognition. This can be done using the image compiler available at https: //hiukim.github.io/mind-ar-js-doc/tools/compile. Compiling results in the binary file targets.mind, which describes the reference points to be tracked.
Other libraries have similar means of obtaining image descriptions, often called NFT (natural feature tracking) marker compilers. Such an image should be visually complex and have a high resolution (details matter here). A visually complex image provides the software many opportunities to track unique and easily recognisable parts of the image.
The physical size of the NFT marker also afects the quality of its recognition: small images should be approached by the mobile device, while large ones should be kept away from it.
The recognition quality also depends on the brightness of the mobile device’s screen; low-resolution cameras usually work better when they are close to the markers.
The Three.js library is a part of MindAR, which significantly simplifies their interaction: the MindARThree class constructor creates the objects necessary for working with Three.js – renderer, scene, and camera, which are available as renderer, scene and camera fields, respectively.
The anchor objects returned by the call to the addAnchor method, whose parameter corresponds to the number of the image to be recognised, are used to track target images and provide the position where the object should be placed.
Instead of adding Three.js objects directly to the scene, they are added to an anchor component – a group object of the THREE.Groupclass that defines a set of related objects whose position, orientation, and visibility can be controlled together. This anchor group is managed by the MindAR library, which will continuously update the group’s position and orientation in accordance with our tracking set.
The start method of the MindARThree class sets up the parameters, turns on the camera, and loads all the necessary data into the web browser’s memory.
For the renderer, camera, and scene to work, you must create a function to render them. In the unnamed callback function created by the setAnimationLoop function, for each frame, the render method is called from the renderer object, whose parameters are the scene and camera objects – this is the animation on the canvas.
The result is a fully functional WebAR application that tracks a single image (figure 8).
The MindAR library has two main sets of modules – for working with images (image) and for working with faces (face).
The similarities between the image-tracking and face-tracking APIs are visible in the MindAR code. Despite the similarities, the addAnchor method interprets the parameter diferently. For image tracking, it was the number of the target image; for face recognition, it is the number of the face reference point.
Face landmark detection is based on the well-known TensorFlow library model [
A face mesh is another type of augmented reality that overlaps images (textures) on all the reference points of a person’s face rather than being linked to individual points. Face meshes are used to create various makeup efects, tattoos, etc. – up to full face virtualisation.
The face mesh is not a predefined 3D model – it is dynamically generated with constant geometry updates.
To apply the mesh to the face, we need a suitable texture.
The mesh is created by calling addFaceMesh. The addFaceMesh method is similar in form to addAnchor, but they are diferent: addAnchor creates an empty group to which objects whose position is controlled by MindAR are added, while the faceMesh returned by addFaceMesh is a single displayed object whose geometry changes in each frame.
The material of the face mesh can be any texture – if you do not set it, the face mesh will look like the one shown in the first image (figure 10).
You can see the structure of this mesh in the second image (figure 10) – to do this, set the wireframe attribute of the image material.
The third and fourth images (figure 10) are examples of the modified texture of facial landmarks. In
the documentation for Meta Spark Studio [
Creating a beautiful mesh requires specific artistic skills, but using the canonical texture (figure 9) is quite simple – apply the desired image over it and remove unnecessary lines.
For machine learning on the Internet, TensorFlow [
TensorFlow.js [
TensorFlow.js has many pre-trained models that can be used immediately [
TensorFlow.js is part of the MindAR library. However, models are not part of Tensorflow.js, so they
need to be connected separately – as shown in the example of the handpose.js model described in
[
The handpose model is loaded from the TensorFlow Hub (since 2023, a part of Kaggle) [
The handpose model processes individual frames taken from the video stream. This is a rather computationally intensive procedure, so, given that, as long as high accuracy of hand identification is not required, you can try to detect them not in every frame. The detect function creates a separate animation loop, in which for every tenth frame, the estimateHands method of the loaded model is called, and a video frame is passed to it. The method returns a predictions containing information about the hand images detected in the frame, so a non-zero array size is a sign that there was a hand in the frame: const video = mindarThree.video; let frameCount = 1; const detect = async () => { if (frameCount % 10 == 0) {
const predictions = await model.estimateHands(video); if (predictions.length > 0) { //...
} frameCount++; window.requestAnimationFrame(detect); }
ifgure 11 shows an example of setting the position of the plane in the detected image so that it reflects the position of the bounding box of the hand in the frames – the efect is quite simple, but it provides an idea of how to use machine learning models in AR applications.
To quickly create and train your model, you can use the Teachable Machine [
To use the Teachable Machine, students are asked to create a new Google account or use an existing one, and then they can choose the type of model they want to create. There are three types of models available: • Image recognition model allows you to identify objects in photos; • Sound recognition model allows you to recognise audio recordings; • Pose recognition model allows you to recognise body movements.
After selecting the model type, you need to provide data for training it through photos, audio recordings, or videos. Once the data is provided, the Teachable Machine will start training the model, which may take some time, depending on the size and complexity of the training. Once the model is trained, it is advisable to test it to ensure it correctly recognises the data. If the model is inaccurate enough, you can provide additional data to improve it. Once the model has been successfully trained and validated, it can be exported to other projects.
With Teachable Machine’s wide range of features, we can recognise sounds, poses, faces, or any image. Nevertheless, to start using it, you must prepare photos and audio recordings for further experiments, train the selected model, and apply it directly to the web environment.
Clicking on the Get Started button on the home page will take you to a new window where you can use a project template or create your own.
When creating your project, choose which model you want to use. We choose Image Project and click on Standard image model. As a source of images, we suggest that students use their webcams and take a series of headshots from diferent angles (tilt and rotation angles), which we save in a pre-prepared catalogue. We will take several diferent images from each participant in the experiment and divide them into classes, noting the corresponding names (figure 12).
For each image class, there is a probability that a particular image belongs to that class. Students can configure additional training parameters, such as the number of iterations and the model’s learning speed.
Next, we move on to training the model – at this stage, all images are converted to the corresponding numerical tensors. The last step is to experiment by choosing images of diferent people (not just the participants in the experiment) and discussing the recognition results (figure 13).
The libraries included in Teachable Machine are based on TensorFlow models: MobileNet for
image classification [
Accordingly, the built face classification model can be exported and used the same way as the previously used models of facial landmarks and hand pose.
Clicking the Export Model button allows you to export in various formats: • TensorFlow.js – placement of the model at https://teachablemachine.withgoogle.com/models/[...] or downloading the model and the JavaScript and p5.js code (figure 14);
• TensorFlow – download Python code and model in h5 (Keras) and Savedmodel (TensorFlow) formats; • TensorFlow Lite – downloading a model in tflite format for IoT devices based on Android and
Coral.
TensorFlow.js (tfjsVersion), Teachable Machine (tmVersion), libraries from the Teachable Machine (packageVersion) and its name (packageName – in our case, it is @teachablemachine/image), date of creation (timeStamp) and model name (modelName – by default tm-my-image-model), image size (imageSize – all images are resized to the same size) and categories (labels) used for data labelling; • model.json – a text file in JSON format containing information about the neural network architecture (modelTopology); • weights.bin – a binary file containing the weighting coeficients of the neural network.
When exporting models, a test code is ofered to verify them, from which you can learn how to connect the tmImage library and load the model by calling load, the parameters of which are the paths to the model architecture and metadata files – model.json and metadata.json.
After loading the model by calling the getTotalClasses method, you can determine the number of categories that the model will distinguish – in our case, this value, stored in maxPredictions, is three.
Just as before, every tenth frame is passed to the model for analysis by calling predict, which returns an array of two objects containing information about the category (className) and the probability that the image belongs to it (probability) – a string with information about them and is visualised.
From figure 15, we can see that the image on the left is identified correctly despite the change in the background compared to the training set (figure 12), while the image on the right is identified incorrectly.
The completed solution to the problem of developing a methodology for teaching the development of augmented reality for the Web with integrated machine learning models made it possible to draw the following conclusions: (93 documents in 2001-2023) databases made it possible to identify the main concepts of the study, grouped into 4 clusters: a) The first cluster connects the basic concepts of augmented reality in education: augmented and virtual reality with education (including medical education) and human learning, including the use of smartphones; b) The second cluster links the concepts of learning environment design: teaching, engineering education, computer-aided instruction, e-learning, students, mobile learning, learning environments, user interfaces, education computing and curricula; c) The third cluster connects the concepts of immersive learning environment implementation: websites, motivation, learning systems, interactive learning environments, augmented reality applications and augmented reality technology; d) The fourth cluster contains the concept of distance education, linked in the first cluster to the concepts of augmented and virtual reality and the concept of education (including medical education), in the second to the concepts of students, engineering education, education computing, e-learning and curricula, and in the third to the concept of websites.
The analysis of the distribution of concepts by the density of links and time made it possible to date the emergence of diferent concepts and track their development from educational applications of user interfaces to their augmentation. 2. The selected tools for developing augmented reality for the Web form three groups: a) fixed assets: • Simple Web Server provides the full functionality you need without installation needs that meet the requirements of simplicity and mobility; • ngrok trafic redirection allows access to a web server located behind a firewall (on a student’s or teacher’s computer), which creates conditions for working together remotely; • RemoteJS remote debugger allows you to debug JavaScript applications on mobile devices using desktop browsers; b) Three.js graphics library is a high-level implementation of the cross-platform WebGL display standard in JavaScript, which allows working with high-level graphical abstractions; c) MindAR augmented reality library allows working with natural images as augmented reality anchors and includes the Three.js and TensorFlow.js libraries – the latter is key for integrating machine learning models created with TensorFlow with WebAR applications built with MindAR. 3. In the process of developing and testing the methodology for developing WebAR applications for face tracking, the expediency of joint use of the MediaPipe Face Mesh model, a convolutional neural network that identifies 468 three-dimensional landmarks on the face, and the MindAR library, which allows to define any of them as an anchor, is substantiated. It is shown that the complete application of the MediaPipe Face Mesh model in the MindAR library is implemented in the form of a face mesh that is dynamically generated with constant geometry updates – a type of augmented reality associated with the overlay of images on all anchor points of the human face. Examples of using face meshes to create makeup efects, tattoos, etc., are presented. 4. The methodology of integrating machine learning models into WebAR applications involves mastering three main steps: a) The first step, integration of standard models, involves familiarisation with pre-trained TensorFlow.js models that can be used in WebAR applications. The article shows the feasibility of considering the handpose.js model used to determine the hand and its components, demonstrates the main problem of WebAR – a significant performance drop when applying the model to each frame, and suggests a way to solve it. As a result of the first step, a WebAR application for gestural size control is created and the position of the virtual object; b) The second step, custom model development, involves creating and training your TensorFlow models using the Teachable Machine, which allows you to build models to solve problems of image, sound, and pose classification; c) The third step, integration of user models, is performed by exporting the face classification model built with the Teachable Machine and modifying the WebAR application developed in the first step: we load our model, determine the number of categories it will classify, and the object of augmented reality is information about each category and the probability that the webcam image belongs to it. The latter provides an opportunity to discuss the issues of classification errors and their dependence on both the settings of the model training parameters and the way the test images are presented to the WebAR application.
This study does not exhaust all the components of the problem, and further research is needed: • history and prospects of WebAR development in education; • a methodology for the joint use of diferent neural network modelling environments; • development of WebAR libraries, in particular, in the direction of implementing ubiquitous augmented reality; • the relationship between real and virtual in training in the context of a pandemic, natural disaster and military conflict.