Augmented Reality (AR) is a widely used technology in fields such as medicine, engineering, and architecture, and is also prevalent in social media platforms like Snapchat, Instagram, and TikTok. In recent years, the availability of AR applications and improvements in hardware have made it afordable for educational training in various disciplines. However, limited options are available for the general construction of AR tutorials in the literature. Most solutions are specific for particular contexts, such as medical procedures or industry-specific tasks. This paper proposes an AR toolkit that enables novice programmers to create tutorials without topic restrictions. Our aim is to keep improving TutorialKit in such a way that it can be used flexibly and efectively in a variety of diferent contexts, enabling it to meet the diverse needs and requirements of users.
Nowadays, Augmented Reality (AR) interfaces are widespread, finding applications in medicine [
In the literature, there are not many options available to ease the construction of Augmented
Reality tutorials in the general case. Most of the solutions are extremely focused on a particular
domain and are created ad-hoc for a specific context. For example, in the medical field, several
applications explain how to do activities such as sterilisation of tools [
Our project aims to provide an Augmented Reality toolkit that enables novice developers to rapidly create tutorials in diferent domains, without the burden of learning all the required aspects of a full-fledged AR toolkit. The toolkit is flexible and applicable in diferent contexts and domains.
In the past few years, numerous AR-based training tutorials have been presented[
Our work enables novice programmers to create AR tutorials without being restricted to a specific field. The structure of tutorials supported by our toolkit is simple, composed of atomic and elementary steps that can be applied in various contexts. As a result, our toolkit allows novice programmers to create AR tutorials that can be tailored to the needs of individual users.
As mentioned earlier, our work aims to make augmented reality tutorial creation easier for less experienced programmers, by providing tools that can be adapted as much as possible to any topic, without necessarily having to write complex code. To do this, we created a simple AR tutorial abstraction, consisting of two key concepts: the task and the tool.
• A Task is an action that a user must take to finish the tutorial. The aim of the tutorial is to guide the user in achieving a goal by completing all tasks.
• A Tool is an object that the user controls to carry out a task.
TutorialKit allows defining tutorials by a structured encoding of tasks and tools. The underlying toolkit support interprets the information and generates the interface for performing the tutorial. The structured encoding consists of a JSON file. For tasks, it includes the following information: title, description, media’s URL (image or video, optional), and tool name (identifying the tool to be used for the task, optional). Every task has a tool, which is an object that will be used to do the task. A tool is defined as follows: name, image (needed for the tracking), and the dimension of the hint. Tasks may be grouped in unordered and ordered lists, which express their temporal relationships. Ordered lists define tasks that must be performed in sequence (e.g. executing a recipe), whereas when ordering is not mandatory (e.g. decorating a Christmas Tree), we use unordered lists.
We describe the support to the execution of the tutorials defined by this structure in Section 3.1, while the show the user interface in Section 3.2.
The implementation of the toolkit relies on the AR Foundation package4 in Unity 3D5. The solution consists of three modules: • The UIManager is the main class, and it provides represents the entry point for library users and coordinates communications between the other classes; • The ClipboardManager handles information displayed on the clipboard, such as the tutorial’s progress of the user, the success or failure callbacks etc.; • The ToolManager identifies the tool in the real environment when performing a task.
The novice developer imports the UIManager in his/her application to implement a tutorial, providing the aforementioned JSON description of tasks and tools. The implementation also requires callback functions for the correct or wrong performance of a given task. After loading the JSON file, the novice developer links the callbacks to trigger positive or negative feedback on a given task, while the interface is completely generated by the underlying toolkit. The feedback consists of a green check or a red cross on the clipboard, for a positive and negative outcome, respectively.
In future work, we would like to open the development of tutorial to non-programmers, through an authoring interface allowing to specify the structure (i.e., the information in the JSON file) and to provide a low or no-code definition of the completion callbacks when a task has been performed correctly or not. This may be achieved by including a set of predefined triggers and actions (e.g., setting an object position or visibility on collision). 4https://unity.com/unity/features/arfoundation 5https://unity.com/
While the UIManager represents the programming interface for the novice developer, the ClipboardManager and the ToolManager are two modules responsible for updating the AR interface. The clipboard is the core of the interface, and it displays the task information and some optional information, which depends on the current task. The Tool Manager is responsible for highlighting the elements required for completing the task in the AR environment, including the item that needs to be handled to finish the task and any external tools that will help do so. Suppose we have a tutorial for making a cake in which one of the task is to whip the cream. Our toolkit could highlight the cream as a necessary object for completing the task, as well as the whisk, which is considered an external object (since an electric mixer might also be used).
The Clipboard is placed in the virtual space of the scene, and it is depicted as a real clipboard. Users interact with an object they are familiar with in suggesting guidance for completing actions. The information in the clipboard contains the task title, its description, a hint about the tool to be used, warnings in case the task can not be performed and feedback on correct or wrong task performance. In addition, in the clipboard interface shows the progress of the activity based on the previously completed tasks.
In the case where tasks have to be performed in an ordered manner (Figure 2), we display a progress bar whose elements change colour as the tasks are performed, ranging from red to green as the bar gets filled like the colours of a battery when it’s charging. Specifically, the progress bar is red when a few tasks have been completed, green when the tutorial is done, and transitions through intermediate colours based on the number of completed tasks.
This feature provides users with a visual representation of their progress, which can help motivate them to complete the tutorial.
In the case where the task can be executed in random order (Figure 3), we display a series of spheres, where each sphere represents a task. We display uncompleted tasks with a gray colour, which changes to green or red, respectively, on success or failure.
The interface provides buttons for navigating the task list. When there are random order tasks, the contour of the sphere that represents the task currently displayed in the clipboard is highlighted in yellow. In both cases, the percentage value of the overall progress in task completion is displayed.
The clipboard allows displaying optional information as media elements, such as videos and images that can aid in task performance. Such audiovisual material is displayed on the left side of the clipboard on a virtual screen that can be turned on or of by the user and, in the case of a video, allows the user to play and pause the content.
The Tool Manager handles the interface guidance on the tool used for the task accomplishment. Through image recognition techniques, we locate the correct tool for the task displayed in the clipboard. We highlight it in AR by placing a semitransparent green cuboid in the center of it. Our current implementation allows highlighting one tool per task. Such a choice was not made for not for technical reasons, but to force the tutorial designer to divide the tasks into simple tasks so that it is easier for the user to understand and execute them. The highlights for that specific task are turned of as soon as it is completed.
In this section, we explain how the application works through two diferent usage scenarios: assembling a PC (with ordered tasks) and decorating a Christmas Tree (without ordered tasks). These two use cases were chosen because the first use case requires accurate order in assembling the components, whereas in the second it is not strictly necessary.
Suppose novice users bought all the necessary components to assemble their PC at home. They discovered that there is an app that enables them to follow instructions to put the PC together. They open the app on his phone and start the PC assembly tutorial. When they start the tutorial, the app prompts them to rotate the phone screen for a better field of view (Figure 2a). Next, the app detects the planes in the scene and places a clipboard on the real-world surface (Figure 2b). The clipboard suggests the tools users will need to assemble the PC (Figure 2c). To tell if users are executing the task correctly, a green mark is displayed in case of success, and a red sign in case of failure (Figure 2d). The app guides users through each process step, suggesting the appropriate tools and providing visual aids to help them complete the task successfully (Figure 2e). Users complete the PC assembly without any dificulties (Figure 2f).
(a) Plane Detection (b) The Clipboard (c) Tool’s suggestion (d) Success/Failure of the task (e) Additional information (f) Tutorial Completed
Consider users who want to decorate their Christmas tree but are unsure of where to begin. They launch an app that ofers a tutorial to help them get started. They can do the tasks in any sequence they choose because the tutorial, in this instance, is made up of unordered tasks. When they start the tutorial, they see a clipboard on the right-hand side of the screen (Figure 3a). Each of the spheres on the clipboard is a diferent task that they must finish. The sphere representing the task they are currently working on is highlighted in yellow (Figure 3b). Users decide to start with the third task, which involves placing a ball on the tree even though they haven’t completed the second task yet. As they work on each task, they can tap on the clipboard to see a small panel appear to the left of it. The panel contains an illustrative image to help them complete the task correctly (Figure 3c). Once users have completed all the tasks, the tutorial is ifnished (Figure 3d).
(a) The Clipboard
(b) Unordered manner (c) Additional Information (d) Tutorial Completed
We discuss here the information needed to construct these two examples with TutorialKit. First of all, the type of tutorial to be conducted must be specified i.e., the way in which the tasks are to be performed, ordered or unordered. In the first example (Assembly a PC) they are ordered, while in the second one (Decoration of a Christmas Tree) they are unordered. Next, we needed to collect all information about the tasks, such as the title, the text that explains the action which the user should do, and all information about the tools, like the name and associated image. The ease of creating these two tutorials lies in the fact that this information is contained within a JSON file that is loaded when the application is started.
In this paper, we introduced an AR toolkit to help novice developers implement tutorials. The toolkit relies on a JSON description of the tasks involved and on callbacks for identifying the successful or wrong completion of the tasks. It manages the visualisation of a clipboard-based guidance interface and the highlighting of the required tools in the AR environment. We included two examples showing the required information and the resulting tutorials.
In future work, we will perform a thorough examination of our prototype. We plan to conduct user studies with novice developers to collect feedback to enhance our work. One of the most important extensions is an authoring environment for end-user developers. It will provide instruments to manipulate JSON files and handle the success and failure of tasks without writing code. Minor improvements include compatibility with further operating systems and devices, such as AR headsets, and the management of haptic feedback on task completion. impact on speed of learning and task performance in aeronautical engineering technology education, The International Journal of Aerospace Psychology 31 (2021) 219–229. doi:10. 1080/24721840.2021.1881403. [15] J.-R. Chardonnet, G. Fromentin, J. C. M. Outeiro, Augmented reality as an aid for the use of machine tools, Research and Science Today Supplement (2017) 25–31. [16] C.-M. Chen, Y.-N. Tsai, Interactive augmented reality system for enhancing library instruction in elementary schools, Computers & Education 59 (2012) 638–652. doi:https: //doi.org/10.1016/j.compedu.2012.03.001. [17] D. Forest, Training the next generation of operators: Afpm immersive learning, Process
Safety Progress 40 (2021). doi:10.1002/prs.12246. [18] D. Eckhof, C. Sandor, C. Lins, U. Eck, D. Kalkofen, A. Hein, Tutar: augmented reality tutorials for hands-only procedures, Proceedings of the 16th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and its Applications in Industry (2018). [19] A. Villanueva, Z. Zhu, Z. Liu, K. Peppler, T. Redick, K. Ramani, Meta-ar-app: An authoring platform for collaborative augmented reality in stem classrooms, CHI ’20, Association for Computing Machinery, New York, NY, USA, 2020. URL: https://doi.org/10.1145/3313831. 3376146. doi:10.1145/3313831.3376146.