Extended Reality: Exploring End User Development Capabilities Valentino Artizzu1 1 Department of Mathematics and Computer Science, University of Cagliari, Cagliari, Italy Abstract The paper outlines the initial efforts of the Author, who is focused on developing solutions to assist end-users in generating Extended Reality experiences, even if they lack programming or 3D modelling expertise. The current research of the Author involves creating a system for vocational training creators to produce processes that function as either training or guidance tools for various Extended Reality devices, as well as an execution engine that can adjust the process based on the environment context of the user. Keywords Extended Reality, Mixed Reality, Augmented Reality, Virtual Reality, End User Development, Natural Language, Rule System, Event-Condition-Action Rules, Vocational Training 1. Introduction Consumer Extended Reality (XR) devices, such as Virtual, Augmented, and Mixed Reality, have gained popularity in recent years, offering immersive experiences mainly for gaming but also in other fields such as education and work tools. However, as with other technologies in the past, the more users adopt XR, the more they will demand control over creating content. At present, it is still challenging to enable end-users to create XR content, as it requires a team of experts in 3D modelling, code development, design, etc. Additionally, current development cycles are not suitable for end-users with evolving needs over time, as involving a professional developer is not always feasible. The goal of the PhD research discussed in this paper is to find solutions that support end-users in creating Extended Reality experiences without requiring programming experience or 3D modelling skills. The paper will discuss the current state of the research, starting with the current literature. 2. Related Work In this section, the relevant work related to the PhD research of the Author will be defined, highlighting the limitations of the current literature. IS-EUD 2023: 9th International Symposium on End-User Development, 6-8 June 2023, Cagliari, Italy Envelope-Open valentino.artizzu@unica.it (V. Artizzu) Orcid 0000-0003-0263-2434 (V. Artizzu) © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) The PhD research of the Author falls under the scope of End User Development (EUD) [1] approaches, and the literature suggests that most available work for XR experiences is limited to defining static scenes or multimedia content overlays. Fungus [2] is an open-source Unity extension for developing visual narratives that uses flowcharts to construct visual novels, but they require a large amount of screen space and reduces end-user comprehension [3]. Fanni et al [4] propose PAC-PAC, a tool for creating point-and-click games using a web- based authoring interface, and the definition of behavior is made through natural language, Event-Condition-Action rules. This work shares its roots with the previous research work of the Author, where the Rule system guidelines have been applied into a VR-focused derivative, ECARules4All [5]. The project extended the foundations of PAC-PAC to accommodate more complex interactions in full VR environments and defined roles for the Template Builders, End User Developers, and Final Users: the first one is the expert developer who creates almost- complete environments (the template), so the End User Developer, an user with average skills in computer use, can customise to get the final product that will be used by the Final User. Ariano et al [6] present an approach to making the process of creating smart home automa- tions more engaging and accessible for non-professional developers. The approach involves using augmented reality technology to provide users with more relevant, context-sensitive representations of connected sensors and objects. Unfortunately, despite they found some interesting insights from their study, it was only limited to smart home automations, and the PhD research of the Author would like to expand the idea to a more general context, where any field can benefit from the EUD paradigm. The vocational training field is another area related to the work of this paper. Many contribu- tions can be found, such as work related to weld training [7] and construction management [8], proving that learning in an immersive environment can be beneficial to the end-user. However, none of them gave the possibility to customize the experience once it was built by the developer. Regarding vocational training for AR, Chiang et al [9] conducted a systematic review and found that AR training has positive effects on vocational training outcomes. XOOM [10] is a tool designed for non-ICT people that lets them create web-based VR applications using 360° videos and superimposing content into the virtual scenes. The proposed work would like to expand one of the possibilites the authors envisioned for XOOM, which is vocational training, using complete VR experiences, with a focus on generalizing the training definition instead of creating universal rules for scene interaction. 3. Current State of Research Upon examining the current state of the art in Extended Reality (XR) and vocational training, it is evident that only a limited number of studies attempt to enable end-users to define XR experiences without relying on professional expertise. Additionally, there is no established method for creating or updating XR-based vocational training experiences by the end-users themselves, as the process tends to be monolithic. Consequently, the proposed research aims to bridge this gap. To involve end users in the development process, the study proposes the creation of a system that allows them to define vocational training procedures using a specialised authoring tool. The Author envisions that the creation of the interface for the tool will be realised by employing a Rule-authoring approach [11], where for every step the End User Developer can define the behaviour of the objects that must interact with the user (or interact with each other) in order to proceed to the following steps. The authoring tool, once a procedure has been finished editing, will produce two serialised files: the procedure and the context file. The former will contain the list of the steps defined by the End User, the main modality and a text description for both of these elements, the latter will contain, for each step, zero or more alternative modalities (with their respective descriptions) that will be activated if a certain user-defined or built-in context event is raised during the execution of the procedure. These files can be transferred to any device compatible with the implementation, covering the entire XR spectrum (e.g., as a VR training experience or an AR guidance tool). To accomplish this, the project also involves developing an execution engine that interprets procedure steps and translates them into XR interactions. The novelty of the system lies in its ability to adapt to the environment of the user and modify the interaction modality (e.g., touch, controller click, voice, gaze-and-commit, etc.) based on the current context. The components of the project include (as depicted in Figure 2): • An authoring tool for creating procedure and context files that can be used or transferred to other devices; • A Procedure Engine that reads and executes the steps of the procedure file based on user interactions (e.g., progressing to the next step upon button click); • A Context Engine that monitors potential issues detected by dedicated sensors (e.g., loud environment, unrecognizable user voice, user preferences, etc.) and adjusts the interaction modality in real-time, either during task execution or at task change if the original next step is incompatible with the current context. In particular, the adaptability of this engine is achieved by linking it to the sensors data of the system-executing device, and triggering context changes found in the context file when an event from the sensors of the device is detected. To ensure compatibility with any device, the Procedure Engine and Context Engine will be enclosed in an importable package that developers can incorporate into their projects. This package will also contain the authoring tool, although developers can choose not to import it as it is not essential for the base features to function correctly. The package will come with a sample interface for swift implementation of the various modes of the system and a set of APIs for adapting the sensor data of the target device into a format usable by the Context Engine. Additionally, the package will include a mechanism for providing feedforward [12] to end-users, allowing them to override the decision of the system if they prefer. The research methodology is structured as follows: • Initial problem identification; • State-of-the-art review on Extended Reality and Vocational Training; • Design, prototyping, and implementation of user-friendly tools for supporting end-users in the authoring process and trainees in the learning process; Figure 1: The current prototype interface. From top left to bottom right: The base interface, in its expanded version; the feedforward prompt; the modality choice list; the interface after a modality override. • Validation of results through user tests and/or comparisons with the current state of the art. The project is currently in the prototyping phase, with a preliminary functional version of the training system implemented using Mixed Reality Toolkit 3 [13]. It currently features an interface that displays the current step description and modality to use; the previous step, next step, and alternatives for the current step can be accessed by pressing the appropriate button at the bottom right side of the main panel (the ”current task” panel, which remains constantly visible throughout the interface). The context change can be activated by pressing predefined keys on the keyboard, which simulate sensor signals. These events can also be viewed by pressing the top right button of the main panel, which will show which event is currently occurring through icons. During task execution, from a graphical point of view, the next step will become the current step, and the task performed by the user will become the previous step. By pressing the button on the top left of the main panel, the user can enable the feedforward option, which notifies the user of the description of next task, and asks if they want to force a modality, in which case the context change will be ignored until the task is completed. At the end of the procedure, the user will be notified of the completion of the steps with a specific message. 4. Conclusions Overall, the paper presents a solution for enhancing XR experiences through End User Develop- ment principles, specifically in the context of vocational training. The proposed methodology involves the use of an authoring tool that allows users to define training procedures and migrate them throughout the XR spectrum. The system is also capable of adapting the procedures based Figure 2: The proposed system structure and components on the environment context of the user, using sensors and user preferences. It is worth noting that the research is currently being developed in collaboration with leading european research groups and will also involve in the future the expertise of an Italian ICT company focused on the valorization of cultural and environmental heritage. 5. University Doctoral Program and Context The PhD project is led by Valentino Artizzu, a second-year PhD student in Computer Science in the Department of Mathematics and Computer Science of the University of Cagliari (Italy), starting from January 2022. The expected end of the program is in December 2024. He has a Master’s Degree in Computer Science at the University of Cagliari in 2021, with a thesis based on the implementation of the rule engine that later became one of the main components of ECARules4All [5], after a scholarship and the beginning of the PhD program. His main research interests are XR technologies, Videogame Development and Design, and Internet of Things. He works in the CG3HCI (Computer Graphics & Human Computer Interaction) research group, founded by prof. Riccardo Scateni, who leads the Computer Graphics part of the group, and later joined by prof. Lucio Davide Spano, who leads the Human Computer Interaction one. 5.0.1. Acknowledgments This work is supported by the Italian Ministry and University and Research (MIUR) under the PON program: The National Operational Program “Research and Innovation” 2014–2020 (PON R&I). References [1] H. Lieberman, F. Paternò, V. Wulf, End user development, volume 9, Springer, 2006. [2] Fungus games, https://fungusgames.com. [3] T. R. G. Green, M. Petre, Usability analysis of visual programming environments: a ‘cognitive dimensions’ framework, Journal of Visual Languages & Computing 7 (1996) 131–174. [4] F. A. Fanni, M. Senis, A. Tola, F. Murru, M. Romoli, L. D. Spano, I. Blec̆ić, G. A. Trunfio, Pac- pac: end user development of immersive point and click games, in: End-User Development: 7th International Symposium, IS-EUD 2019, Hatfield, UK, July 10–12, 2019, Proceedings 7, Springer, 2019, pp. 225–229. [5] V. Artizzu, G. Cherchi, D. Fara, V. Frau, R. Macis, L. Pitzalis, A. Tola, I. Blecic, L. D. Spano, Defining configurable virtual reality templates for end users, Proceedings of the ACM on Human-Computer Interaction 6 (2022) 1–35. [6] R. Ariano, M. Manca, F. Paternò, C. Santoro, Smartphone-based augmented reality for end- user creation of home automations, Behaviour & Information Technology 42 (2023) 124–140. URL: https://doi.org/10.1080/0144929X.2021.2017482. doi:10.1080/0144929X. 2021.2017482 . [7] R. Stone, E. McLaurin, P. Zhong, K. Watts, Full virtual reality vs. integrated virtual reality training in welding (2013). [8] R. Sacks, A. Perlman, R. Barak, Construction safety training using immersive virtual reality, Construction Management and Economics 31 (2013) 1005–1017. [9] F.-K. Chiang, X. Shang, L. Qiao, Augmented reality in vocational training: A systematic review of research and applications, Computers in Human Behavior 129 (2022) 107125. [10] F. Garzotto, M. Gelsomini, V. Matarazzo, N. Messina, D. Occhiuto, Xoom: An end-user development tool for web-based wearable immersive virtual tours, in: Web Engineering: 17th International Conference, ICWE 2017, Rome, Italy, June 5-8, 2017, Proceedings 17, Springer, 2017, pp. 507–519. [11] End-user development for personalizing applications, things, and robots, International Jour- nal of Human-Computer Studies 131 (2019) 120–130. URL: https://www.sciencedirect.com/ science/article/pii/S1071581919300722. doi:https://doi.org/10.1016/j.ijhcs.2019. 06.002 , 50 years of the International Journal of Human-Computer Studies. Reflections on the past, present and future of human-centred technologies. [12] T. Djajadiningrat, K. Overbeeke, S. Wensveen, But how, donald, tell us how? on the creation of meaning in interaction design through feedforward and inherent feedback, in: Proceedings of the 4th conference on Designing interactive systems: processes, practices, methods, and techniques, 2002, pp. 285–291. [13] Mixed Reality Toolkit 3 Developer Documentation - MRTK3, 2023. URL: https://learn. microsoft.com/en-us/windows/mixed-reality/mrtk-unity/mrtk3-overview/.