Persuasive XR Training: Improving Training with AI and Dashboards

Abstract

With the rapid growth of Extended Reality (XR) technologies for training purposes, it has become essential to incorporate Artificial Intelligence (AI) modules into the simulations to assist trainers and trainees. One powerful AI solution is the use of recommender systems (RS) to enhance user interactions and experiences in immersive training. This work explores the integration of a RS into an XR training platform and focuses on the design of persuasive interfaces to present recommendations. Personalised training goals can be achieved for more successful outcomes by allowing trainers to modify training scenarios during the exercise. The goal of this work is to illustrate how effective integration of AI and persuasive interfaces in XR training platforms can result in successful and personalised training outcomes.

Keywords

• eol>Persuasive interface
• XR Training
• Persuasive technology
• Artificial Intelligence1

1. Introduction

The implementation of Extended Reality (XR) training systems has gained significant attention in various domains, including but not limited to law enforcement, medical first responders, and CBRNe specialists [22,25,27]. These systems offer a highly customisable and realistic simulation of complex environments, providing trainers and trainees with a valuable tool to prepare for real-world scenarios. Training within these domains often includes simulation of high-risk and high-stress environments, not permitted or too challenging to replicate in a realworld training setting. The inclusion of vulnerable populations (e.g., infants, people with disabilities) or hazardous equipment (e.g., explosives or poisonous substances) would be an example. VR training offers the opportunity to increase scenario repetitions with increasing levels of difficulty necessary to produce optimal training outputs. Stress levels of scenarios can be increased by including additional stressors in a scenario (e.g. weapons, number of injured avatars, loud noise). For a successful simulation training the right level of stress exposure plays a crucial role in optimising training outcomes [27].

Despite the benefits of these systems, one of their limitations is the lack of adaptability during a training session. For instance, consider a scenario where the virtual environment is too stressful for the trainees, but it is not possible for the trainer to make changes to reduce the stress level during the session. To ad- dress this issue, the integration of a recommendation system (RS) and a control dashboard can be particularly advantageous. The dashboard serves as a bridge between the AI and the trainer, allowing for real-time presentation of automatic recommendations adapted to the conditions of the participants. This enhances the personalisation of the training experience, by promoting engagement and positive learning outcomes.

The aim of this work is to present how successful and personalised training outcomes can be achieved through the effective integration of AI and persuasive interfaces in XR training platforms.

2. Related Works

The use of RSs has become ubiquitous in websites and applications as a means to enhance the user experience through the provision of personalised recommendations based on the user’s interaction history with the system. The primary objective of RSs is to advertise products or services that are likely to be of interest to the user. In the field of RSs, research efforts have mainly been directed towards the computational aspects, with a focus on improving the efficacy and relevancy of the recommendations. This has led to the development of various AI techniques, including deep and geometric learning [ 14, 29 ] and Bayesian and collaborative filtering methods [ 7, 24 ]. However, despite the significant progress made in improving the computational aspects of RSs [28], little attention has been paid to enhancing the way users interact with these systems. In recent years, there has been a growing interest in the presentation level of RSs [ 12 ], with a focus on understanding the impact of these systems on persuasion and satisfaction [17]. As the quality of recommendations increases, it becomes increasingly important to provide users with transparent and interpretable visualizations that support their understanding and interaction with the RS [23]. With the rapid advancement and widespread adoption of virtual and augmented reality, there has been a growing interest in exploring the integration of RSs into these technologies. However, these works have largely focused on the computational aspects of the recommendation, neglecting the important elements of presentation and explainability [ 8, 11, 26 ].

In the context of XR training, research has been conducted to provide trainers with dashboards to review a training session with the possibility to playback the actions, flag relevant events and control trainees’ stress levels [ 13, 16, 19 ]. Albeit successful, these solutions only provide trainers with an overview of what is currently happening in the virtual environment without the possibility of making changes to the scene, such as introducing an additional stressor, if necessary.

Following the direction set in [20], which presented a framework for implementing AIbased RS in the context of XR training, this work will discuss how persuasive interfaces (i.e. dashboards) and AI methods might improve the effectiveness of XR training systems. The next sections will introduce the proposed solution and present the design for a persuasive dashboard. Further, the explain- ability and persuasive elements of the suggested AI will be discussed.

3. Envisioned Solution

3.1. AI-Supported Training

In order to address the challenges of creating adaptive training scenarios, a novel strategy is proposed. A machine learning approach is implemented to provide meaningful recommendations for elements such as stressors, weather conditions, and NPCs that can be used to enhance the training experience and customise the level of difficulty trainees face. The data collected during previous training sessions are used to tailor each scenario to the trainees’ needs. In the current version stress level measurement data is based on heart rate (HR), heart rate variability (HRV) and electrodermal activity (EDA). The proposed solution (Figure 1) leverages data from both the trainer and the trainees to enhance the effective- ness of the training. By allowing trainers to modify specific stressors during the exercise, the system can provide personalised training experiences that better meet the needs of the participants.

The envisioned system for generating recommendations relies on a combination of supervised and unsupervised learning techniques, as well as reinforcement learning. Supervised learning would be used to train the system on existing data, while unsupervised learning would be used to identify patterns and trends in the data that may not be immediately obvious. The system would also use reinforcement learning techniques to optimise the training scenarios and ensure that they provide the best possible experience for the trainees. The insights gained from supervised and unsupervised learning, along with the optimisation provided by reinforcement learning, would be used to generate personalised XR training recommendations based on various factors, such as response to stressors, behaviour during training, and feedback. Once the model has generated recommendations, it would use persuasive techniques to encourage trainers to follow these recommendations. 2

3.2. Envisioned Solution

The user interface for the persuasive RS plays a critical role in helping trainers make informed decisions for their trainees. The interface utilises data analysis and machine learning algorithms to offer customised recommendations. Additionally, it employs persuasive strategies to motivate trainers to implement these recommendations. Trainers were consulted within the Med1stMR project [ 9 ]1 to identify the essential features required in a training dashboard for an XR system developed to train medical first responders in handling mass casualty incidents. Table 1 outlines the requirements identified by trainers, including the need for real-time feedback, and the ability to modify stressors and other elements of the training scenario. Based on these requirements, our solution was built with a focus on providing a user-friendly dashboard that allows trainers to easily access and analyse data from previous training sessions, as well as receive personalised recommendations for enhancing the training experience. 1 The project is funded by the European Union’s Horizon 2020 Research and Innova- tion Program under grant agreement No 101021775

View of the status of the trainee: stress measurement

Weather conditions

Time of day/night

Additional casualties

Inclusion of stressors

The trainer should be informed about upcoming scheduled events. Trainer has final say about changes

Ensure trainee’s safety

Modify training scenario to prevent redundancy in training

Adjust training scenarios

based on trainee performance

Trainer keeps final control over the scenario

The interface is designed with persuasive strategies in mind [18], as the goal is to influence trainers to make decisions that are most beneficial for their trainees. One persuasive strategy used in the interface is the principle of social proof [ 3 ], which involves showing trainers evidence of the effectiveness of certain training scenarios or stressors based on past data. This can help to convince trainers to adopt these recommendations and use them in their training sessions. Another persuasive strategy used in the interface is the principle of authority [ 3 ], which involves using the system’s knowledge and experience, based on previous training sessions and feedback from past training, to influence the decisions of trainers. For example, the system could use natural language processing to provide real- time feedback during training sessions, offering guidance and advice based on the performance of the trainees. This could help to establish the system as an expert in the field of training and encourage trainers to follow its recommendations. Figure 2, presents an initial prototype of the proposed interface which includes the above-mentioned features.

By leveraging data analysis, machine learning, natural language processing, and persuasive techniques, the system can help trainers to make the best deci- sions for their trainees, leading to more effective training outcomes and improved performance. Finding the optimal stress level to maximize learning performance is a challenge in simulation training [ 10 ] and can be different for each individual. It is therefore currently strongly recommended to keep the trainer in the loop (human-in-the-loop) so they can review and validate the recommendation made by the system. Keeping the trainer involved may have the additional benefit to build a relationship between the RS and the trainer, increasing the trust in the technology.

3.3. Explainability

The efficacy of the presented dashboard relies on the ability of the trainer to understand and follow the suggestions provided by the AI model. To ensure a dependable relationship between the model and the user, the works of [ 1 ], [ 2 ] and [ 6 ] were used to implement explainable design solutions in the dashboard. For instance, in the top right corner of Figure 2, the AI explains to the trainer that the stress level is currently below the optimal level and that worsening the victim’s condition could resolve the issue. Similarly, in the bottom right corner of Figure 2, the model presents elements included in previous training that could increase the stress of the trainees by showing the expected increase in stress level.

3.4. Evaluation

Evaluation studies will be performed to assess the design and effectiveness of the proposed dashboard. In particular, the evaluation will focus on examining the quality of the presented recommendations, the design of the persuasive interface, and the user experience of the trainers.

To assess the quality of the recommended items, the ResQue framework [21] will be used. This user-centered framework allows for easy identification of areas of improvement in the recommendation system as well as in the interface and user interaction modalities. The evaluation will measure the effectiveness of the proposed persuasive strategies in motivating trainers to adopt effective teaching practices.

To evaluate the design of the persuasive interface, an adapted version of the Technology Acceptance Model (TAM) questionnaire [ 4 ] will be used. This adaptation of the TAM, developed in [ 5 ], has been shown to be effective in measuring user acceptance in persuasive interfaces [ 15 ]. The questionnaire will assess factors such as perceived ease of use, usefulness, and attitude towards the dashboard, providing insight into the effectiveness of the persuasive strategies employed in the interface design.

Overall, the evaluation studies will provide valuable feedback on the effectiveness of the proposed dashboard and persuasive strategies, as well as identify areas for further improvement to enhance the learning outcomes of trainees in XR training programs.

4. Conclusion

This work demonstrates how the integration of AI and persuasive interface design may enhance the performance of trainers in XR training systems. By utilising AI algorithms to analyse data collected from trainers and trainees, the system can provide real-time feedback to trainers regarding their teaching strategies and effectiveness. Moreover, by incorporating persuasive design principles, such as authority and social influence, the system can motivate trainers to increase their engagement with the training process. By presenting data and insights in a clear and compelling manner, the system can encourage trainers to adopt effective teaching practices and improve the quality of their training. Overall, this approach has the potential to improve the quality and acceptance of XR training systems and contribute to the overall success of the training exercise.

In conclusion, future work will aim to improve the persuasive design of the trainer’s dashboard through user evaluations conducted in both laboratory and real-world settings. In addition to the proposed persuasive strategies of authority and social proof, the dashboard should also incorporate other approaches such as gamification and the persuasive strategy of commitment and consistency. for Extended Reality, Artificial Intelligence and Neural Engineering (MetroXRAINE). pp. 69–74. IEEE (2022) [17] Nanou, T., Lekakos, G., Fouskas, K.: The effects of recommendations’ presentation on persuasion and satisfaction in a movie recommender system.

Multimedia systems 16(4), 219–230 (2010) [18] N´emery, A., Brangier, E.: Set of guidelines for persuasive interfaces: Organization and validation of the criteria. Journal of Usability Studies 9(3) (2014) [19] Nguyen, Q., Jaspaert, E., Murtinger, M., Schrom-Feiertag, H., EggerLampl, S., Tscheligi, M.: Stress out: translating real-world stressors into audio-visual stress cues in vr for police training. In: Human-Computer Interaction–INTERACT 2021: 18th IFIP TC 13 International Conference, Bari, Italy, August 30–September 3, 2021, Proceedings, Part II 18. pp. 551–561. Springer (2021) [20] Pretolesi, D.: Personalised training: Integrating recommender systems in xr training platforms. In: Marky, K., Gru¨nefeld, U., Kosch, T. (eds.) Mensch und Computer 2022 - Workshopband. Gesellschaft fu¨r Informatik e.V., Bonn (2022) [21] Pu, P., Chen, L., Hu, R.: A user-centric evaluation framework for recommender systems. In: Proceedings of the fifth ACM conference on Recommender systems. pp. 157–164 (2011) [22] Regal, G., Murtinger, M., Schrom-Feiertag, H.: Augmented cbrne responderdirections for future research. In: 13th Augmented Human International Confer- ence. pp. 1–4 (2022) [23] Richthammer, C., S¨anger, J., Pernul, G.: Interactive visualization of recommender systems data. In: Proceedings of the 4th Workshop on Security in Highly Connected IT Systems. pp. 19–24 (2017) [24] Schafer, J.B., Frankowski, D., Herlocker, J., Sen, S.: Collaborative Filtering Recommender Systems. Springer Berlin Heidelberg, Berlin, Heidelberg (2007) [25] Schneeberger, M., Paletta, L., Wolfgang Kallus, K., Reim, L., Sch¨onauer, C., Peer, A., Feischl, R., Aumayr, G., Pszeida, M., Dini, A., Ladst¨atter, S., We- ber, A., Almer, A., Wallner, D.: First Responder Situation Reporting in Vir- tual Reality Training with Evaluation of Cognitive-emotional Stress using Psy- chophysiological Measures. Cognitive Computing and Internet of Things 43 (2022). https://doi.org/10.54941/ahfe1001841 [26] Shravani, D., Prajwal, Y., Atreyas, P.V., Shobha, G.: Vr supermarket: a virtual reality online shopping platform with a dynamic recommendation system. In: 2021 IEEE International Conference on Artificial Intelligence and Virtual Real- ity (AIVR). pp. 119–123. IEEE (2021) [27] Zechner, O., Kleygrewe, L., Jaspaert, E., Schrom-feiertag, H., Hutter, R.I.V., Tscheligi, M.: Enhancing Operational Police Training in High Stress Situations with Virtual Reality : Experiences , Tools and Guidelines (2023) [28] Zhang, Q., Lu, J., Jin, Y.: Artificial intelligence in recommender systems. Complex & Intelligent Systems 7(1), 439–457 (2021)

References

[1] Abdollahi , B. , Nasraoui , O. : Transparency in Fair Machine Learning: the Case of Explainable Recommender Systems , pp. 21 - 35 . Springer International Publishing, Cham ( 2018 )

[2] Bilgic , M. , Mooney , R.J.: Explaining recommendations: Satisfaction vs. promotion . In: Beyond personalization workshop, IUI. vol. 5 , p. 153 . Association for Computing Machinery , New York, NY, USA ( 2005 )

[3] Cialdini , R.B. , Goldstein , N.J.: The science and practice of persuasion . CornellHotel and Restaurant Administration Quarterly 43 ( 2 ), 40 - 50 ( 2002 )

[4] Davis , F.D.: User acceptance of information technology: system characteristics, user perceptions and behavioral impacts . International journal of man-machine studies 38(3) , 475 - 487 ( 1993 )

[5] Davis , F.D. , Venkatesh , V. : Toward preprototype user acceptance testing of new information systems: implications for software project management . IEEE Trans- actions on Engineering management 51(1) , 31 - 46 ( 2004 )

[6] Dragoni , M. , Donadello , I. , Eccher , C. : Explainable ai meets persuasiveness: Trans- lating reasoning results into behavioral change advice . Artificial Intelligence in Medicine 105 , 101840 ( 2020 )

[7] Ekstrand , M.D. , Riedl , J.T. , Konstan , J.A. , et al.: Collaborative filtering recom- mender systems . Foundations and Trends® in Human-Computer Interaction 4 ( 2 ), 81 - 173 ( 2011 )

[8] Haoyun , L. , Yukun , C. , Xie , X. : Vr-explorer: A demonstration of a virtual reality interactive data exploration system . In: The 2023 ACM International Conference on Supporting Group Work (GROUP'23) Companion . pp. 15 - 17 ( 2023 )

[9] Harthum , B. , Schrom-Feiertag , H. , Wenighofer , R.: Desaster szenario unter tage- med1stmr-neue ansatze im training von medizinischen ersthelfern . BHM Berg-und Hu¨ttenm¨annische Monatshefte 166 ( 12 ), 589 - 595 ( 2021 )

[10] Hernando-Gallego , F. , Art´es-Rodr´ıguez, A.: Individual performance calibration us- ing physiological stress signals . ArXiv abs/1507 .03482 ( 2015 )

[11] Huang , J. , Zhang, H., Lu , H. , Yu , X. , Li , S.: A novel position-based vr online shop- ping recommendation system based on optimized collaborative filtering algorithm . arXiv preprint arXiv:2206.15021 ( 2022 )

[12] Jugovac , M. , Jannach , D. : Interacting with recommenders-overview and research directions . ACM Transactions on Interactive Intelligent Systems (TiiS) 7 ( 3 ), 1 - 46 ( 2017 )

[13] Koutitas , G. , Smith , S. , Lawrence, G.: Performance evaluation of ar/vr training technologies for ems first responders . Virtual Reality 25 , 83 - 94 ( 2021 )

[14] Leng , Y. , Ruiz , R. , Dong , X. , Pentland , A. : Interpretable recommender system with heterogeneous information: A geometric deep learning perspective . SSRN Electron.J ( 2020 )

[15] Meschtscherjakov , A. , Wilfinger , D. , Scherndl , T. , Tscheligi , M. : Acceptance of future persuasive in-car interfaces towards a more economic driving behaviour . In:Proceedings of the 1st international conference on automotive user interfaces and interactive vehicular applications . pp. 81 - 88 ( 2009 )

[16] Murtinger , M. , Uhl , J. , Schrom-Feiertag , H. , Nguyen , Q. , Harthum , B. , Tscheligi , M. : Assist the vr trainer-real-time dashboard and after-action review for police vr training . In: 2022 IEEE International Conference on Metrology