Gamifying a flight instructor simulator with an evidence-based training approach Javier Cruz1, Antonio J. Guerra1, Rubén Dapica2, Federico Peinado1,* 1 Complutense University of Madrid, Department of Software Engineering and Artificial I ntelligence. c/ Profesor José García Santesmases 9, 28040 Madrid, Spain 2 European University of Madrid, Department of Mechanical and Aerospace Engineering. c/ Tajo s/n, Urb. El Bosque, 28670 Villaviciosa de Odón, Madrid, Spain Abstract Even today, the human factor is one of the causes that contributes to the majority of air accidents. That is why, in recent years, more emphasis has been placed on improving the formation received by pilots. To this end, the aviation industry is driving a new paradigm called Evidence-Based Training, which collects data and integrates information on pilot performance across all necessary competencies, both technical and non-technical. Evaluation of non-technical skills requires well-trained instructors, so a gamified simulator has been developed with the purpose of teaching to identify observable behaviors of non-technical competencies in pilots, under this new paradigm. A first experiment with 8 experts provided a lot of feedback with which to improve this game as a complement to aeronautical training. Keywords Aeronautics, Game Development, Assessment, Non-Technical Competencies, Observable Behavior 1. Introduction Despite all the advances in the field of aviation safety, the human factor continues to be one of the causes that contributes to the majority of air accidents [1]. For this reason, in recent years, both the International Civil Aviation Organization (ICAO) [2] and the International Air Transport Association (IATA) [3] have been promoting a new paradigm called Evidence-Based Training (EBT). This paradigm attempts to adapt training to the real needs of pilots, integrating both technical and non-technical competencies. This new training model is concerned not only with pilot technical skills (i.e. the correct execution of manoeuvres, knowledge of the aircraft controls, etc.), but also with their non- technical skills, more related to the so-called Human Factors. Flight instructors who are required to train pilots using the EBT model must observe certain patterns of pilot conduct called Observable Behaviours (OBs) that reveal whether or not the pilot has the corresponding competencies, such as Communication or Leadership. I Congreso Español de Videojuegos, December 1–2, 2022, Madrid, Spain * Corresponding author. $ javiercldo2000@gmail.com (J. Cruz); aguerragarduno@gmail.com (A. J. Guerra); rubendap@ucm.es (R. Dapica); email@federicopeinado.com (F. Peinado) € https://federicopeinado.com/ (F. Peinado)  0000-0002-1891-1625 (J. Cruz); 0000-0003-3189-306X (A. J. Guerra); 0000-0002-0750-7424 (R. Dapica); 0000-0002-8893-0020 (F. Peinado) © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) It is not easy to follow the guidelines of the assessment process in this new paradigm and often subjective assessments are made in which the instructor does not simply observe and record the pilot’s OBs. Instructors are therefore faced with this new training paradigm that is quite different from that traditionally used in aviation, so the question “Who trains the trainers” arises, the central topic of a previous paper by Hernández, Dapica and Peinado [4]. 2. Related Work Aviation has a long tradition in the use of simulators, mainly due to the need to ensure a safe and controlled environment for pilot training. That is why the industry has not stopped investing and innovating in the development of these devices. Proof of this is the fact that the European Aviation Safety Agency (EASA) is fully adapted to new virtual reality technologies1 . Serious games, originally defined as games for educational purposes that are still fun, have been used since the 19th century, even if they were originally paper-based [5]. Today, techno- logically advanced serious games of different kinds serve a wide range of purposes [6]. The combination of simulation and gaming gives rise to the gamified simulator, a very useful model for training professionals that is the main proposal of this work. Some examples of gamified simulators in other areas of application are Afghanistan Pre-deployment: Cultural Awareness [7], used by the US armed forces to train their soldiers before being deployed in Afghanistan at the beginning of this conflict; STAR Classroom Simulator [8], where a teacher can improve his skills as a tutor by interacting with virtual students; and the Versa Project 2 , about developing soft skills through the use of commercial video games, on PhD students from different countries under the European University Alliance “Aurora”. 3. Flight Instructor Simulator After a first phase of documentation and design, advised by aeronautics experts, the flight instructor simulator FI-SIM, was developed. For its development we use the latest version of the Unity development environment, and some help from the educative game company Didactoons3 . Its architecture allows different flight scenarios to be loaded and different pairs of virtual pilots to be tested, with the player taking the role of an instructor who, seated behind the cockpit, evaluates their performance in the different situations that arise (see Figure 1). Currently the simulator contains a basic tutorial and a detailed scenario based on the real crash of Air Niugini 73, in 2018[9]. We were assisted by expert flight instructors and the official accident report, to achieve the most credible simulation possible. The simulator is gamified by simplifying the player’s objective and establishing a clear dynamic, with short-term rewards. The game is available for free at Itch.io4 , and the source code is also available in a public repository on GitHub5 . 1 https://www.easa.europa.eu/newsroom-and-events/press-releases/easa-approves-first-virtual-reality-vr-based- flight-simulation 2 https://gecon.es/versa-project/ 3 https://didactoons.com 4 https://narratech.itch.io/fi-sim 5 https://github.com/Narratech/TFG-Cruz-Guerra Figure 1: FI-SIM in-game footage, showing virtual pilots in action and possible OBs that can be identified by the player. 4. Results and Discussion As a first experiment, we conducted a test session at Global Training Aviation6 ’s facilities with 8 expert instructors, who were to play FI-SIM’s tutorial and test scenario (6-minute mission in which the dialogues of the pilots are very close to the real ones). Experts can choose when, whom, and what observable behaviors of the pilot to assess, so this interaction helped us a lot to get relevant information for the evaluation of our game. Compatible apps with mobile devices (Android and iOS) has been produced that allows validation of its functionality and usability with real users in those devices. In short, we can say that all experts considered the new EBT paradigm useful and that the majority (75%) understood the purpose of the proposed gamified simulator. Only 50% of the experts were convinced by the tutorial, in order to learn basic game mechanics, and only 37.5% were convinced by the clarity and the possibility to perform their work well as instructors inside the game, which means there is still plenty of room for improvement. Most of them conveyed that they lacked information to fairly evaluate the pilots in the scenario. Some examples of improvements they suggested in this regard were to have a briefing before starting the scenario so that the player knows what the pilots in the scenario are going to face, to show the state of the aircraft by representing it in real time, or to know the pilot’s previous experience. In addition, we were recommended that each scenario should focus only on one or two competencies, thus achieving a better dosage of knowledge, which would translate into better performance when it comes to putting it into practice. 6 https://www.globaltrainingaviation.com/en 5. Conclusions A computational model for flight simulation scenarios based on EBT has been created with enough credibility and depth to be accepted as an educative proposal by aviation specialists. Thanks to the help of official crash reports, as well as independent experts, we were able to build a realistic simulation, including the evaluation criteria for the pilots training. In addition, a flight instructor simulator (FI-SIM) based on that model has been designed and developed to show scenarios allowing the player to reproduce the main part of the EBT instructor’s job, recognizing whether the OBs of pilots reveal the acquisition of non-technical skills necessary to carry out their missions safely and effectively. Although we have managed to reproduce the essentials of an instructor’s work, there is still room for improvement, as expert comments say that in some situations of the proposed scenario the correct assessment is debatable, and several or all of the observable behavioral options could be selected as correct, and a more holistic approach to the problem is desirable. It will also be necessary to apply a formal methodology, such as [10], to complete the gamification and validate its playful effectiveness. References [1] H. Kharoufah, J. Murray, G. Baxter, G. Wild, A review of human factors causations in com- mercial air transport accidents and incidents: From to 2000–2016, Progress in Aerospace Sci- ences 99 (2018) 1–13. doi:https://doi.org/10.1016/j.paerosci.2018.03.002. [2] M. I. C. A. Organization, Manual of evidence-based training, International Civil Aviation Organization, 2013. URL: https://skybrary.aero/sites/default/files/bookshelf/3177.pdf. [3] I. A. T. Association, Evidence-Based Training Implementation Guide, International Air Transport Association, 2013. [4] A. Hernández, R. Dapica, F. Peinado, Who trains the trainers? towards a flight instructors simulator based on training scenarios, in: VI Congreso de la Sociedad Española para las Ciencias del Videojuego, 2020. [5] J. Torrente, A. del Blanco, E. J. Marchiori, P. Moreno-Ger, B. Fernández-Manjón, lt;e- adventure gt;: Introducing educational games in the learning process, in: IEEE EDUCON 2010 Conference, 2010, pp. 1121–1126. doi:10.1109/EDUCON.2010.5493056. [6] A. De Gloria, F. Bellotti, R. Berta, Serious games for education and training, International Journal of Serious Games 1 (2014). doi:10.17083/ijsg.v1i1.11. [7] A. Barbieri, A. Tesei, A. Bruzzone, R. Ion, D. Beligan, Learning impact evaluation of the serious game “cultural awareness – afghanistan pre-deployment”, International Journal of Serious Games 1 (2014). doi:10.17083/ijsg.v1i3.27. [8] B. DaCosta, Virtual character conceptual framework: Assessing, preparing and retaining star teachers for effective classroom management in urban schools (2007). [9] A. I. Commission, Final Report AIC 18-1004, Technical Report, Accident Investigation Commission, 2018. [10] F. J. Gallego-Durán, C. J. V. Arnedo, R. Satorre-Cuerda, P. Compañ-Rosique, R. Molina- Carmona, F. Llorens-Largo, A guide for game-design-based gamification, Informatics 6 (2019) 49.