While artificial intelligence (AI) has become part of more and more areas of daily life - both private and business - this development has not yet progressed as far in the military sector. This is just changing with the development of new projects, such as the Future Combat Air System (FCAS) - a highly ambitious European defense project planned as a replacement of systems such as the Eurofighter from 2040 onwards. To facilitate and accelerate discussions on the ethical implications of the use of AI in the military domain, we developed the FCAS Ethical AI Demonstrator. We chose the Target Detection, Recognition, and Identification as one highly probable use case and implemented a simulation to showcase the ethical implications of the collaboration between the operator and an AI-assisted system in that application. To help the operator understand and assess the classifications of the used automatic target recognition, explanations of the AI results are computed with an Explainable AI (XAI) method and then provided in the user interface. With this hands-on demonstrator, we are pleased to contribute to the discussions on the ethical implications of the use of AI in military applications.
Artificial intelligence (AI) has become a technology that influences social and economic life
in many ways – ChatGPT shows this vividly [
AI will be utilized in FCAS to help operators focus on the information relevant to the
situation at hand. However, the use of AI in such a sensitive domain poses serious ethical
and legal questions. To investigate the responsible use of new technologies in FCAS and to
determine necessary guidelines for such a system, the FCAS Forum [
Using Explainable AI (XAI) methods is imperative here, since it is crucial for the operator to
understand and be able to assess the AI’s assistance. Extracting the information on which neural
networks base their decision, thus making the decision process explainable resp. interpretable,
is the focus of the research field XAI. It was heavily influenced by the XAI program the
Defense Advanced Research Projects Agency (DARPA) launched in 2017 [
In the following, we first elaborate on the motivation of developing the FCAS Ethical AI Demonstrator. This requires a rough introduction into the operational context of a military situation. After that, we explain the actual scope and content of the demonstrator. We then highlight the technical details including the relevance of XAI in it. Finally, the necessity of XAI for developing ethical AI is discussed.
The use of AI is expected to have an immense impact on the conduct of military operations [21]. Performing operational activities in a dynamic environment requires a quick adaptation of decisions. This is formalized in the OODA-Cycle [22], short for Observe, Orient, Decide, and Act. It describes four stages of decision-making in fast changing environments. A key question in relation to the use of AI in these stages of decision-making is the degree of human involvement. In this context, a machine’s level of authority can be described by its dependency on human actors in the execution of the OODA-Cycle activities, especially in light of operational uncertainty [23]: Human-in-the-loop (human makes decisions and acts), Human-on-the-loop (systems make decisions and act, human is monitoring and can intervene), or Human-out-of-the loop (systems make decisions and act, no human intervention possible)[24].
The role of the human in the application of AI in military operations and the allocation of
responsibility for machine executed authorities appear central for the development of regulatory
frameworks and the discussion of ethical implications [
The FCAS Ethical AI Demonstrator showcases the collaboration between an operator and an AI performing DRI. One exemplarily implemented scenario shows an unmanned aerial system (UAS) flying ahead of the main forces to detect and identify hostile air defense systems (see Figure 1). On the ground, military vehicles are intermixed with civilian infrastructure. The user of the demonstrator, i.e. the operator of the UAS, has to decide upon the AI’s detections. They must verify or reject a detected target or mark it for further investigation.
To keep the operator with meaningful control in the loop the target detection is presented together with an explanation. For this, the well-established XAI method LIME (Local Interpretable Model-Agnostic Explanations, [26]) is used to generate heatmaps visualizing the features of the targets which were essential for the detection. The user of the demonstrator is thus set in a possible real-life situation where they can experience the impact an AI-based assistant could have on the process of target selection. This especially facilitates discussions about ethical issues which might arise.
The FCAS Ethical AI Demonstrator is implemented as a web-based application system. Correspondingly, the synthetic video, ATR data and LIME explanation are combined and controlled in a web-based user interface. In the following, we first briefly describe the synthetic video. Then, we describe the used ATR and how its detections are explained with LIME. Finally, the overall application architecture and user interface (UI) are described. As a data basis for the demonstrator, we use a 4K video generated within the high fidelity Airbus pilot training environment. It is based on a scriptable aircraft model carrying a controllable video pod and observing a manually defined automated ground scenario. The necessary metadata for geolocalization is embedded into the recording.
The ATR AI model used on the synthetic video is based on the Airbus proprietary CeMoreDeep architecture. It combines a feature extractor of a convolutional neural network with a highly optimized support vector machine to detect and classify objects. Furthermore, the Airbus proprietary software AI Engine RT stabilizes the generated tracks and transfers the predicted position into latitude and longitude for geolocalization and visualization on a map. As this ATR model is provided as a black box, a model agnostic XAI method is necessary to explain its classifications. While several approaches like SHAP [ 27] and Ablation-CAM [28] are suitable, we have chosen LIME [26], a well-established surrogate based XAI model, for this purpose. LIME works by sampling input images and creating locally interpretable models around each given image. By doing so, it identifies the critical areas of the image that the ATR model uses to make its predictions. This allows users to better understand the decision-making process of the ATR model and evaluate the quality of its classifications.
The demonstrator is implemented as a distributed architecture using web-based technologies. After logging in into the web UI, the user can play back the video; ATR and LIME data are loaded from a web server and displayed accordingly. The UI guides the user through the experience and additionally collects his or her reactions.
Jobin et al. [29] analysed the guidelines and principles that were issued to constitute ethical AI. Five dominant principles were identified: transparency, justice and fairness, non-maleficence, responsibility, and privacy [29, 30]. XAI results, here the visualization of relevant image parts, help to make the decision-making process of the AI more transparent. While ethics provides principles that should be met by an AI in order for it to be considered ethical, XAI can be an enabler to meet the requirements and uncover other ethical issues that may arise in a particular context. With the FCAS Ethical AI Demonstrator, we contribute to the discussions on the ethical use of AI in a specific context, the military domain. [16] W. Samek, G. Montavon, S. Lapuschkin, C. J. Anders, K.-R. Müller, Explaining deep neural networks and beyond: A review of methods and applications, Proceedings of the IEEE 109 (2021) 247–278. [17] D. Minh, H. X. Wang, Y. F. Li, T. N. Nguyen, Explainable artificial intelligence: A comprehensive review, Artif. Intell. Rev. 55 (2022) 3503–3568. URL: https://doi.org/10.1007/ s10462-021-10088-y. doi:10.1007/s10462-021-10088-y. [18] C. Molnar, Interpretable Machine Learning, 2 ed., 2022. URL: https://christophm.github.io/ interpretable-ml-book. [19] A. Holzinger, A. Saranti, C. Molnar, P. Biecek, W. Samek, Explainable AI Methods - A Brief
Overview, 2022, pp. 13–38. doi:10.1007/978-3-031-04083-2\_2. [20] U. Schmid, B. Wrede, What is missing in XAI so far?: An interdisciplinary perspective, KI - Künstliche Intelligenz 36 (2022). doi:10.1007/s13218-022-00786-2. [21] P. Svenmarck, L. Luotsinen, M. Nilsson, J. Schubert, Possibilities and challenges for artificial intelligence in military applications, in: Proceedings of the NATO Big Data and Artificial Intelligence for Military Decision Making Specialists’ Meeting, 2018, pp. 1–16. [22] R. Coram, Boyd: The fighter pilot who changed the art of war, Hachette+ ORM, 2002. [23] M. Firlej, A. Taeihagh, Regulating human control over autonomous systems, Regulation &
Governance 15 (2021) 1071–1091. [24] M. Azzano, S. Boria, S. Brunessaux, B. Carron, A. Cacqueray, S. Gloeden, F. Keisinger, B. Krach, S. Mohrdieck, The responsible use of artificial intelligence in FCAS—an initial assessment (2021). White Paper. Available online at https://www.fcasforum.eu/articles/responsible-use-of-artificial-intelligence-in-fcas. [25] F. E. Morgan, B. Boudreaux, A. J. Lohn, M. Ashby, C. Curriden, K. Klima, D. Grossman, Military applications of artificial intelligence: ethical concerns in an uncertain world, Technical Report, RAND PROJECT AIR FORCE SANTA MONICA CA SANTA MONICA United States, 2020. [26] M. T. Ribeiro, S. Singh, C. Guestrin, "why should i trust you?" explaining the predictions of any classifier, in: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, 2016, pp. 1135–1144. [27] S. M. Lundberg, S.-I. Lee, A unified approach to interpreting model predictions, Advances in neural information processing systems 30 (2017). [28] H. G. Ramaswamy, et al., Ablation-cam: Visual explanations for deep convolutional network via gradient-free localization, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2020, pp. 983–991. [29] A. Jobin, M. Ienca, E. Vayena, The global landscape of AI ethics guidelines, Nature Machine
Intelligence 1 (2019) 389–399. [30] H. Vainio-Pekka, M. O.-o. Agbese, M. Jantunen, V. Vakkuri, T. Mikkonen, R. Rousi, P. Abrahamsson, The role of explainable AI in the research field of AI ethics, ACM Trans. Interact. Intell. Syst. (2023). URL: https://doi.org/10.1145/3599974. doi:10.1145/3599974, just Accepted.