This paper describes the motivation and themes for the workshop on Multimodal, Afective and Interactive eXplainable Artificial Intelligence (MAI-XAI), which was collocated with the 2024 European Conference on Artificial Intelligence (ECAI). In the era of Artificial Intelligence (AI), data scientists aim to enhance decisionmaking and automate processes, but many AI systems are challenging to interpret due to their opaque, black-box nature. To address this, eXplainable AI (XAI) seeks to develop intelligent agents that provide understandable decisions and explanations, fostering better human-machine interaction while ensuring ethical principles like fairness and transparency are upheld. The workshop focuses on improving XAI efectiveness through three key areas: Multimodal XAI, Afective XAI, and Interactive XAI, emphasizing the need for responsible and trustworthy AI systems. The workshop received 18 submissions covering all the three main topics mentioned in the call for papers. All the submissions received three reviews. Out of these papers, 9 were selected for presentation at the workshop, yielding an acceptance rate of 50%.
In the age of Artificial Intelligence (AI), data scientists apply AI techniques in a broad spectrum of tasks, aiming to enhance decision-making, automate repetitive processes, or automatically extract knowledge from massive data. The opacity of many AI systems, particularly those relying on machine learning models and deep neural networks, poses a challenge as the decisions made by these systems are often dificult to interpret, due to their black-box nature. Our focus is therefore on how to enhance, in an environmental-friendly way, human–machine interaction in the context of eXplainable AI (XAI).
XAI is an endeavour to evolve AI methodologies and technology by focusing on the development of intelligent agents capable of both generating decisions that a human can understand in each context, and explicitly explaining such decisions. This way, it is possible to scrutinize the underlying data and intelligent models. Moreover, analysing explainability of data and models is crucial to understand, detect and mitigate bias. Accordingly, XAI systems are expected to naturally interact with humans, thus providing comprehensible explanations of decisions made automatically. Moreover, data processing must be made in an eficient, scalable and sustainable computational way. Thus, XAI contributes to the development of Responsible and Trustworthy AI. By ensuring that automated decisions are made based on accepted rules and principles, they can be trusted and their impact justified, while respecting the ethical principles of human agency, prevention of harm, fairness and explainability.
XAI involves not only technical but also ethical, legal, socio-economic and cultural (ELSEC) issues. In addition to Ethical Guidelines and Codes of Conduct, the European General Data Protection Regulation (GDPR) and the European AI Act remark the need to push for a human-centred responsible, explainable and trustworthy AI that empowers citizens to make more informed, and thus better, decisions. In addition, as remarked in the XAI challenge stated by the US Defense Advanced Research Projects Agency (DARPA): “even though current AI systems ofer many benefits in many applications, their efectiveness is limited by a lack of explanation ability when interacting with humans”.
In order to improve the efectiveness of explanations and increase the ability of XAI systems when interacting with humans, the workshop explores three key topics: i) Multimodal XAI, ii) Afective XAI and iii) Interactive XAI.
Multimodal XAI
Multi-modality is demanded at the level of both data and models. Multi-modality requires dealing properly with structured and non-structured heterogeneous data (i.e., tabular data, text, images, sound, video, etc.). Multi-modal explanations must be customisable and easy to adapt not only nEvelop-O †These authors contributed equally. https://citius.gal/team/jose-maria-alonso-moral/ (J. M. Alonso-Moral); https://www.nup.ac.cy/faculty/zach-anthis/ (Z. Anthis); https://www.uji.es/departaments/lsi/base/estructura/personal&p_profesor=65331 (R. Berlanga); https://citius.gal/team/alejandro-catala-bolos/ (A. Catalá); (P. Cimiano); http://www.cs.bris.ac.uk/~flach/ (P. Flach); (E. Hüllermeier); https://uqtmiller.github.io/ (T. Miller); (D. Mindlin); https://mds.inf.ethz.ch/team/detail/kacper-sokol (K. Sokol); https://ixa2.si.ehu.eus/asoroa (A. Soroa) 0000-0003-3673-421X (J. M. Alonso-Moral); 0000-0001-5359-4111 (Z. Anthis); 0000-0002-9155-269X (R. Berlanga); 0000-0002-3677-672X (A. Catalá); 0000-0002-4771-441X (P. Cimiano); 0000-0001-6857-5810 (P. Flach); 0000-0002-9944-4108 0000-0002-9869-5896 (K. Sokol); 0000-0001-8573-2654 (A. Soroa)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). to either user preferences or user needs, but also adaptable to diferent communication channels, in the form of natural phenotropics multi-lingual human–machine interactions. Nontheless, most existing resources are developed ad-hoc for specific applications, usually considering only one or two modalities, being hard to combine, reuse and recycle in a human-centred and sustainable way. Afective XAI The extent to which XAI systems should be equipped with abilities to detect and express human emotions remains an open question. Some researchers have hypothesized that including an afective component might increase the predictability of systems and help users in reasoning about the causality of systems and predictions. The technical challenges for the systems developed within the afective computing spectrum are related to multimodal natural language processing such as sentiment analysis tools that use natural language processing and text analysis in addition to emotion detection from signals and modalities including gestures, posture, facial information, heart rate, electrodermal activity, voice, speech rate, pitch and intensity.
Interactive XAI Beyond regarding an explainee as a mere passive receiver of an (adapted) explanation,
previous research has proposed that explainees should have a more active role, being able to actively
co-shape the explanation in an interactive process. Rohlfing et al. [
The workshop received 18 submissions covering all the three main topics mentioned in the call for papers. All the submissions received three reviews. Out of these papers, 9 were selected for presentation at the workshop, yielding an acceptance rate of 50%.
Topics The call for papers mentioned the following topics as particularly relevant for the workshop: • Multimodal XAI – XAI for multi-modal data retrieval, collection, augmentation, generation and validation:
From data explainability to understanding and mitigating data bias – XAI for Human-Computer Interaction (HCI): From explanatory user interfaces to interactive and interpretable machine learning approaches with human-in-the-loop – Augmented reality for multi-modal XAI – XAI approaches leveraging application-specific domain knowledge: From concepts to large knowledge repositories (ontologies) and corpus – Design and validation of multi-modal explainers: From endowing explainable models with multi-modal explanation interfaces to measuring model explainability and evaluating quality of XAI systems – Quantifying XAI: From defining metrics and methodologies to assess the efectiveness of explanations in enhancing user understanding and trust – Large knowledge bases and graphs that can be used for multi-modal explanation generation – Large language models and their generative power for multi-modal XAI • Afective XAI – Proof-of-concepts and demonstrators of how to integrate efective and eficient XAI into real-world human decision-making processes – Ethical, Legal, Socio-Economic and Cultural (ELSEC) considerations in XAI: Examining ethical implications surrounding the use of high-risk AI applications, including potential biases and the responsible deployment of sustainable “green” AI in sensitive domains – Explainable afective computing in healthcare, psychology and physiology – Explainable afective computing in education, entertainment and gaming – Privacy, fairness and ethical considerations in afective computing and explainable AI applied in afective computing – Bias in afective computing and explainable AI applied in afective computing – Multimodal (textual, visual, vocal, physiological) emotion recognition systems – User environments for the design of systems to better detect and classify afect – Sentiment analysis and explainability – Social robots and explainability – Emotion aware recommender systems – Accuracy in emotion recognition and explainable AI applied in afective computing – Afecive XAI – Afective design – Machine learning using biometric data to classify biosignals – Virtual reality in afective computing – Human–Computer Interaction (HCI) and Human in the Loop (HITL) approaches in afective computing • Interactive XAI – Dialogue-based approaches to XAI – Use of multiple modalities in XAI systems – Approaches to dynamically adapt explainability in interaction with a user – XAI approaches that use a model of the partner to adapt explanations – Methods to measure and evaluate the understanding of the users of a model – Methods to measure and evaluate the ability to use models efectively in downstream tasks – Interactive methods by which a system and a user can negotiate what is to be explained – Modelling the social functions and aspects of an explanation – Methods to identify users’ information and explainability needs Program Committee We thank the following PC members for reviewing papers and helping to reach final decisions on acceptance: