In the context of the Internet of Things (IoT), intelligent systems (IS) are increasingly relying on Machine Learning (ML) techniques. Given the opaqueness of most ML techniques, however, humans have to rely on their intuition to fully understand the IS outcomes: helping them is the target of eXplainable Arti cial Intelligence (XAI). Current solutions { mostly too speci c, and simply aimed at making ML easier to interpret { cannot satisfy the needs of IoT, characterised by heterogeneous stimuli, devices, and data-types concurring in the composition of complex information structures. Moreover, Multi-Agent Systems (MAS) achievements and advancements are most often ignored, even when they could bring about key features like explainability and trustworthiness. Accordingly, in this paper we (i) elicit and discuss the most signi cant issues a ecting modern IS, and (ii) devise the main elements and related interconnections paving the way towards reconciling interpretable and explainable IS using MAS.
In the current decade, Internet of Things (IoT) systems, devices, and frameworks boomed, demanding contributions from both industry and academia. People's daily lives are getting entangled with uncountable cyber-physical devices capable of sensing, acting, and reasoning about the surrounding environment and who populates it. This leads to an intriguing set of socio-technical challenges for the Arti cial Intelligence (AI) researchers and practitioners. The complexity of the IoT systems is increasing at a fast pace, employing underlying AI techniques such as Machine Learning (ML) in the system core mechanisms to analyse, combine, and pro le heterogeneous sets of data. For instance, virtual assistants such as ? Copyright c 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)
Alexa, Cortana, Siri, or Google Home [
The bene ts of such AI-powered evolution are remarkable. Industries can now
reach a novel degree of automation, whereas customers can now enjoy a plethora
of new services mediated by their devices, as data and services now can fuel
unprecedented business opportunities and markets. However, such a transition
is unlikely to occur without costs. Besides ethical and sociological issues, the
current usage of AI is far from being socially acceptable. In particular, the recent
hype on ML, Deep Learning (DL), and other numeric AI methods { commonly
referred as \third AI spring" { has led to a situation where several decisions are
delegated to subsymbolic predictors out of human control and understanding|
as demonstrated by the many cases where they blatantly misbehaved [
Furthermore, as broadly acknowledged by many research communities, we argue that the development process of current intelligent systems is awed by the following issues: Lack of generalisation | Most tasks in AI require very speci c modelling, design, and development/training process. As a result, the integration, combination, and comparison of di erent { yet similar { methods in AI is either impossible or achieved through highly human-intense ad hoc (thus not scalable/extendable) system design.
Opaqueness | When numeric (data driven) methods are exploited, predictions come without an understandable motivation|or, more generally, without a model. Unfortunately, in most applications, data scientists only care about predictive performance and generalisation capability. However, the adoption of opaque methods or predictors reduces the scope of application of intelligent systems|possibly due to either practical or legal constraints, and, more concerning, to the lack of trust manifested by people and organisations.
Lack of automation (in the development process) | Despite AI is a tool
ultimately adopted to seek automation, the training process of most numeric
predictors is far from being automatic. The experience and the background
of the data scientist are still heavy discriminants for the overall predictive
performance. Methodologies and guidelines do not ensure any success in the
general case, and a lot of trials-and-errors are typically unavoidable.
Centralisation of both data and computation: it is often required or preferred
given the centralised nature of most algorithms or the legal constrains data
is subject to. Centralisation poses severe technical limitations to the way
data is managed, and to the design of computing systems.
Both the industry and the academia tend to tackle such problems individually,
without looking at the whole picture. As a result, most of research activities
focus on: (i) creating ad-hoc integration of AI sub-systems tailored on speci c
problems; (ii) developing techniques easing the interpretation of speci c numeric
predictors/predictions, exploiting results from the eXplainable AI (XAI) research
area [
In this paper we claim that Multi-Agent Systems (MAS) [
Contribution. This paper drafts a long-term research line supporting our claim.
In particular, we provide a i [
The reminder of this paper is organised as follows. Section 2 recalls the main research topics to be involved and combined in our research activity. Section 3 presents and discusses the aforementioned i modelling. Finally, Section 4 provides some remarks and concludes the paper. 2 2.1
eXplainable Arti cial Intellingence (XAI)
Modern intelligent systems (IS) often leverage on black-box predictive models
which are trained through ML or DL, or other numeric approaches. The
\blackbox" expression refers to models where knowledge is not explicitly represented
[
On the one hand, we borrow the de nition of \interpretation" from logic, where the word essentially describes the operation of binding objects to their actual meaning in some context. Thus, as far as numeric models are concerned, the goal of interpretability is to convey to humans the meaning hidden into the data and the mechanisms/decisions characterising the predictors.
On the other hand, we de ne \explanation" as the act making someone understand the information conveyed in a given discourse. It worth to highlight that the act of explaining is an activity involving at least two interacting parties, one explaining (explainer) and the other(s) willing to understand (explainee).
In the context of IS, the goal of explainability is to transfer to the receiver
(possibly humans) given information (e.g., awareness of the reasons leading the
system to act in a certain way) on a semantic level, aligning the State of Mind
(SoM) [
Such a distinction between interpretability and explainability is crucial since it shows how most XAI approaches proposed into the recent literature mostly focus on interpretability. Thus, while research into interpretable ML is widely recognised as important, a joint understanding of the concept of explainability still needs to evolve. Symbolic AI for Explainability. XMAS targets both the aspects characterizing understandability. However, di erently from other research lines branching from XAI, our vision poses a remarkable emphasis on explainability. In particular, in XMAS, we commit to symbolic AI as the main means for explainability.
By \symbolic AI" we mean the branch of AI focusing on symbolic knowledge
representation, automatic reasoning, constraint satisfaction, and logic
programming [
The reasons behind this commitment are threefold. Firstly, we let XMAS support the wide gamma of results, methods, algorithms, and toolkits developed under the umbrella of symbolic AI. Secondly, as further discussed in the following sections, we believe that the adoption of symbolic AI to be an enabling choice for the full exploitation of MAS. Finally, we argue that symbolic representations (e.g., the language of 1OL formulas), may act as a lingua franca for knowledge representation and exchange among heterogeneous IS.
In particular, the potential of logic-based models and their extensions is
mainly due to their declarativeness and explicit knowledge representation {
enabling knowledge sharing at an adequate level of abstraction { modularity, and
separation of concerns [
Symbolic knowledge extraction. The generality of symbolic approaches is
also due to the many research works recently pointing out that the
explanation capability of numeric predictors can be achieved via symbolic knowledge
extraction (SKE) [
SKE groups methods and techniques for extracting symbolic representations of the numeric knowledge buried in data and captured through ML during predictors training. Indeed, one of the main issues in symbolic AI is that human experts must often handcraft symbolic knowledge relying on their background and experience. This is not what happens in ML, where useful { yet hard to interpret { numeric knowledge is mined from data. Therefore, SKE can enable the exploitation of both symbolic and numeric AI without their respective shortcomings. In turn, XMAS aims at leveraging on the symbolic knowledge extracted from ML-powered predictors as a basis for providing explanations of its predictions and functioning.
Many SKE techniques have been proposed in the literature. Some of them
focus on speci c sorts of ML predictors, such as neural networks { and they are
therefore called \decompositional" {, whereas others are more general any may
virtually target any predictor|and they are thus called \pedagogical". Several
relevant contributions to the topic are outlined in surveys such as [
Multi-agent systems (MAS)
Multi-agent systems (MAS) represent an extensive research area placed at the
intersection between computer science, AI, and psychology, studying systems
composed by interactive, autonomous, and usually intelligent entities called agents
[
The agent abstraction has been described in many ways. However, most de nitions agree on the following traits. (i) Agents are entities operating into an environment possibly perceived through sensors and a ect through actuators. (ii) Agents are autonomous in the sense that have the capability of deciding on their own which actions are to be performed in order to achieve (or maintain) the goals they have been provided with|which in most cases are explicitly represented through some symbolic language. (iii) Agents are social, meaning that they can (and usually need to) interact (e.g., communicate, cooperate, and/or compete) with each other or with human users in their attempt to achieve/maintain their goals. (iv) Agents have a mind consisting of a belief base (BB)|storing symbolic data representing the (possibly wrong, steady, or biased) information each agent believes to know about the environment or other agents. The content of a given agent's BB is a ected by its perceptions and interactions, and it may in uence its actions. The general notion of agent is so wide that both software entities and human beings may t it. Such formal laxity is deliberate and useful. In fact, it allows human-machine and machine-machine interactions to be captured at the same level of abstraction and to be described through a coherent framework.
Dialogical argumentation. A central role in agent sociality is played by
argumentation [
Dialogical argumentation, in particular, is the activity performed by a number
of agents dynamically discussing about some topic they are concerned with from
di erent perspectives, in an attempt to agree on some shared truth about that
topic. Thus, dialogical argumentation accounts for how arguments and
counterarguments are generated and evaluated, how the agents interact { i.e. what kinds
of dialogical moves they can make {, and how agents can retract arguments,
update beliefs, etc. Usually, it is set against monological argumentation [
The i
modelling language Modelling a domain is a human-intensive, non-automated task. The domain IoT-powered IS is characterised, by itself, by complex requirements, theories, and methods converging from several scienti c elds. Also, the XMAS vision intertwines results from disparate research areas, which historically has been kept mostly disjoint.
To generate a clear and structured understanding of our vision, we adopt
the Goal-Oriented Requirement Engineering (GORE) approach [
Here, an i model consists of a graph whose vertices are elements of four kinds: Goals (ranged by Gi), Soft Goals (ranged by SGj), Tasks (ranged by Tk), Resources (ranged by Rl); edges (a.k.a. links) represent relations of various sorts among the aforementioned elements.
Figure 1 depicts how elements and links are graphically represented. A more complete description of the i graphical formalism can be found on the i wiki web page3. Informally, goals are desired properties or objectives whose achievement can either be satis ed or not, in a discrete fashion. Conversely, soft goals are (non-necessarily measurable) desirable properties or objectives which can be 3 http://istar.rwth-aachen.de/tiki-index.php?page=iStarQuickGuide satis ed qualitatively or up to some degree. Tasks represent activities to be performed as an attempt to satisfy (resp. positively a ect) one or more goals (resp. soft goals). Resources represent entities to be produced or consumed by tasks, and whose availability may favor or hinder the satisfaction of goals.
As far as links are concerned, soft goals are usually connected to each other via \contribution links", which specify their contribution in ful lling the needs| e.g., positive: Some+, or negative: Some-; goals are connected via \means-end" arrows; tasks are connected via \decomposition" links. Other sorts of links mostly de ne generic \dependencies". 3
Figure 2 graphically represents the i modelling of the XMAS vision, and highlights its main aspects. The representation aims at providing an intuitive graphical assessment of the various elements and their interconnections concurring in advancing state of the art of XAI in IS. We argue that having a overall mapping of XMAS requirements, objectives, as well as of their mutual interdependencies, can also facilitate the assessment, presentation, design, and implementation of a coherent research activity aimed at supporting our vision.
About the main soft goals. Our vision stems from the recognition that the success of IoT-powered IS is due to the general, increasing demand of analytical and predictive performance { corresponding to the SG0 weak goal in Figure 2 { transversally pervading the productive fabric of most developed societies. However, we also acknowledge several other desiderata { mostly targeting the issues highlighted in Section 1, and corresponding to as many weak goals into our i model { which are required to some extent by modern AI, mostly due to the increasingly pervasive adoption of AI techniques.
For instance: { IS need to be understandable (SG1), meaning both interpretable and (above all) explainable, in the sense outlined in Section 2. { Understandability, in turn, is only one of the aspects concurring in making modern cyber-physical systems perceived as trustworthy (SG2). Other aspects are important as well, like, egg,, having some degree of control on the behaviour of autonomous agents, and on the data and knowledge they are relying upon. In particular, when it comes to data and software, integrity and tampering-resistance are properties of paramount importance, strongly a ecting the trustworthiness of IS. { The IoT landscape also stresses the need to widen researchers' focus towards the \system of systems" dimension. The vertical specialisation (G1) of MLbased solution w.r.t. speci c tasks is not the only concern any longer. Indeed, the horizontal integration (SG3) of heterogeneous systems is important as well. There, we expect IS to acquire the capability of dynamically and autonomously integrate, complement, and extend their respective knowledge, in a similar way to what human beings do when talking to each other. { At the same time, the need for a higher degree of automation (SG4) in IS development is impelling, as the current bottleneck in the development of IS is due to the deep dependence of the process on human intuition. { Finally, in spite of the many legal and ethical constraints a ecting data and their usage, IoT-powered IS eventually need to overcome the current tendency to data centralization (SG5), as it imposes severe limitations over their e ectiveness, e ciency, and adoption.
The XMAS vision pursues the goal of tackling { or at least improving { all such issues, and, ultimately, of making intelligent systems more trustworthy.
To do so, we analysed the current trends in this area and understood that the combination of numeric methods with more classical symbolic AI approaches, possibly mediated by MAS, may provide bene cial e ects at several levels. On predictive capability. However, before describing how and why our proposal may provide an advantage, we need to brie y recall the most relevant aspects in IS development, as well as their mutual interdependencies.
In their quest for higher predictive performances (SG0), IS designers simply apply numeric ML (T1) methods to the data (R1) they have collected through IoT devices. Such a process is far from being automatic, as it requires the experience and the trial-and-error work of well-prepared data-scientists. It is aimed at the creation of predictors (R2), which are mathematical models capable of providing numerical predictions (G2) in situations analogous to the ones described by the data they have been trained with.
Nevertheless, even after the deployment of the IS, its vertical specialisation (G1) on the speci c problem described by data remains an ever-lasting process, as new data keeps being produced/captured by the systems in production. Most researchers or data scientists prefer to focus on such sorts of tasks as they are very valuable and numerically quanti able|in terms of predictive performance. On understandability. As far as the soft goal of understandability (SG1) is concerned, we argue that it can be tackled either by easing predictors/data interpretability (SG6) or by providing means for their explanability (SG7).
When it comes to let humans interpret the predictions provided by MLpowered predictors, the most common way to do so is to employ the most adequate technique for the problem at hand, possibly mediated by some analytical or visualisation toolkit (T2) and let the human intuition of experts (T3) do the magic. Thus, despite being very e ective in speci c cases, such an approach lacks generality and hinders automation.
Conversely, when it comes to providing explanations to the users, the XMAS vision recognises the prominent role of interaction in letting knowledge be transferred from IS to humans (and possibly vice versa). In particular, we envision a scenario where intelligent agents exchange symbolic knowledge with humans through various channels, interfaces, and languages|i.e., we envision several possible means for Human-Computer Interaction (HCI, G3).
As a rst step in this direction, XMAS leverages on symbolic knowledge extraction (SKE, T4) as a means for attaining logic-based, symbolic rules and facts (R3) out of numeric predictors (R2) and raw data (R1).
The next step consists of employing such symbolic rules and facts as knowledge bases for cognitive, distributed agents (G4). More precisely, we state a one-to-one correspondence among numeric predictors and the agents to be deployed. Thus, we say that each predictor is wrapped by an agent.
Such a wrapping is an enabling step in several directions. For instance, we expect that by employing dialogical argumentation (T5), cognitive agents may become able to compare and complement the knowledge they have extracted from numeric predictors.
The capability of knowledge revision is particularly interesting, especially if one of the agents involved in the argumentation process is a human being. Indeed, if adequately constrained, dialogical argumentation may act as a means for providing interactive explanations (SG7) to the users, concerning the symbolic knowledge wrapped by agents.
In particular, such explanations can be even more e ective if the interaction among users and software is mediated by some textual, vocal, or avatar-based user interface aimed at easing human-computer interaction (G3). On the bene ts of argumentation. However, the adoption of extracted symbolic knowledge and dialogic argumentation is not merely aimed at supporting the explanations.
Instead, it may also positively a ect what we call the horizontal integration (SG3) of heterogeneous IS attained by di erent { yet related { data. This, in turn, enables the integration and exploitation of di erent perspectives on the information carried by data|which implies that di erent points of view can be merged to more precise predictions, as well as alternative predictive scenarios can be produced. Horizontal integration could thus make (more) valuable the many degrees of freedom and the inherent randomness characterising the processes of data retrieval, selection, engineering, partitioning, and analysis.
At the same time, the agents' capability of mutually updating and correcting their belief bases (G5) may pave the way towards the development of IS where predictions can be attained without relying on the centralization of data on a speci c computational facility, nor on its transfer outside the organizational domain it belongs to. In other words, XMAS enables the decentralization of knowledge and computation (SG5). Despite data being usually subject to strict regulations limiting { among the others { its transfer, this is possible because aggregated { thus anonymous { data, such as the high-level rules extracted from data or predictors, are subject to less limiting regulations.
Similarly, argumentation may be conceived as a means for supporting a higher degree of automation (SG4) in the development of IS. In particular, protocols could be de ned, letting new agents query other agents for symbolic knowledge they do not have. By doing so, cognitive agents can learn predictive or explanatory rules autonomously, even without needing direct access to the data. On trustworthiness. If the XMAS vision will be accomplished, the e ect of a handcrafted malicious (or buggy) agent, deliberately or mistakenly attempting to inject wrong knowledge into an agent society could be nefarious|and, by assuming an open and distributed society such as the IoT, this contingency cannot be excluded. This is another critical issue preventing people from fully trusting IS nowadays.
To mitigate such concerns, DLT (R4) could be exploited to prevent the
tampering of data or software (G6), or, to keep track of agents' reputation|
assuming some reputation-enforcing protocol (T6) [
In this paper we point out a number of issues a ecting modern IoT, and in general distributed IS whose intelligence leverages on ML. In particular, focusing on the data analytics layer of most IoT-based applications, we argue that a number of issues are still far from being completely closed. For instance, we discuss why most ML-powered IS lack transparency, automation (in the development process), and decentralisation (of both data and computation).
Elaborating on such open issues, we discuss a research line { called eXplainability through Multi-Agent Systems (XMAS) { aimed at addressing them altogether in a coherent and e ective way. In the XMAS vision, we plan to integrate a number of contributions from the symbolic AI, MAS, and XAI research areas.
Accordingly, in this paper we provide an overview of the state of the art of the aforementioned areas, shortly discuss their main achievement and limitations in the XMAS perspective, and present a formal model of the XMAS vision using the i modelling language.