-In the era of digital revolution, individual lives are going to cross and interconnect ubiquitous online domains and offline reality based on smart technologies-discovering, storing, processing, learning, analysing, and predicting from huge amounts of environment-collected data. Sub-symbolic techniques, such as deep learning, play a key role there, yet they are often built as black boxes, which are not inspectable, interpretable, explainable. New research efforts towards explainable artificial intelligence (XAI) are trying to address those issues, with the final purpose of building understandable, accountable, and trustable AI systems-still, seemingly with a long way to go. Generally speaking, while we fully understand and appreciate the power of sub-symbolic approaches, we believe that symbolic approaches to machine intelligence, once properly combined with sub-symbolic ones, have a critical role to play in order to achieve key properties of XAI such as observability, interpretability, explainability, accountability, and trustability. In this paper we describe an example of integration of symbolic and sub-symbolic techniques. First, we sketch a general framework where symbolic and sub-symbolic approaches could fruitfully combine to produce intelligent behaviour in AI applications. Then, we focus in particular on the goal of building a narrative explanation for ML predictors: to this end, we exploit the logical knowledge obtained translating decision tree predictors into logical programs. Index Terms-XAI, logic programming, machine learning, symbolic vs. sub-symbolic
I. INTRODUCTION
Artificial intelligence (AI), machine learning (ML), and
deep learning (DL) are nowadays intertwined with a growing
number of aspects of people’s every day life [
For instance, banks and insurance companies have adopted ML and statistical methods since decades, in order to decide whether or not to grant a loan to a given customer, or to estimate the most profitable insurance plan for her. Similarly, ML has been employed in order to help doctors with their diagnoses, provided that a set of symptoms has been properly identified for a given patient; whereas statistical and probabilistic inference have been employed to test drugs, in order to prove them effective or safe. Furthermore, virtually any person, as a consumer of services and goods, lets a number of ML-trained agents decide or suggest what to buy, like, or read—as any consumer is likely to be profiled by most of the companies and organisations he/she has interacted.
In spite of the large adoption, intelligent agents whose behaviour is the result of automatic synthesis / learning procedures are difficult to trust for most people—in particular when people are not expert in the fields of computer or data sciences, AI, statistics. This is especially true for agents leveraging on machine or deep learning based techniques, often producing models whose internal behaviour is opaque and hard to explain for their developers too.
There, agents often tend to accumulate their knowledge into black-box predictive models which are trained through ML or DL. Broadly speaking, the “black-box” expression is used to refer to models where knowledge is not explicitly represented – such as in neural networks, support vector machines, or Hidden Markov Chains –, and it is therefore difficult, for humans, to understand what a black-box actually knows, or what leads to a particular decision.
Such difficulty in understanding black-boxes content and
functioning is what prevents people from fully trusting –
and thus accepting – them. In several contexts, such as the
medical or financial ones, it is not sufficient for intelligent
agents to output bare decisions, since, for instance, ethical
and legal issues may arise. An explanation for each decision
is therefore often desirable, preferable, or even required. For
instance, applications dealing with personal data need to face
the challenges of achieving valid consent for data use and
protecting confidentiality, and addressing threats to privacy,
data protection, and copyright. Those issues are particularly
challenging in critical application scenarios such as
healthcare, often involving the use of image (i.e., identifiable) data
from children. While issues of data ownership, data security,
and data access are important, other ethical issues may arise:
since the diagnostic accuracy and value of the result is
determined by the amount and quality of data used in model
training, the first potential concern is to avoid algorithmic
bias, which may lead to social discrimination and result in
inequitable access to healthcare, just related to the provenience
of the collected data [
Furthermore, it may happen that black-boxes silently learn
something wrong (e.g., Google image recognition software
that classified black people as gorillas [
To tackle such trust issues, the eXplainable Artificial
Intelligence (XAI) research field has recently emerged, and
a comprehensive research road map has been proposed by
DARPA [
The main idea behind XAI is to employ explanators [
In this paper we focus on the specific problem of building a
narrative explanation of ML techniques—thus positioning our
contribution into the specific Narrative Generation DARPA
category [
To this end, we propose an automatic procedure aimed at translating a ML predictor – here in particular we consider the case of decision trees (DT) – into logical knowledge. We argue that, when the source DT has been trained over a set of real data in order to produce a predictor, the corresponding logic program may be employed to produce a narrative explanation for any given prediction.
Despite being mostly focused on DT, our proposal represent
a first step towards a more general approach. In fact, DT have
been proposed as a general means for explaining the behaviour
of virtually any black-box model [
Accordingly, the reminder of this paper is organised as follows. Section II briefly recalls the ML concepts and terminology used in the paper as well as the main research efforts in the field. Then Section III introduces our vision of a framework for the integration of symbolic and sub-symbolic techniques. Finally, Section IV discusses early experiments alongside the prototype implementation.
on opaque models, thus hiding their internal logic to the user. This hinders explainability, and represents both a practical and an ethical issue for ML. As a result, many research approaches in the XAI field aim at overcoming that crucial weakness, sometimes at the cost of trading off accuracy against interpretability. So, we first (Subsection II-B) summarise the state of the art as well as the goal of XAI, then (Subsection II-A) introduce some background notions to define the terminology adopted.
A. Background
nition, financial and medical decision support systems – can
be reduced to supervised ML – which can be further grouped
in terms of either classification or regression problems [
In those cases, a learning algorithm is commonly exploited to estimate the specific nature and shape of an unknown prediction function (or predictor) p∗ : X → Y, mapping each input vector x from a given input space X into a prediction from a given output space Y. To do so, the learning algorithm takes into account a number N of examples in the form (xi, yi) such that xi ∈ X ⊂ X , yi ∈ Y ⊂ Y, and |X| ≡ |Y | ≡ N . There, each xi represents an instance of the input data for which the expected output value yi is known or has already been estimated. Such sorts of ML problems are said to be “supervised” because the expected targets Y are available, whereas they are said to be “regression” problems if Y consists of continuous or numerable values, or “classification” problems if Y consists of categorical values.
The learning algorithm usually assumes p∗ ∈ P, for a given family P of predictors—meaning that the unknown prediction function exists, and it is from P. The algorithm then trains a predictor pˆ ∈ P such that the value of a given loss function λ : Y × Y → R – computing the discrepancy among predicted and expected outputs – is minimal or reasonably low—i.e.: pˆ = argmin nPiN=1 λ(yi, p(xi))o.
p∈P
Depending on the predictor family P of choice, the nature
of the learning algorithm and the admissible shapes of pˆ may
vary dramatically, as well as the their interpretability. Even if
the interpretability of predictor families is not a well-defined
feature, most authors agree on the fact that some predictor
families are more interpretable than others [
Under such hypotheses, a DT predictor pT ∈ Pdt assumes explanation to guarantee some properties, e.g., soundness,
a binary tree T exists such that each node is either completeness, and compactness [
• a leaf, carrying and representing a prediction, i.e. and This is why our proposal aims at integrating sub-symbolic
assignment for y, approaches with symbolic ones. To this end, DT can be
• an internal node, carrying and representing a decision, i.e. exploited as an effective bridge between the symbolic and
a formula in the form (nj ≤ c)—where c is a constant sub-symbolic realms. In fact, DT can be easily (i) built from
threshold chosen by the learning algorithm. an existing sub-symbolic predictor, and (ii) translated into
symbolic knowledge – as it is shown in the reminder of this
Each node ν inherits a partition Xν ⊆ X of the original input paper – thanks to their rule-based nature.
data, from its parent. Since the root node ν0 has no parent, it Decision trees are an interpretable family of predictors that
is assigned to the whole set of input data—i.e. Xν0 ≡ X. The have been proposed as a global means for explaining other,
decision carried by each internal node splits its Xν into two less interpretable, sorts of black-box predictors [
Since the adoption of interpretable predictors usually comes them with symbolic approaches, which can play a key role in
at cost of a lower potential in terms of predictive performance, enhancing the interpretability and explainability of the system.
explanations are the newly preferred way for providing under- In this paper we focus on such a matter.
standable predictions without necessarily sacrificing accuracy. We believe that a logical representation of DT may be
The idea, and the main goal of XAI is to create intelligible interesting and enabling for further research directions. For
and understandable explanations for uninterpretable predictors instance, as far as explainability is concerned, we show how
without replacing or modifying them. Thus explanations are logic-translated DT can be used to both navigate the
knowlbuilt through a number of heterogeneous techniques, broadly edge stored within the corresponding predictors – thus acting
referred to as explanators [
The state of the art for explainability currently recognises map DT onto logical clauses, since DT are finite and with a
two main sorts of explanators, namely, either local or global. limited expressivity (if / else conditions).
While local explanators attempt to provide an explanation for
each particular prediction of a given predictor p, the global III. VISION
ones attempt to provide an explanation for the predictor p as Many approaches to ML nowadays are increasingly
foa whole. In other words, local explanators provide an answer cussing on sub-symbolic approaches – such as deep learning
to the question “why does p predict y for the input x?” – with neural networks [
To this end, since logic-based approaches already have
a well-understood role in building intelligent (multi-agent)
systems [
The potential of logic-based models and their extensions is
first of all related to their declarativeness as well as to explicit
knowledge representation, enabling knowledge sharing at the
most adequate level of abstraction, while supporting
modularity and separation of concerns [
Another relevant point is that LP has been already proven to
work well both as a knowledge representation language and as
an inference platform for rational agents [
Accordingly to this vision, here we propose an integrated framework of hybrid reasoning – where symbolic and subsymbolic techniques fruitfully combine to produce intelligent behaviour.
Indeed, looking in depth at pervasive socio-technical systems, it turns out that agents (either human or software) effortlessly undertake a complex decision making process in almost all situations, which seamlessly integrates perceptions (and actions) at two different scales—the macro and the micro: • at the macro scale, by considering the knowledge of the global system, rules of general validity and concerning the most likely situation; • at the micro scale, we modulate such decision by considering all the contingencies arising during the precise situation – such as, for instance, a last minute inconvenient, etc. As a consequence, we adapt the original plan to the local perceptions we gather while enacting it.
In order to better illustrate the above remarks, one may
consider as a concrete example the case of a disease diagnosis
in a hospital, where the notions of micro and macro scale w.r.t.
to the nature of algorithms and techniques can be declined as
follows:
• at the macro level, the main concerns regard a mid/long
term horizon and focus the issue of analysis of
highdimensional and multimodal biomedical data train
algorithms to recognize cancerous tissue at a level comparable
to trained physicians—there including, for instance,
representation and recognition of patterns and sequences in
the input data. With such a sort of goals to pursue, it
is not surprising that most IT tools supporting decision
making are based on sub-symbolic approaches such as
deep learning, Bayesian networks, machine vision, latent
Dirichlet analysis, and in general any kind of statistical
approach to ML [
etc.) which represent the commonsense knowledge necessary to inhabit the environment and specific rules, with a validity bound in space and time (school hours and days, open-air market hours and days, unpredictable events such as incoming emergency vehicles the need to gather at an evacuation assembly point), which represent the contextual or expert knowledge necessary to deal with transient, unforeseen, and unpredictable situations.
That is why in the framework envisioned here we plan to combine sub-symbolic techniques with symbolic ones (LP in particular): sub-symbolic techniques are exploited for training the system and learn new rules (commonsense knowledge), rules are translated into logical knowledge (contextual / expert knowledge), and the two approaches interact and interleave to share knowledge and learn from each other in a coherent framework.
The framework architecture, depicted in Fig. 1, shows the embodiment of the vision discussed above: sensor data and dataset are translated into the logic knowledge base. In particular the Machine Learning Interface allows for the interaction of different kinds of ML algorithms with the framework: a standard interface is proposed in order to combine the specific features of each algorithm in a coherent manner. ML to Prolog is the core of the translation into logical knowledge, while the Prolog to ML returns insights of the logical KB to the ML predictor—for instance, new inferred rules, or rules learned by a specific situation. The blocks on the left (Knowledge Base, Demonstration) reflect the standard architecture of a Prolog engine. Overall, the framework looks general enough to account for the variety of ML techniques and algorithms, and also to ensure the consistency between symbolic and sub-symbolic approaches. Finally, the block Prolog to ML currently expresses our vision, and is obviously subject of future research.
construction of a narrative explanation of the prediction generated exploiting the ML technique, thus achieving interpretability and making a step towards explainability.
With respect to Fig. 1, we experiment the predictor
translation into logical rules, provided by the ML to Prolog. The
experimental results refer to the case in which the predictor
corresponds to a decision tree or to the corresponding crisp
rules [
For a concrete example, let us consider the “Acute in- ✞
flammations data set”1 [
Input parameters collect all the patient symptoms, each instance represents a potential patient. The data was created by a medical expert as a data set to test the expert system, which performs the presumptive diagnosis of two diseases of urinary system. The dataset considered is summarised in TABLE I and TABLE II.
Starting from the general form Head ← Body for a logical clause, a predicate in the Head is generated for the decision of the predictor—in the example, the diagnosis predicate.
Inside the predicate, a term for each input/output attribute is instantiated with the value of the decision tree (leaf).
In our example, the following predicate is generated: diagnosis(temperatureOfPatient(T), occurrenceOfNausea(N), lumbarPain(L), urinePushing(U), micturitionPains(M), burningOfUrethra(BU), nephritis(no), confidence(1.00)) :- T =< 37.95.
✆ ✆ ✝ ✡ representing the fact that if the temperature of patient is lesser inferring hidden knowledge in the rules. It is worth noticing or equal of 37.9, it is unlikely the patient presents nephritis that similar results (emphasising the relations between decision of renal pelvis; the answer contains a degree of confidence output) can be obtained manipulating the dataset a priori— based on the case of the dataset that confirm the rule—in the i.e. before the ML algorithm training (a common operation case 1.00 stands that all the patients in the dataset that have but not always applicable). The manipulation of the above a temperature lower that 37.9 do not present the disease. dataset, for instance, can build a unique decision output
To improve readability, the rule above could be written as Result that combines the two different diseases and their ✞diagnosis(temperatureOfPatient(T), _, _, _, _, _, nephritis symptoms. In such a case the dataset is enriched with the (no), confidence(1.00)) :- T =< 37.95. Result attribute containing the complete diagnosis, i.e., it can ✡✝ ✆assume the values Healthy, Inflammation, Nephritis, Both. The by omitting the undefined variables, i.e., highlighting the input corresponding decision tree and LP knowledge is depicted in attribute that are effectively to be considered as influencer. Fig. 3.
Fig. 2 (left) depicts the whole picture: the decision trees d) Interpretable narrative explanation: LP makes it posgenerated as output of the example dataset when we run sible to generate a narration for each answer of the predictor. the basic classification tree algorithm2 and the corresponding The inference Prolog tree becomes inspectable, tracking the translation into LP rules. With respect to Fig. 1, the decision path for obtaining the answer. For instance, w.r.t. the KB of trees are the output of the Machine Learning Interface block Fig. 3 – including all diseases –, the diagnosis in the case of and become the input for the ML to Prolog block. the following symptoms:
Fig. 2 represents experiments of running the ML algorithm ✞
with no manipulation of the dataset: so, since the ML algo- diagnosis(
rithm allows only one decision output to be considered for tleummpbearraPtauirne(yOefsPa)t,ieunrti(n3e6P.u5s)h,inogc(cnuor)r,enceOfNausea(yes),
producing the corresponding decision tree, the information and micturitionPains(yes), burningOfUrethra(yes), _, _).
the related knowledge is fragmented into two different trees – ✡✝
the first obtained running the algorithm with decision output would produce the corresponding narration:
nephritis and the second with decision output inflammation of ✞
urinary bladder. By running the ML to Prolog block of Fig. 1 The diagnosis is healthy, with a full confidence because
we translate the two DT in LP rules as depicted in Fig. 2 the patient has no fever.
(right). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
a) Interpretabilty: The LP program provides an inter- fIonllpoawritnigcuplaatrh:the solution has been built across the
pretable explanation of virtually any predictor. At a glance, Solution: result(healthy) with confidence(1.00).
the user can identify which attributes are meaningful and con- For the proof, the following clauses are considered:
sidered for response and which are not. In case of nephritis, the [
c) Relations between outputs: As emphasised by Fig. 2, relations between outputs are lost, and possible links between the diseases are not clearly highlighted having two different decision trees. Instead, once obtained a LP representation, it is easy to run simple queries on it in order to get much more information with respect to the two different decision tree.
For instance, we can learn that in case of fever (temperature of patient > 37.95) not presenting nephritis (i.e. no lumbar pain detected), the only case in which inflammation of urinary bladder is present is when urine pushing is detected in absence of symptoms of micturition pains. With the logical representation, relations between output can be recovered by
e) Exploitation of LP extension / abduction on the KB: Moreover, we believe that exploiting abduction techniques we could pave the way to hypothetical reasoning with incomplete knowledge, i.e., learning new possible hypotheses that can be assumed to hold, provided that they are consistent with the given knowledge base. The idea, to be explored in future research, is to provide the most likely solution given a set of evidence. The conclusion would leave a degree of uncertainty while highlighting a plausible answer based on the collected information. In the healthcare field, for instance, it could be represented by having the collection of symptoms (although incomplete) and finding the most likely disease for them.
translator and SciKit-Learn CART [
Output Decision: Inflammation of urinary bladder {yes, no} Output Decision:
Nephritis of renal pelvis origin {yes, no}
V. CONCLUSION tive policing. Nevertheless, concerns about the intentional and unintentional negative consequences of AI systems are
AI systems nowadays synthesise large amounts of data, legitimate, as well as ethical and legal concerns, mostly related learning from experience and making predictions with the to darkness and opaqueness of AI decision algorithm. For that goal of taking autonomous decisions—applications range from reason, recent work on interpretability in machine learning and clinical decision support to autonomous driving and predic✞ diagnosis(_, _, _, urinePushing(no), _, _, inflammation(no),
confidence(1.00)). diagnosis(_, _, lumbarPain(yes), urinePushing(yes),
micturitionPains(no), _, inflammation(no), confidence(1.00)). diagnosis(_, _, lumbarPain(no), urinePushing(yes), micturitionPains
(no), _, inflammation(yes), confidence(1.00)). diagnosis(_, _, _, urinePushing(yes), micturitionPains(yes), _, inflammation, confidence(1.00). diagnosis(temperatureOfPatient(T), _, lumbarPain(no), _,
_, _, result(healthy), confidence(1.00)) :- T > 37.95. diagnosis(temperatureOfPatient(T), _, lumbarPain(yes), _, micturitionPains(no), _, result(nephritis), confidence(1.00)) :- T > 37.95. diagnosis(temperatureOfPatient(T), _, lumbarPain(yes), _, micturitionPains(yes), _, result(both), confidence(0.66)) :- T > 37.95. ✞ ✝ ✡ ✞ ✝ ✡ ✝ ✡ ✝ ✡ ✆ ✆ ✆ ✆ ✆ ✆
criteria used to make decisions.
In this paper we focus on building a narrative explanation of the machine learning techniques: we first translate a ML predictor into logical knowledge, then inspect the proof tree leading to a solution. The narration is built tracking the path (i.e., the rules) that leads from the query to the answer.
Along this line, we foresee a broader vision that involves the design of a consistent framework where symbolic and sub-symbolic techniques are fruitfully combined to produce intelligent behaviour in AI applications while exploiting the benefits of each approach—like, in the case of symbolic ones, interpretability, observability, explainability, and accountability.
The results presented here represent just a preliminary exploration of the potential benefits of merging symbolic and sub-symbolic approaches—where, of course, many critical issues are still unexplored and will be subject of future work. However, despite its simplicity, the case study already allows us to point out the feasibility and the potential benefits of the exploitation of symbolic techniques towards XAI.