-
Arti cial Intelligence (AI) and Machine Learning (ML) techniques in
particular have had tremendous success in various tasks including medical diagnosis,
credit card fraud detection, or face recognition [
11
]. These systems, however, are
often opaque and usually do not provide human-understandable explanations
for their predictions [
23
]. This situation is problematic because it can adversely
a ect the understanding, trust, and management of ML algorithms [
23
]. While
not every (benign) algorithmic decision needs to be explained in detail,
explainability is necessary when dealing with incomplete problem statements including
aspects of safety, ethics, or trade-o s [
18
]. Additionally, legal considerations of
AI accountability add to the relevance of explainable decision systems [
19
].
Copyright c 2019 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
The umbrella term Explainable Arti cial Intelligence (XAI) is often used in
academia to refer to a variety of approaches attempting to make ML methods
explainable, transparent, interpretable, or comprehensible. Due to its relevance
a plethora of research on XAI exists, including literature reviews of popular
methods and techniques (see [
2
] or [
22
] for example). However, many of those
approaches rely on a purely technical analysis of the black-box ML models. For such
approaches Cherkassky and Dhar [
14
] argue that model explainability cannot be
achieved. The authors further stipulate that explainability is highly dependent
on the usage of domain knowledge and not data analysis alone. This idea has
been adapted more recently by di erent authors arguing that the incorporation
of Semantic Web Technologies might be a key to achieve truly explainable
AIsystems [
26, 27
]. Since existing surveys on XAI have not explored this promising
avenue of research in detail, we provide a literature-based overview of the
usage of Semantic Web Technologies alongside ML methods in order to facilitate
explainability. Speci cally, we focus on addressing three research questions:
1. What combinations of Semantic Web Technologies and ML have been
proposed to enhance model explainability?
2. Which domains of applications and tasks are especially important to this
research eld?
3. How are model explanations evaluated and presented to the user?
The remainder of this paper is organized as follows. Section 2 provides
relevant background information pertaining to explainability of ML systems.
Subsequently, Section 3 brie y describes the research design before presenting the
main ndings of this research. Based on these insights, implications for future
research are presented in Section 4. Finally, Section 5 concludes this research.
2
Background and Scope of the Literature Review
Explainability of Arti cial Intelligence is not a new stream of inquiry. Mueller et
al. [
39
] analyzed the temporal development of XAI and showed that the topic has
been intensively studied from the 1970s to the early 1990s within the context of
Expert and Tutoring Systems. In the following two decades, only little research
has been produced in the eld. Recently, however, there has been a resurgence
of the topic due to the interest in Machine Learning and Deep Learning [
39
].
Despite recent frequent publications on the topic of XAI there is no agreement
upon a de nition of explainability [
34
]. For the purpose of this survey, we follow
Adadi and Berrada [
2
] in di erentiating interpretable systems which allow users
to study the (mathematical) mapping from inputs to outputs from explainable
systems which provide understanding of the system's work logic. In this context,
Doran et al. [
17
] postulate that truly explainable systems need to incorporate
elements of reasoning which make use of knowledge bases in order to create
human-understandablable, yet unbiased explanations. Furthermore, it is worth
mentioning that interpretability or explainability not only depends on a speci c
model but also the knowledge and skills of its users [
24
].
Within the domain of ML a number of surveys address the topic of
explainability and interpretability. For example, Biran and Cotton [
10
] review
algorithmic and mathematical methods of interpretable ML models, Abdul et al. [
1
] focus
on explanations from a human-centered perspective, and Adadi and Berrada [
2
]
provide a holistic survey which also covers aspects of evaluation and perception.
However, these studies often do not touch upon how tools such as Semantic
Web Technologies might foster ML system explainability. In contrast, within
the related eld of Data Mining and Knowledge Discovery the interpretation of
data patterns via Semantic Web and Linked Open Data has been described in a
detailed survey by Ristoski and Paulheim [
45
]. While Data Mining, Knowledge
Discovery, and ML certainly overlap in some areas, a clear overview of the
combination of Semantic Technologies and Machine Learning is still missing. In this
context it is worth mentioning that the scope of this review is on classical ML
techniques as opposed to elds such as Inductive Logic Programming (ILP) [
40
].
ILP combines ideas from ML (learning from positive and negative examples)
with logical programming in order to derive a set of interpretable logical rules.
The interested reader can nd a summary of how ontologies can be used in the
ILP framework in [
35
]. While some researchers see ILP as a subcategory of ML
(e.g. [
50
]), we follow Kazmi et al. [
30
] in di erentiating the two elds and focus
on more classical ML while touching upon ILP only brie y.
3
Semantic Web Technologies
In this section we brie y lay out the research design of this survey before
summarizing the insights of the conducted analysis. To answer the posed research
questions we carried out an extensive literature review [
58
] by searching
major academic databases including ACM Digital Library, SCOPUS, and
peerreviewed pre-prints on arXiv. The latter has been incorporated because XAI is a
dynamically evolving eld with a number of contributions stemming from
ongoing work. We conducted a search based on keywords relating to three categories:
Machine Learning, Semantic Web Technologies, and explainability.1 The
resulting list of papers was evaluated for relevance based on their abstracts and the
remaining papers based on their full content. A forward and backward search
[
59
] has been conducted to complement the list of relevant research articles.
To shed light on the rst research question, we categorized the relevant
models based on their usage of ML and Semantic Web Technologies. Speci cally,
we distinguished ML approaches along their learning rules (supervised,
unsupervised, reinforcement learning) [
48
] and characterized the used Semantic Web
Technologies by their semantic expressiveness. In doing so, we focused on the
actually exploited knowledge rather than the underlying representation. For
example, if a system incorporates an ontology but exclusively makes use of
taxo1 Search strings included but were not limited to: "machine learning" OR "deep
learning" OR "data mining"; "explanation*" OR "interpret*" OR "transparen*";
"Semantic Web" OR "ontolog*" OR "background knowledge" OR "knowledge graph*"
nomical knowledge, it is categorized as a taxonomy. We followed Sarker et al.
[
47
] in di erentiating knowledge graphs from ontologies insofar that the former
are usually a set of triples most often expressed using the Resource Description
Framework (RDF) while the latter additionally posses type logics and are
regularly expressed using Web Ontology Language (OWL). We addressed the second
research question by observing the application domains and tasks of the
analyzed systems. We provide answers to the third research question by describing
in what form explanations are given to the user and how their quality is assessed.
3.1
Combining Semantic Web Technologies with Machine Learning
The results of categorizing the relevant literature along the dimensions laid out
before are presented in Table 1. From a general point of view, one can
observe that Semantic Web Technologies are used primarily to make two types of
ML models explainable: supervised classi cation tasks using Neural Networks
and unsupervised embedding tasks. The Semantic Web Technologies utilized
alongside Neural Networks are quite diverse, while embedding methods usually
incorporate knowledge graphs. Further, systems which attempt to enhance the
explainability of ML systems agnostic of the underlying algorithms mainly
harness ontologies and knowledge graphs. Table 1 also illustrates that only one of
the reviewed articles covers reinforcement learning. In the following paragraphs
we present more in-depth ndings for each type of ML approach.
Concerning supervised learning (classi cation) techniques, Table 1
illustrates that Neural Networks are the dominant prediction model. The
architectures proposed are manifold and include, among others, recurrent (e.g. [
16, 57
])
and convolutional (e.g. [
13
]) networks as well as autoencoders (e.g. [
5, 6
]). In
combining these models with Semantic Web Technologies one approach is to
map network inputs or neurons to classes of an ontology or entities of a
knowledge graph. For example, Sarker et al. [
47
] map scene objects within images to
classes of the Suggested Upper Merged Ontology. Based on the image classi
cation outputted by the Neural Network, the authors run DL-Learner on the
ontology to create class expressions that act as explanations. Similarly, in the
work of [
56
], image contents are extracted as RDF triples and then matched to
DBpedia via the predicate same-concept. In order to answer questions provided
by the user about an image, the system translates each question into a SPARQL
query which is run over the combined knowledge base. The results of this
operation are then used to give an answer and substantiate it with further evidence
that acts as an explanation. A related approach is used in [
21
] to explain image
recognition on classes that have not been part of any training data (zero-shot
learning). Furthermore, Selvaraju et al. [
49
] learn a mapping between individual
neurons and domain knowledge. This enables the linking of a neuron's weight
(importance) to semantically grounded domain knowledge. Another common
approach within the supervised classi cation group is to utilize the taxonomical
information of a knowledge base. These hierarchical relationships aid the
explanation generation in di erent ways. For instance, Choi et al. [
15
] and Ma et
al. [
36
] design attention mechanisms while authors such as Che et al. [
12
] and
* Supervised learning comprises of classi cation approaches only because in this review
regression models were only used in systems developed for multiple techniques.
** Markovian Decision Process (MDP)
Jiang et al. [
29
] employ model regularization based on this domain knowledge.
It should be noted, however, that these systems focus more on interpretability
than explainability. Since these approaches are often found in the health care
domain they are more thoroughly discussed in Section 3.2.
Regarding unsupervised learning, we identi ed two groups within the
reviewed literature. As shown in Table 1, a signi cant body of research aims at
creating explainable embeddings of or with knowledge graphs. For the most part
these approaches are part of some recommendation engine and are thus explained
in more detail in Section 3.2. Apart from these, a smaller number of scholars
strive to increase the level of interpretability or explainability for clustering
algorithms. Batet et al. [
7
] use the taxonomical knowledge encoded in WordNet to
derive a semantic similarity function which leads to more interpretable clusters.
The authors present an extension to their work [
8
] which allows the incorporation
and merging of multiple ontologies within their framework. However, no speci c
explanations are provided by the system as to how cluster membership of data
points can be justi ed. Tiddi et al. [
50, 51
] go beyond semantic similarity
functions and propose to explain clusters or data patterns (agnostic of the clustering
algorithm) by traversing a knowledge graph to nd commonalities among the
clusters. The system, called Dedalo, uses ILP to generate candidate explanations
based on the background knowledge and the given clusters. The former is built
by dynamically following the URI links of the items in the data set. However,
such a technique raises the question of explanation delity, thus asking whether
the given explanation actually agrees with the underlying predictive model.
As stated above, only one reviewed system aims at explaining
reinforcement learning. In this research [
31
] the authors utilize an ontology to
incorporate domain knowledge into the explanation process of an MDP recommendation
system. The ontology is used to provide information which is not available from
the data alone and to perform inference to create rules which limit the number
of actions recommended. Finally, Semantic Web Technologies such as ontologies
can be used to aid explainability and interpretability from a more general and
model agnostic point of view. Along these lines, Krishnan et al. [
32
] design
an explainable personal assistant that uses an ontology to dynamically grow
a knowledge base, interact with other modules, and perform reasoning. In
addition, Racoceanu and Capron [
44
] design a medical imaging platform which
provides decision reproducibility and traceability powered by an ontology. Even
more general, some authors propose ontologies or interlingua to declaratively
represent aspects and dimensions of explainability. For instance, McGuinness et
al. [
38
] create three ontologies with concepts and relation about data provenance,
trust, and justi cations, thus o ering an explanation infrastructure. Similarly,
by constructing an ML schema, Publio et al. [
43
] aim at exposing the semantics
of such systems which can positively a ect model explainability.
Lastly, we want to highlight another insight relating to the performance of
the explainable systems. It is worth noting that in using Semantic Web
Technologies alongside ML algorithms, explainability is not raised at the cost of
performance. Rather, the reviewed systems often achieve state-of-the-art
performance in their respective tasks. This is particularly notable because these results
exemplify how to overcome the often assumed trade-o between ML accuracy
and interpretability by the means of structure and logic [
46
].
3.2
Domains and Applications
The combinations of ML algorithms and Semantic Web Technologies are also
driven by the respective application domains and tasks to be accomplished. Table
2 provides an overview of the most frequent domains and tasks of the reviewed
systems. Regarding the former, it becomes apparent that { while many systems
are developed agnostic of a speci c domain { health care is a strong driver
for interpretable ML systems. Regarding the tasks of the reviewed systems, we
found the recommendation task and image analysis to be of great importance.
For brevity we limit the following paragraphs to the health care domain and the
recommendation task.
Tasks and Domains Authors
isanHGeeanletrhalCare [[
13
]2,],[1[
16
]5,],[2[
24
]9,],[2[
38
]6,],[3[
48
]2,],[4[
43
]4,],[4[
57
]4,],[5[600{]52], [
55, 56
], [
61
]
mEntertainment [
9
], [
41
], [
57
]
oDCommercial [
4
], [
33
], [
37
]
Recommendation [
4
], [
9
], [
28
], [
31
], [
37
], [
41
], [
55
], [
57
]
sImage Annotation or Classi cation [
5, 6
], [
21
], [
44
], [
47
], [
49
], [
60
]
sakTKrnaonwsfleerdgoer BZaersoe-CShoomtpLleetaironning [[1234]],, [[2512]],, [[4691]]
TDiagnosis Prediction [
12
], [
15
], [
36
]
Visual Question Answering [
3
], [
56
]
Note: Multiple selections possible.
Systems in the domain of health care often combine classi cation tasks such
as diagnosis prediction with taxonomical knowledge found in medical diagnosis
codes or medical ontologies. For instance, Jiang et al. [
29
] use the hierarchical
information of the International Classi cation of Diseases (ICD) to introduce a
regularization penalty to their logistic regression which produces a sparse model
where non-zero features tend to be localized within a limited number of
subtrees instead of being scattered across the entire hierarchy. This kind of feature
weighting might make the algorithmic prediction process more explicit
(interpretability), but it does not provide explanations and justi cation for laymen
(e.g. patients). Similarly, Chen et al. [
12
] incorporate hierarchical ICD
knowledge in a Neural Network architecture to regularize the output layer of the
network and learn clinically relevant features. Yan et al. [
60
] use hierarchical
relationships within an ontology to expand a set of medical labels by inferring
missing parent labels. For example, the label "right mid lung" is expanded to
"right lung", "lung", and "chest". The authors also utilize exclusive relationships
between labels to learn hard cases and improve accuracy. When making
predictions on medical images, their system is able to provide input examples similar
to the given model output as prediction evidence. Finally, KAME [
36
] is a
diagnosis prediction system inspired by [
15
] which uses medical ontologies to learn
(embedded) representations of medical codes and their parent codes. These are
then utilized to learn input representations of patient data which are fed into
a Neural Network architecture. The authors exploit an attention mechanism
which learns weights that allow to interpret the importance of di erent pieces of
knowledge. Summing up, within the domain of health care many interpretable
ML models have been proposed. These mainly use taxonomical knowledge to aid
performance and interpretability. The reason for the relative abundance of such
systems in the health care domain stems from the high stakes characteristics of
the eld as well as the existence of di erent medical ontologies.
Due to their extensive use of knowledge graphs, recommendation systems
are an important branch of research in the reviewed eld. More speci cally, these
systems commonly combine embedding models with knowledge graphs. For
example, Bellini et al. [
9
] inject the DBpedia knowledge graph into an autoencoder
network which is constructed to mirror the structure of the knowledge base. After
training such a system for each user, the learned weights map to explicit semantic
concepts from the knowledge graph and user-speci c explanations can be
generated based on these insights. Another special case of embedding is RippleNet [
55
]
where the triples of a constructed knowledge graph (based on Microsoft Satori)
are iteratively compared to the embeddings and then propagated. This way the
path from a user's history to a recommended item can be used as an explanation
for the recommendation. Further, there are approaches which use Semantic Web
Technologies agnostic of the underlying recommendation algorithm. One such
system is ExpLOD [
41
] which makes use of the Linked Open Data paradigm.
The framework rst maps liked items and recommended items into a knowledge
base such as DBpedia, then builds a graph, ranks the properties in this graph
based on relevance, and nally creates a natural language explanation from the
top properties retrieved. While being model agnostic, the issue of explanation
delity can be raised again here because the given explanation might not
correspond to the actual underlying model process. Finally, it is worth mentioning
that explainability in recommender systems is mainly driven from a user-centric
perspective with the aim to increase user satisfaction and acceptance.
3.3
Explanation Forms and Evaluation
The conducted analysis revealed that the presentation and form of the given
explanations is highly diverse { even within similar domains or prediction tasks.
For example, some scholars combine di erent types of explanations (e.g. visual
and textual [
49
]) in order to increase explainability while others provide only
minimal explanation towards the user (e.g. [
7
] or [
52
]). Moreover, only few
authors present explanations in natural language. For instance, Musto et al. [
41
]
incorporate a dedicated natural language generator into their recommendation
algorithm. The authors utilize a template-based approach which is also used
by other authors [
4, 31
]. A more frequently employed explanation form consists
of textual (semi)-logical or rule-like notation. Further, explanations are usually
designed to optimally justify correct model output. One deviation from this is
the work of Alirezaie et al. [
5, 6
] where the errors of a Neural Network image
classi er are explained by performing ontological reasoning upon objects of a
scene. To illustrate the range of explanation forms used, Table 3 provides
selected examples of textual explanations encountered in this review. Apart from
the ambiguity of the term explainability, one potential reason for this
diversity includes the relevancy of an explanation for a given system: While in most
reviewed cases, explainability is an explicit goal, in a subset of models,
explainability is treated as a secondary goal and Semantic Web Technologies are used
to primarily address other issues such as data sparseness (e.g. [
15
]).
Furthermore, we found most systems to o er rather static explanations
without much user interaction. In this context, the work of Liao et al. [
33
] is an
exception as the proposed recommendation system enables user-feedback on
human-interpretable domain concepts. Moreover, looking into the future, Sarker
et al. [
47
] envision their explanation tool for image classi cation to be used in
an interactive human-in-the-loop system where a human monitor can correct
algorithmic decisions based on the given explanations. On the whole, however,
we notice a lack of user-adaptive or interactive explanation approaches in the
reviewed literature.
Finally, when it comes to evaluating the goodness of the explanations, only
few authors go beyond a subjective assessment of the proposed system. Bellini
et al. [
9
], for instance, perform an evaluation of their knowledge-aware
autoencoder recommendation system by conducting A/B testing with 892 volunteers.
Similarly, Musto et al. [
41
] designed a user study in which 308 subjects lled out
a questionnaire involving questions such as "I understood why this movie was
recommended to me". Through this evaluation, the authors gain further insights
into di erent aspects of how their explanation system a ects end users. Other
authors propose more quantitative evaluation metrics to determine the
goodness of the given explanations. Zhang et al. [
61
] explain their link predictions
by nding patterns within a knowledge graph which are similar to the predicted
ones (see Table 3) and measure explanation reliability by the number of similar
patterns found. Further, Jiang et al. [
29
] measure the interpretability of their
predictive system by quantifying the sparseness of their linear model while
taking into account the taxonomical structure of their data. Overall, from these
ndings it becomes obvious that there is no accepted standard for evaluating
explanations within XAI.
4
Trends for Future Research
Based on our review of the relevant literature we articulate opportunities and
challenges for future research in the eld. We generate these insights based on
our analysis and comparison among all reviewed papers as well as on the basis
of the challenges put forward within each of the articles.
4.1
Semantic Web Technologies for Explainability
The combination of Semantic Web Technologies and ML o ers great potential
for facilitating explainable models. We identi ed the matching of ML data with
knowledge base entities { which has been called knowledge matching [
21
] { as
one central challenge which needs to be overcome by future research. Speci cally,
automated and reliable methods for knowledge matching are required. In this
context, Wang et al. [
56
] suggest string matching between identi ed objects and
ontology classes and Liao et al. [
33
] propose to mine concepts and relationships
automatically from online sources. Further research in this area as well as
related elds like semantic annotation are needed to enable e ective and e cient
knowledge matching.
Moreover, we found a certain concentration on speci c ML techniques and
Semantic Web Technologies. More work needs to be conducted on explainable
reinforcement learning and clustering. In this context, we also note that the work
across di erent disciplines and tasks still remains somewhat isolated even though
concepts like linked data provide the tools for integrating various domains. Some
existing research acknowledges the need to extend the range of tasks performed
by explainable systems [
12
] or their domains of application [
32
]. Other authors
envision the use of more data [
60
] or more complex background knowledge [
41,
42, 47
]. Hence, the areas of ontology or knowledge graph learning as well as
knowledge base matching play an important role in accomplishing this goal.
Future work will therefore need to nd ways to mitigate the potential lack of
data interconnectedness and the increased complexity of such systems.
Finally, we highlight the need for future work to aim for truly explainable
systems which incorporate reasoning and external knowledge that is
humanunderstandable. To achieve this goal, future explanation systems need to ensure
that the explanations given are truthful to the underlying ML algorithm.
Further, such approaches should be able to explain not only how an output relates
to some representation of interest but also how this representation has been
obtained. For example, it is not enough to justify that a human face has been
detected by stating that eyes, mouth, and nose were recognized and that these
features are part of a human face (e.g. inferred via ontology). A truly explainable
system should also be able to explain why these features have been recognized.
This point relates to the question of user interaction, which is discussed below.
4.2
Human-Centric Explanations
Since explanations are forms of social interactions [
2
], their e cacy and quality
depend to a large extent on their intelligibility and comprehensibility as
perceived by the user. In other words, an explanation is only useful if the user is
able to understand it. In this review we have shown that the form and appearance
of explanations di ers signi cantly among current systems and many of those
do not provide explanations in natural language. Therefore, we believe that the
eld of Natural Language Processing (NLP) and Natural Language Generation
(NLG) in particular o ers a useful starting point. For example, Vougiouklis et
al. [
53
] generate natural texts from Semantic Web triples using Neural
Networks. Moreover, Ell et al. [
20
] translate SPARQL queries to English text that
is understandable by non-experts. More generally, the eld of (Visual) Question
Answering can be a source of inspiration since questions and answers are usually
given in natural language [
56
].
Additionally, we believe that explanations need to be adaptive and interactive
in order to generate the greatest bene t for the user. Structured knowledge
bases could allow users to scrutinize and interact with explanations in various
forms. For example, user could browse among di erent possible explanations
or drill down on a speci c explanation to extract more speci c reasons that
contributed to a prediction. Khan et al. [
31
] envision a system that allows for
such follow up questions. Similarly, Bellini et al. [
9
] plan to incorporate the
possibility for users to correct their system in a continuous loop. As described
above, Sarker et al. [
47
] also regard this course of action as an important task
for future studies. However, there seems to be no consensus regarding the actual
mode of interaction. In order to nd optimal ways of presenting and interacting
with explanations, future research needs to incorporate ndings from a greater
variety of research elds. Existing studies [
1, 2
] show that there is a growing
body of diverse and interdisciplinary work addressing the question of
humanunderstandable explanations that can be leveraged in this context.
4.3
Common Grounds for Evaluation
We believe that meaningful progress in the eld of XAI is not only dependent on
novel explanation algorithms but also on common grounds for model evaluation
and comparison. In light of this, Doshi-Velez and Kim [
18
] put forward the need
for a shared language relating to factors of ML explainability. We have shown
that Semantic Web Technologies can help in creating such a common lingua.
Future work, however, needs to proof how to utilize such constructs e ectively
in the context of explainability. Another way forward could be to develop and rely
on standard design patterns for combining ML with Semantic Web Technologies.
The work of van Harmelen and ten Teije [
25
] already provides a collection of
patterns for such hybrid systems. Moreover, common evaluation criteria need
to be established so that subjective assessments of model explainability can be
replaced by more rigorous practices.
5