In a context where Machine Learning models are becoming increasingly widespread and complex, the importance of techniques to make these models less opaque is widely recognized. The research fields of Neuro-symbolic Artificial Intelligence and Explainable Artificial Intelligence both revolve around techniques to increase the transparency of complex Machine Learning systems. The research proposed in this paper aims at defining a novel framework to exploit symbolic structures, in particular ontologies, to provide innovative solutions to the problem of understanding Machine Learning models. The core of our proposed approaches is represented by the Ontology Based Data Management paradigm, which is exploited to define a relation between the sub-symbolic data samples used in Machine Learning models, and the symbolic knowledge possessed by domain experts, formalized in an ontology. The key innovation of the presented framework is the definition of a novel form of mapping, which allows to reconcile symbolic and sub-symbolic knowledge. In particular, leveraging this framework, we are interested in studying two problems: formulating explanations for the behavior of models, and performing reasoning over these Machine Learning systems. We argue that investigating these problems could lead to valuable contributions to the research field.
In recent years, the use of Machine Learning (ML) models spread across numerous domains, many of which involve significant ethical implications. As a result, the problem of opacity and low interpretability of these models has become a crucial area of interest. In the Artificial Intelligence (AI) research field, two of the branches that investigate this issue are Explainable AI and Neuro-symbolic AI. Explainable AI focuses on techniques to provide explanations to the decisions of ML models, usually classifiers. Neurosymbolic AI instead, is the research field that studies how to integrate symbolic domain knowledge with sub-symbolic data representations used by ML models, as neural networks. The work presented in this paper falls in both these research fields, as we aim to exploit the Neuro-symbolic AI approach towards model interpretability and to provide symbolic explanations to the behavior of the models.
Specifically, the key idea of our proposed approach lies in the use of ontologies to represent the domain knowledge, and in employing the Ontology Based Data Management (OBDM) paradigm to reconcile the sub-symbolic representation of data sample with the symbolic domain knowledge of the ontology. The main advantage of this novel approach is represented by the high level languages that can be used to formalize the logical rules. Ontologies, by definition, formalize the knowledge in terms of concept and roles, which are particularly close to the human approach to reasoning. We argue that this approach could contribute to the field of Neuro-symbolic AI, making the symbolic components of these systems, such as logical rules and explanations, easier to define, manipulate, and understand, therefore enhancing interpretability, transparency and trustworthiness of the models.
The first objective we aim to achieve is the definition of a novel formal framework, based on the OBDM paradigm, that allows to reconcile the numerical representation of the data samples with the symbolic knowledge defined by the ontology. Then, we aim to study two main applications of this framework: performing reasoning over ML models and providing semantic explanations for their behavior.
The rest of the paper is organized as follows. In Section 2 we review the main techniques that form the foundation of this research. Section 3, which constitutes the core of this paper, is structured in three paragraphs: first outlines the open questions we aim to address; then presents the framework we intend to define; lastly discusses two interesting applications for the framework, along with some preliminary results, and future works. Finally, Section 4, concludes the paper by providing an overview of the presented research, its main motivations and objectives.
As mentioned in Section 1, the research presented in this paper falls under both the Neuro-symbolic AI and Explainable AI categories, and exploits the OBDM paradigm to address the described problems. In the following we provide a brief overview of these fields, together with some of their most recent and prominent approaches.
In the early days of AI research, symbolic approaches represented the dominant paradigm. However, in recent years, the most popular approaches to AI are sub-symbolic, i.e., focused on learning implicit data representations rather than using formal symbols and logical rules. Bridging the gap between these two approaches is at the core of Neuro-symbolic AI, which aims to obtain systems that leverage the strengths of both paradigms [1]. In particular, sub-symbolic systems, as neural networks, are able to solve highly complex problems by learning implicit patterns in the data, that would be much more dificult to explicitly formalize using semantic rules. Symbolic techniques, on the other hand, use human-readable representations of knowledge and data, making these models much more interpretable. Combining symbolic and sub-symbolic approaches ofers several promising strategies, which typically include neural networks as the sub-symbolic component, and difer by the way symbolic strategies are integrated within the network [2]. Among the various contexts where Neuro-symbolic AI shows potential, our primary focus is on its ability to make ML models more understandable and interpretable.
This field of research emerged in response to the growing complexity of ML models, which resulted in highly opaque models, often treated as black-boxes. However, in recent years, these models became widespread, even in high-risk contexts, and their opacity raised significant ethical concerns [ 3, 4, 5]. This has led to a growing interest in techniques that aim to explain the behavior of these models, or ofer insights into their internal processes [6].
The majority of the studies on explanations assume the model to explain is a classifier, and the subject of the explanation is its output decision. The most notable techniques are based on the following definitions: local explanations provide explanations for individual predictions, i.e., the output computed for a specific input instance; global explanations, instead provide explanations for whole classes, hence focusing on the behavior of the classifier in general [ 7]; abductive explanations [8], also known as prime implicants [9] or suficient reasons [ 10], focus on answering the question of why an instance is classified in a specific way; contrastive, or counterfactual, explanations, on the other hand, focus on the question of why an instance isn’t classified diferently [ 11]; post-hoc explanations consider the model as a black-box, and the computation of the explanation is only based on its input and output [8, 11]; intrinsic explanations instead rely on the model itself to explain its behavior, within its algorithm [12].
OBDM [13] is a paradigm based on a three level architecture, constituted by: a source schema, an ontology, and a mapping defining the relations between the two. This approach is a form of information integration, in which the global schema is formulated through an ontology. The ontology, however, allows us not only to reconcile diferent sources, but also to enrich those with symbolic knowledge from the domain of interest, which is the main reason why we resort to OBDM in this study.
Formally, an OBDM specification is a triple = ⟨, , ⟩, where is the ontology, is the relational schema representing the source data, and is a mapping from to . The semantics of the OBDM system is defined by the models of , i.e. the interpretations that satisfy both the logical theory of and the mapping assertions in . This implies that the semantics of the OBDM specification strongly depends on the semantic specified for , i.e., sound, complete or exact.
Problem Statement and Research Questions The problem at the core of our research is the lack of interpretability of many commonly used ML models. The main research question we aim to address is how to integrate symbolic domain knowledge with the sub-symbolic representation of data. Furthermore, we are interested in investigating the following scenarios: how to perform semantic reasoning tasks over sub-symbolic ML models; how symbolic reasoning can be used to investigate their behavior, check consistency, or even carry out new derived classification tasks; and lastly, how to compute semantic explanations to the behavior of ML models.
The Proposed Framework The key idea behind the proposed approach lies in the OBDM paradigm [13]. Therefore, the framework we aim to define is structured as a three-layer architecture: data layer, ontology layer, and mapping. The data layer and the ontology layer formalize, respectively, what we know on the sub-symbolic level, i.e., how is the numerical representation of the data samples defined, and what we know on the semantic level, i.e. the domain knowledge possessed by experts. However, the key innovation represented by this framework lies in the definition of a novel form of mapping. While mappings are well-known constructs, our approach introduces a specific form able to relate formulae expressed over the sub-symbolic, often numerical, representations of the data samples, to high level domain knowledge. This form of mapping, to the extent of our knowledge, represents a completely novel approach, which we argue is a promising technique for many of the questions presented before. In the following we provide a brief description of the framework architecture.
The data layer defines a formal representation of the data samples that are subject to a classification task. In ML models, data samples are usually represented as vector of features, also referred to as attributes. In our framework each attribute is formalized as a function, that associates to each data sample the corresponding value for that attribute. This formalization is crucial, as we are interested in defining a formal language to express formulae over the data samples, and therefore we need a formal representation for them. Specifically, we aim to study formulae that allow classical mathematical operators as +, · , and comparison symbols as <, ≤, >, ≥, = . It can be easily verified that these are some of the operators that appear in the decision functions of many classifier models. The intuition behind this choice is that one could use this language to formalize part of, or all, the behavior of classifiers. An example of formula we aim to express with this language is () = (( 1()+2()) ≤ 3)∨( 3() = 1), where 1, 2, 3 are functions representing attributes, and is a data sample for the classification task.
The role of the ontology is to formalize the knowledge possessed by experts regarding the domain of interest of the classifier. The ontology can provide both intensional and extensional knowledge to the domain. Intensional knowledge is expressed through logical assertions, usually referred to as the Tbox, and provide rules that should be satisfied by the data. These rules are formulated in terms of predicate symbols, defining classes and relations between individuals of the knowledge base. This approach closely mirrors human reasoning, thus facilitating understanding. Moreover, the ontology can also provide extensional knowledge, i.e. data populating the knowledge base, usually known in the literature as the Abox. Extensional knowledge can be particularly useful to inject in the domain knowledge also information about known instances, possessed by domain experts.
A mapping assertion in our framework is defined as a tuple of formulae, where the first one is a formula expressed using the data language described before, and the second one is a first-order logic formula over the ontology alphabet. For the semantics definition we envision that it might be valuable to investigate both the sound and the exact approaches. A very simple example of mapping assertion could be ⟨{1() + 2()) ≤ 3}, {()}⟩ , where the first element is a formula over the attributes 1, 2 of a data sample , while the second one is the concept of student in an ontology. Under the sound semantics this would imply that all the data samples satisfying the formula over the attributes are students. The exact semantics would also imply that all the students must satisfy the left formula.
The first task to address is query answering over the defined framework. To this end, we call ontological specification a tuple = ⟨, ⟩ where is the mapping and the ontology. Query answering in this context is tightly bounded to the concept of certain answers, and therefore to the definitions of models of , and . Given a query over the ontology , its certain answers with respect to the mapping are all those data samples that are answers to that query in all the models of . Intuitively, all those samples that satisfy both the logical theory of and the assertions in .
Applications This paragraphs presents a more detailed description of the tasks we aim to address within the presented framework. In particular, we focus on the two research questions that are the focus of this paper: how to compute semantic explanations, and how to perform and exploit semantic reasoning over ML models.
As mentioned in the previous sections, an application of this framework we are interested in studying is the problem of computing explanations to the decisions of classifiers. The intuition behind the explanation definition is that it should be a formula, expressed over the ontology alphabet, such that the following properties hold: the classified data sample is in the certain answers of that formula; and all the certain answers of that formula are assigned the same class of by the classifier. This definition follows from the idea that an explanation should be able to represent the sample to explain, and no sample that belongs to a diferent class. This approach to the explanation problem falls under the category of local, abductive, post-hoc explanations. However, studying diferent definitions of the explanation is a problem we plan to address.
Regarding this specific explanation definition, we already presented some results in [ 14]. This paper provides a first formal definition of the framework, and studies the problem of verifying and computing an explanation for the output decision of a classifier. Being an early work, the framework is studied under several limitation assumptions. Specifically, the mapping is studied under the assumption that the left formula is quantifier-free, and uses only finite attributes, i.e., attribute functions with a finite codomain. Also, the formulae on the right-side of the mapping assertions allow only for formulae of the form (), ∃.(, ), and ∃.(, ). And finally, the Tbox is formulated in DL-Lite [15], and the Abox is empty. Moreover, we limited the study to explanations in the form of conjunctive queries. Under these assumptions, we provided algorithms for both the verification and computation problems, and studied the computational complexity of the verification, which was proved to be coNP-complete.
A first task on which we are currently working is the extension of the defined framework, with the intent of accommodating more powerful languages, and thus manage more complex problems. In particular, we are working on languages that allow the presence of constants in the ontology, as DL-Lite [16]. We argue that this is a fundamental requirement to be able to express meaningful domain knowledge. It is easy to verify that this extension requires the reformulation of the entire framework in terms of many-sorted first-order logics, which should distinguish between constant values and objects to classify.
The second task we aim to address is performing reasoning using symbolic knowledge across one or multiple ML models. A fundamental observation for the study of this tasks is that, for many popular ML models, the learned decision function can be expressed using the data language previously described. For example, Perceptrons, Support Vector Machines, Linear Regressions, all learn a decision function in the form of a linear combination 11 + ... + ≥ 0 , where 1, ..., are the weights, usually rational numbers, and 1, ..., are the values of the data sample features. It is immediate to check that this can be expressed using the framework data language as 11() + ... + () ≥ 0 where 1, ..., are the attribute functions that associate to the sample its feature values 1, ..., .
An immediate consequence of this observation is that it could be interesting to define mapping assertions where the left formula is the decision function of a ML model, while the right-side one is the class learned by that model. For example, one might have on the left-side the decision function of a classifier for students, and on the right-side the concept (). These particular mapping assertions pave the way for several applications of this framework. Assume a context in which each mapping assertion represents a classifier, with its sub-symbolic decision function and symbolic ontology class. A first application of query answering over this mapping could be gaining insights on the behavior of the models, to reveal inconsistencies or undesired behaviors. Also, it could be used to synthesize new classifiers. This could be obtained by formulating queries defining a particular class, not learned by the classifiers present in the mapping. Intuitively, rewriting this query would lead to the sub-symbolic formula representing all the data samples in that class, hence a synthesized classifier for it. Lastly, another interesting application is ensemble learning. Query answering involves rewriting the query with respect to all the mapping assertions, which means that every classifier is taken into consideration in order to answer the query. This can be seen as a form of ensemble learning, where all the models participate to the classification task. Furthermore, we argue that it might be valuable to study this framework with probabilistic ontology languages, which could integrate the concept of confidence of a model, in the form of probability, or likelihood, that an individual is assigned a certain class.
The work presented in this paper introduces an innovative approach within the Neuro-symbolic AI and Explainable AI fields of research, leveraging ontologies to enhance the interpretability and transparency of ML models. Exploiting the OBDM paradigm, the presented framework allows to reconcile the sub-symbolic representation of data used by ML models, with the high-level symbolic representation of the knowledge possessed by expert users. First, we provided a description of the framework we intend to achieve, and then we presented two application scenarios for it: providing semantic explanations, and performing reasoning tasks. For the first one, we provided a summarization of the results already obtained, along with notes on its limitations and the intended future extensions. For the latter instead we described several scenarios in which classical reasoning tasks, as query answering, could be valuable in solving new interesting challenges.
In conclusion, we argue that the presented work can provide a valuable contribute to the research areas of Explainable AI and Neuro-symbolic AI, with the potential of providing novel techniques to enhance interpretability, transparency and trustworthiness of ML models.
The work presented in this paper has been supported by PNRR MUR project PE0000013-FAIR.
The authors have not employed any Generative AI tools. [1] M. Garnelo, M. Shanahan, Reconciling deep learning with symbolic artificial intelligence: representing objects and relations, Current Opinion in Behavioral Sciences 29 (2019) 17–23. URL: