The idea of integrating symbolic and sub-symbolic approaches to make intelligent systems (IS) understandable and explainable is at the core of new fields such as neuro-symbolic computing (NSC). This work lays under the umbrella of NSC, and aims at a twofold objective. First, we present a set of guidelines aimed at building explainable IS, which leverage on logic induction and constraints to integrate symbolic and sub-symbolic approaches. Then, we reify the proposed guidelines into a case study to show their efectiveness and potential, presenting a prototype built on the top of some NSC technologies.
a natural choice for building human-intelligible IS [
Our work lays under the umbrella of NSC, and its contribution is twofold. On the one hand, we present a set of guidelines aimed at building explainable IS, even when they exploit sub-symbolic techniques. The guidelines leverage on logic induction and logic constraints as the two main techniques integrating symbolic and sub-symbolic approaches. In particular, logic induction makes it possible to extract the knowledge from black-box ML-based predictors – typically, the sub-symbolic part of an IS – ofering a corresponding symbolic, logical representation. Conversely, logic constraints are exploited to inject some logic knowledge into the black box, thus restricting the numerical underlying model.
On the other hand, we reify the proposed guidelines into a case study to show their efectiveness and potential. In particular, we present a prototype built over some promising NSC technologies. The resulting system is then assessed to verify its capability of being adjusted (i.e., debugged and fixed) in case some unexpected behaviour in the sub-symbolic part of the system is revealed. Accordingly, we show how the prototype is correctly performing w.r.t. the proposed guidelines.
The paper is structured as follows. Section 2 provides a brief overview of the field of symbolic and sub-symbolic integration. It also includes the main related works on the use of a hybrid system as a mean for explainability. In Section 3, we first present the guidelines for building explainable systems; then, in Section 4 we discuss a possible instantiation of the proposed guidelines. In Section 5, we proceed with the assessment of our prototype and the discussion of results. Finally, Section 6 concludes the work.
In the last years, deep and machine (ML) learning methods have become largely popular and successful in real-world intelligent systems. However, their use raises the issue of understanding and explaining their behaviour to humans. Neural networks in particular – which are the most hyped and widely adopted approach within sub-symbolic AI – mostly sufer from the problem of opaqueness: the way they obtain their results and acquire experience from data is unintelligible to humans.
One of the proposed approaches addressing the explainability problem [
Among the others, the neuro-symbolic computing (NSC) field is a very recent and promising
research area whose ultimate goal is to make symbolic and sub-symbolic AI techniques
efortlessly work together. Due to the freshness of the topic, however, a well-established and coherent
theory of NSC is still missing. For this reason, a variety of methods have been proposed so far,
focusing on a multitude of aspects not always addressing interpretability and explainability
as their major concern. Nevertheless, some attempts to categorise XAI-related existing works
under the NSC umbrella exist [
Despite NSC does not explicitly include XAI among its primary goals, in this paper we
borrow some ideas from the NSC field to show how the explainability of modern IS may benefit
from the integration of symbolic and sub-symbolic AI. In particular, our work focuses on
two main NSC sub-fields: namely, the logic as a constraint, for the constraining module, and
diferentiable programming for the induction module—since the proposed prototype exploits
logic induction via diferentiable programming [
Within the scope of logic as a constraint, most approaches exploit some sort of logic formulae
to constrain the behaviour of the sub-symbolic predictor—in most cases, a neural network.
This formula is then vectorised – i.e. translated into a continuous function over vectors of real
numbers – and exploited as a regularisation term in the loss function used for training the
subsymbolic predictor [
With respect to the second research area, some works laying under the umbrella of
diferentiable programming fruitfully intertwine ML and inductive logic programming (ILP) [
As far as the intersection of logical systems and numerical models is concerned, the main
contributions come from [
Although these works ofer a good starting point in the search of a solution to the transparency problem, some remarks should be pointed out. First, exploiting a knowledge base obtained only from the training data is not suficient to acquire the knowledge required to explain the entire network behaviour. That would lead to a system giving explanations according to the network optimal functioning, not accounting for the training errors. Moreover, according to these models, also the constraining part should be driven by the rules inferred from the training data, hardly limiting the potential of those techniques. Indeed, the possibility for users to impose their own rules would also give them the ability to mitigate the errors derived from an incomplete or incorrect training set.
The work presented here aims at building a model overcoming both these limitations. As for the explanations’ coherence problem, the use of the black box as a data source in the logic induction process should guarantee the correct correlation between the black box itself and the derived logic theory. Furthermore, logic can be leveraged so as to combine the IS with the user experience and knowledge, thus exploiting all the advantages of the constraining techniques.
In this paper, we present a general model for explainable data-driven IS, and a set of guidelines supporting their construction. The novelty of our approach lays in the fruitful intertwining of symbolic and sub-symbolic AI, which aims at providing both predictive accuracy – through the exploitation of state-of-the-art machine learning (ML) techniques – and transparency—through the exploitation of computational logic and logic programming (LP). The proposed model, in particular, aims at overcoming the well-known limitations of ML-based AI w.r.t. interpretability. Accordingly, it leverages on a number of contributions from the NSC and LP research field, as well as two basic techniques—namely, induction and constraining.
The main idea behind our work is that IS should feature both predictive precision and interpretability features. To preserve predictive precision, IS should keep exploiting highperformance, data-driven, black-box predictors such as (deep) neural networks. To overcome the interpretability-related issues, IS should couple sub-symbolic approaches with logic theories obtained by automatically extract the sub-symbolic knowledge of black boxes into symbolic form. This would in turn enable a number of XAI-related features for IS, providing human users with the capabilities of (i) inspecting a black box – also for debugging purposes –, and (ii) correcting the system behaviour by providing novel symbolic specifications.
Accordingly, in the reminder of this section, we provide further details about the desiderata which led to the definition of our model. We then provide an abstract description of our model and a set of guidelines for software engineers and developers. Finally, we provide a technological architecture to assess both the model and the guidelines.
Regardless of the architectural and technological choices performed by designers and developers, the IS adhering to our model are characterised by several common desiderata (Di) w.r.t. their overall functioning and behaviour. Generally speaking, these are aimed at making IS both prediction-efective and explainable. Indeed, following this purpose, IS should D1 attain high predictive performances by leveraging ML and data-driven AI
A key enabling point in satisfying these desiderata is knowledge representation. While ML and data-driven AI are certainly required to mine efective information from data eficiently, they soon fall short when it comes to satisfying desiderata D2–D5. This happens because they mostly leverage on a distributed, sub-symbolic representation of knowledge which is hard to interpret for human beings. Therefore, to support D2 and D4, we need an alternative human-intelligible representation of the sub-symbolic model and a procedure to perform such a representation transformation. Furthermore, to support D3 and D5, we also need to link symbolic and sub-symbolic representations in a bidirectional way—meaning that an inverse procedure aimed at converting symbolic information back into sub-symbolic form is needed as well.
Accordingly, the focus of our model is both on the extraction of symbolic representation from the black-box predictor and, vice-versa, on the injection of symbolic representation (constraints) in the corresponding black-box predictor. The purpose of this dichotomy is twofold: guaranteeing the comprehensibility of the black-box model for humans – as symbolic representations are to some extent inherently intelligible –, and enabling debugging and correction of the black-box behaviour, in case some issue is found through inspection.
Generally speaking, we model an explainable, hybrid IS as composed by a black box, a knowledge base (KB) and an automatic logic reasoner, as depicted in Figure 1. Explainable recommendations or suggestions are provided to the end-user via the logic reasoner – based on symbolic rules and facts included in the KB by domain experts – and via black-box predictions—based on data. Accordingly, the reasoner combines several sorts of inference mechanisms (e.g. deduction and induction). Furthermore, to balance the knowledge coming from data with the domain experts knowledge, the model exploits induction and constraining techniques to improve the black box with the logical knowledge and vice-versa.
The black-box module is the core of the IS, making it capable of mining efective information from data. Any sub-symbolic model providing high predictive performances – e.g. neural networks, SVM, generalised linear models, etc. – can be used for the module implementation. This, however, may bring opaqueness-related issues. Thus, the black-box module is complemented with the other two modules to provide explanation and debugging facilities.
The reasoner module is aimed at providing explainable outcomes to the end-users. In particular, explanations are given in terms of logic KB, capable of approximating the black box. The construction of the logic KB relies on the induction capabilities ofered by this module. More precisely, the outcomes generated by the black box are exploited to build a logic theory, mimicking the work of the black-box predictor with the highest possible fidelity. An inductive process is then fed with the resulting extended theory. This leads to a theory containing a number of general relations, which can be exploited to provide intelligible information to the end-users. The described workflow supports explainability in two ways: (i) it provides a global explanation of how the black box works in terms of general relations that must always hold; (ii) it provides local explanations, justifying each black-box conclusion by enabling the use of deductive reasoning on the connected logical knowledge. In other words, this mechanism is what enables IS to be interpretable and debuggable by users. Finally, the KB module aims at storing the logical knowledge approximating the black-box behaviour. The knowledge can be modified by domain experts. When this is the case, it becomes of paramount importance to keep the black-box module coherent with the human-proposed edits. To this end, constraining is performed to align the black-box behaviour with the KB. This mechanism is what enables IS to be fixed by users.
In the reminder of this section, we delve deeper into the details of these mechanisms. 3.2.1. Logic induction While deductive reasoning moves from universal principles that are certainly true to specific conclusions that can be mechanically derived, inductive reasoning moves from specific instances to general conclusions. Induction is a key mechanism of our model. Assuming IS can transform the data they leverage upon for training black boxes into theories of logic facts, inductive reasoning can be used to extract the rules that best explain that data.
The induction procedure is not meant to replace the black box as a learning or data-mining
tool—as it would be quite dificult to obtain the same performance in terms of accuracy and
ability to scale over huge amounts of data. Conversely, it aims at “opening the black box”,
letting humans understand how it is performing and why. In other words, following the abstract
framework presented in [
In our model, the induction process is fed with both the raw data and the outcomes of the black box. In the former case, induction leads to the unveiling of latent relations possibly buried in the original data: we refer to the logic theory obtained as the reference theory. In the latter case, induction leads to a symbolic approximation of the knowledge the black box has acquired from data via ML: we refer to the logic theory obtained as explanation theory. Assuming the soundness of the induction process, discrepancies between these two theories could reveal some possible rules that have not been correctly learned by the sub-symbolic model. It is then possible to fix it by enforcing the correct relations via logic constraining.
Finally, one last point is worth to be discussed. Unless the induction process possesses the same learning capabilities of the black box, it is impossible to detect all its learning errors. In this case, in fact, the reference theory would be the optimal solution itself, and the sub-symbolic model would be useless. As the induction process aims at opening the box, its use on the raw data aims at providing insights on the accuracy of the training phase. Thus, it does not aim at providing an optimal solution. 3.2.2. Logic constraining While induction is the mechanism aimed at translating sub-symbolic knowledge into symbolic form, constraining is the inverse process aimed at injecting some symbolic knowledge into a black box. In this way, both the induced rules and those coming from domain experts are used to constrain the black box and its outcomes. This is another key mechanism of our model.
In particular, the ability to encode some prior knowledge within a model is interesting for two main reasons. On the one side, it enables a reduction in the data needed to train the black box. In fact, handmade rules may be exploited to model a portion of the domain not included in the training data. So, rather than creating a more exhaustive training set, an expert may directly encode his/her knowledge into rule and train a constrained black box. We call this procedure domain augmentation. On the other side, one may exploit the constraining process to guide and support a black-box learning, e.g., helping it to avoid biases. We call this procedure bias correction.
3.3.1. Type of logic In the following, we discuss the main aspects to be considered when designing a system conforming to our model. As a first step in this direction, we first handle some important aspects concerning data representation.
The first aspect concerns the type of logic used by the IS. Logic, in fact, plays a fundamental role: it heavily afects the explainability properties of the final system. For this reason, the choice of the most appropriate logic formalism is a primary concern.
For our model to be efective, the selected logic formalism should provide high flexibility in the description of the domain. At the same time, to keep a system as simple as possible, it should be coherent in every part of it: from the constraining module to the induction one, every part should share the same representation of the logical knowledge. 3.3.2. Sort of data The second aspect concerns the sort of data used to feed the IS. For the logical induction process to be carried out on data, the first thing to do is transforming the data itself. In other words, the available data should be translated into a logical theory according to the chosen formalism. For the translation to be efective, the resulting theory should preserve as much as possible the information contained in the original data. Anyway, when dealing with some particular type of data some problems have to be taken into account.
When unstructured data – e.g., images, audio, time-series, etc – come into play, the transformation process may become considerably harder. In that case, a two-step transformation must be performed, involving: 1. extraction of semantically meaningful, structured features from unstructured data; 2. translation of extracted features into the desired logical formalism.
Of course, this procedure adds a whole range of new problems related to the accuracy of the features extracted. In fact, the explanation process is mainly linked to the reliability of the data used as the basis for the induction process. The exploitation of data generated through an automatic process makes a discrepancy between the data extrapolated by the inductive process and the behaviour of the black box more likely. 3.3.3. Sorts of black box In the general case, we require the architectures adhering our model to be as agnostic as possible w.r.t. the particular sort of black box to be adopted. In fact, the choice of the most adequate sort of black box is strongly afected by (i) the nature of the data at hand, (ii) the availability of some symbolic extraction technique making the black box less opaque, (iii) the possibility of constraining the black box with logic formulae.
Strictly speaking, the choice of the black box should be delayed to the implementation phase. Here we describe a number of criteria to be taken into account when performing this choice. We recall, however, that the other guidelines described so far are agnostic w.r.t. the sub-symbolic model to be used.
As far as the nature of the data is concerned, traditional ML techniques [
As far as opaqueness is concerned, virtually any ML techniques is afected by that to some
extent: yet, neural networks remain the most critical from this point of view. Nevertheless, the
vast majority of rule induction procedures are either black-box agnostic – a.k.a. pedagogical [
This subsection provides a technological description of an IS prototype adhering to our model. It is based on the concrete architecture detailed in Figure 2, which specialises the abstract model from Figure 1 through a number of technological commitments.
First, we present our choices in relation to the points examined in Subsection 3.3. As for the type of logic, within the scope of this paper, we adopt first-order logic (FOL) as our formalism of choice, and Prolog as our reference concrete language. FOL is likely to ofer the best trade-of between flexibility and expressiveness of the representation language. Furthermore, the choice of FOL enables the exploitation of many existing approaches supporting both conditioning and induction. Finally, the choice of the Prolog syntax enables the direct exploitation of the induced theories within existing automatic reasoners—e.g. tuProlog, SWI-Prolog, etc.
As far as the sort of the data is concerned, in this paper we only describe a prototype based on structured data. In fact, the feature extraction module can be omitted without hindering the generality of our approach. Moreover, the greater simplicity in the data helps to avoid the problems related to the possible information loss due to their transformation. In the future, however, we plan to extend our prototype to support arbitrary datasets including any sort of data. This requires the creation of a module for feature extraction.
In terms of the sort of the black-box used, we focus on a prototype based on neural networks, being them the most critical from the opacity perspective. The remaining technological choices derive consequently.
The prototype exploits two main technologies: DL2 [
The choice of NTP as the induction engine is more straightforward. Indeed, NTP is among
the few ready-to-use technologies ofering both deductive and inductive logic capabilities
in a diferentiable way. On the one hand, diferentiability is what makes induction more
computationally eficient w.r.t. similar technologies—and this is why we choose it. On the other
hand, NTP deductive capabilities are not mature enough. In fact, training a model correctly
approximating a logic theory in a reasonable amount of time is very challenging—especially
when the complexity of the theory grows. While this could be acceptable for the induction
process, it is still very limiting for deductive reasoning. Moreover, traditional logic engines
combine a very large ecosystem of supporting tools – IDEs, debuggers, libraries – that have
potential to hugely improve the efectiveness of the reasoning phase. For this reason, we adopt
tuProlog (2P) [
While the entire system is built around the aforementioned technologies, Python is used as the
glue language to keep modules together. In particular, the sub-symbolic module is implemented
via the PyTorch learning framework [
In this section we assess our model from the explainability perspective leveraging on the prototype implementation described in Section 4. Particularly, we train the sub-symbolic module through a real-world ML problem, and we test whether and to what extent our architecture actually provides (i) the capability of building a logical representation of sub-symbolic knowledge acquired from data via induction, (ii) the capability of altering or fixing the system behaviour via conditioning, and, ultimately, (iii) the inspectability and debuggability of the system as a whole. In the following subsections we set up an ad-hoc experiment aimed at performing these (a) Network default training (b) Decision tree built on test data tests. A detailed presentation of the experiments and their results follows.
The proposed experiment aims at comprehensively testing the operation of the prototype against some real classification problem. In particular, we consider a binary classification task on structured data. Roughly speaking, the experiment works by artificially constructing a buggy neural network for the classification task, and then showing how our architecture makes it possible to reveal the bug. More precisely, the experiment is structured as follows: 1. a neural network is trained until reaching maximum validation-set accuracy for the classification task; 2. by combining the training data and the predictions of the network, a coherent Prolog theory is extracted; 3. the induction module is used to extract the latent relations from the theory, until at least one relation which properly divides the data is found; 4. through constraining, we inject an error in the network, in such a way that it misclassify some instances; 5. by repeating Item 2 and Item 3, we show that the approximated logic theory reveals the injected error.
This workflow lets us verify the behaviour of our prototype and of all its components. In particular, Item 3 and Item 5 aim at demonstrating that a logic theory that optimally approximates a neural network is actually attainable. In terms of the ability of debugging and correcting a network, the whole procedure aims at demonstrating their feasibility. The extraction of the correct theory in the initial part of the experiment is comparable to the one extracted from a malfunctioning classifier. The inclusion of the fake constraint, as well as its recognition in the theory extracted at the conclusion of the experiment, shows the feasibility of the network correction process.
Two fundamental points are worth to be taken into account for the experiment to be meaningful. The first point is the ability to accurately evaluate the accuracy of the logic theory recovered. In fact, in order to verify that the theory extracted from the neural network is actually the correct one, it is either necessary (i) to have an optimal knowledge of the domain, or (ii) to use easily analysable and verifiable datasets. As for the first case – being an expert of the analysed domain – it turns out to be very easy, by verifying the correctness of the logical relations. The only alternative comes from the usage of an easily verifiable dataset as the base for the experiment. Indeed, it should be possible also for a domain novice to understand the ratio behind the data. For instance, a simple way to verify the correctness of the rules is to use an alternative ML tool – one that can guarantee a high level of transparency – to analyse the data. In the case of a classifier, the best choice could be a decision tree (DT). In fact, DTs training produces a symbolic model that can be eficiently verified by the user. Hence, DT output can be used as a reference in the evaluation of the induction results.
The second point is the exploitation of constraining as a means to inject bugs into a network in a controlled way. In fact, in order to verify the prototype capability of revealing and correcting bugs in the black-box module, it is first necessary to construct a buggy network. However, in case of a poorly training, as in that case, it can not be clear where the bug is. Hence, to evaluate the actual capabilities of the prototype, the bugs to be spotted must be a priori known in order to have a reference.
We base our experiment on the Congressional Voting Records data set1 from the UCI Repository
[
Given the relatively small amount of instances in the dataset, along with the small number of attributes, an intuitive understanding of the classification problem can be assumed. However, to further simplify the analysis of the results, we use a DT trained over the data as a reference.
A neural network capable of distinguishing between Democrats and Republicans based on voting intentions is trained and assessed as described above. The experiments code is available at the Github repository2. We now proceed discussing the results we obtained in each step of the experiment.
The training of the aforementioned neural network is performed via ordinary stochastic gradient descent until reaching a 95% accuracy score on the validation set (containing a randomly selected 20% of whole data). The training process is depicted in Figure 3a.
Through the induction process on the data generated by the network it was possible to recover the following relationships: (1, . . . , ) : ℎ (1, . . . , ), (1, . . . , ), (1, . . . , ). (1, . . . , ) : ℎ (1, . . . , ), (1, . . . , ). (1, . . . , ) : ℎ (1, . . . , ), (1, . . . , ). (1, . . . , ) : ℎ (1, . . . , ), (1, . . . , ), (1, . . . , ), (1, . . . , ).
To verify their validity we can examine the DT generated using the original data as source (Figure 3b)—the C4.5 algorithm has been used. As expected, the inductive process managed to recover the relationship that most discriminates between Democrats and Republicans: the tendency of the Democrats to be against the freezing of the cost of medical expenses.
The verification of the correction process is based on the reversal of the relationship retrieved above. Formally, the conditioning of the network occurred on this rule: (1, . . . , ) :
ℎ (1, . . . , ).
In natural language, this Prolog rule expresses that, given a set of votes (1, . . . , ), if these share the opposition to the freezing of medical expenses, then the voter is likely from the Republican Party. This constraint translates into a very simple result: all the votes should belong to Republicans. Indeed, if initially only the Democrats were against the freezing, by forcing the network to consider them as Republicans, we reach a situation where the whole set of voters is Republican.
More in detail, constraining rules are codified through the DSL ofered by the D2L implementation. For example, the above rule can be expressed in the form: dl2.Implication( dl2.EQ(x[FeesFreeze], 0), dl2.LT(y[Dem], y[Rep])) This rule is then automatically converted in a numerical function. Its evaluation contributes to the final loss function adopted by the target neural network.
For the experiment, a completely-new network is trained considering the new constraint. The results in Table 1 show the efect of the constraint on the network. The Democrats/Republican imbalance – which is made evident by Table 1 – reflects in the result of the induction process on the data generated by the constrained network: (1, . . . , ) : (1, . . . , ), ℎ (1, . . . , ). (1, . . . , ) : ℎ (1, . . . , ), (1, . . . , ), (1, . . . , ). (1, . . . , ) : ℎ (1, . . . , ), (1, . . . , ), (1, . . . , ), (1, . . . , ).
The inducted knowledge base only contains rules concerning the Republican wing, thus conifrming the footprint of the relation imposed during the conditioning process.
Summarising, the results of the assessment are positive. It was first possible to obtain the relationships implied in the operation of the classifier in a logic form. By these rules has been possible (i) to identify the more discriminant features in the dataset; (ii) to enable the correction of the black box using the user knowledge. The constraining part has also demonstrated efective. Indeed, the imposition of the user-crafted rule has led to a coherent change in the black-box behaviour—enabling its correction. The results demonstrate how the presented guidelines lead to an IS giving explanations about its functioning, thus allowing the user intervention on its behaviour.
The solutions proposed so far in the literature to the opaqueness issue – one of the main problem of today AI technologies – are disparate. In this work, we showed how symbolic logic can be a crucial element in this panorama.
On the trails of the NSC models, we presented a series of guidelines aimed at correctly integrating a ML-based predictor – i.e., a black box – with a logic-based subsystem. In particular, our guidelines support the creation of IS exposing clear insights about their own functioning, thus enabling end users to intervene on the IS behaviour via a logical interface. We then tested our guidelines against a prototype IS, in order to study if and to what extent our approach is feasible and useful. Notably, the prototype assessment confirms our approach is feasible exploiting technologies already available in the research scene. Nevertheless, the prototype has been tested only on a single scenario. In order to confirm the eficacy of our approach, we need to perform a more exhaustive range of experiments. Moreover, we plan to extend the prototype assessment with more complex use-cases. For example, as anticipated in Section 4, we intend to enhance the prototype with the support for unstructured data. This extension would considerably improve the applicability of the studied approach, allowing its assessment also on the more complex area of image classification.
Through the above experimental investigation – in the case of more positive results – we aim at introducing a rigorously formalised version of the proposed model—presented in this paper in a more intuitive and preliminary shape. Consequently, we should also better investigate and verify the preliminary guidelines provided. We aim at obtaining an accurate and comprehensive guide that would allow developers eficiently integrating opaque AI systems and logic.