Explaining Trained Neural Networks with Semantic Web Technologies: First Steps

Abstract

The ever increasing prevalence of publicly available structured data on the World Wide Web enables new applications in a variety of domains. In this paper, we provide a conceptual approach that leverages such data in order to explain the input-output behavior of trained arti cial neural networks. We apply existing Semantic Web technologies in order to provide an experimental proof of concept.

Introduction

Trained neural networks are usually imagined as black boxes, in that they do not give any direct indications why an output (e.g., a prediction) was made by the network. The reason for this lies in the distributed nature of the information encoded in the weighted connections of the network. Of course, for applications, e.g., safety-critical ones, this is an unsatisfactory situation. Methods are therefore sought to explain how the output of trained neural networks are reached.

This topic of explaining trained neural networks is not a new one, in fact there is already quite a bit of tradition and literature on the topic of rule extraction from such networks (see, e.g., [ 2,9,16 ]), which pursued very similar goals. Rule extraction, however, utilized propositional rules as target logic for generating explanations, and as such remained very limited in terms of explanations which are human-understandable. Novel deep learning architectures attempt to retrieve explanations as well, but often the use-case is only for computer vision tasks like object or scene recognition. Moreover, explanations in this context actually encode greater details about the images provided as input, rather than explaining why or how the neural network was able to recognize a particular object or scene.

Semantic Web [ 4,12 ] is concerned with data sharing, discovery, integration, and reuse. As eld, it does not only target data on the World Wide Web, but its methods are also applicable to knowledge management and other tasks o the Web. Central to the eld is the use of knowledge graphs (usually expressed using the W3C standard Resource Description Framework RDF [ 3 ]) and type logics attached to these graphs, which are called ontologies and are usually expressed using the W3C standard Web Ontology Language OWL [ 11 ].

This paper introduces a new paradigm for explaining neural network behavior. It goes beyond the limited propositional paradigm, and directly targets the problem of explaining neural network activity rather than the qualities of Copyright © 2017 for this paper by its authors. Copying permitted for private and academic purposes. the input. The paradigm leverages advances in knowledge representation on the World Wide Web, more precisely from the eld of Semantic Web technologies. It in particular utilizes the fact that methods, tool, and structured data in the mentioned formats are now widely available, and that the amount of such structured data on the Web is in fact constantly growing [ 5,18 ]. Prominent examples of large-scale datasets include Wikidata [ 22 ] and data coming from the schema.org [ 10 ] e ort which is driven by major Web search engine providers. We will utilize this available data as background knowledge, on the hypothesis that background knowledge will make it possible to obtain more concise explanations. This addresses the issue in propositional rule extraction that extracted rulesets are often large and complex, and due to their sizes di cult to understand for humans.

While the paper only attempts to explain input-output behavior, the authors are actively exploring ways to also explain internal node activations.

An illustrative example Let us consider the following very simple example which is taken from [ 14 ].

Assume that the input-output mapping P of the neural network without background knowledge could be extracted as p1 ^ q ! r

p2 ^ q ! r: Now assume furthermore that we also have background knowledge K in form of the rules p1 ! p

p2 ! p: The background knowledge then makes it possible to obtain the simpli ed inputoutput mapping PK , as

p ^ q ! r:

The simpli cation through the background knowledge is caused by p acting as a \generalization" of both p1 and p2. For the rest of the paper it may be bene cial to think of p, p1 and p2 as classes or concepts, which are hierarchically related, e.g., p1 being \oak," p2 being \maple," and p being \tree."

Yet this example is con ned to propositional logic.1 In the following, we show how we can bring structured (non-propositional) Semantic Web background knowledge to bear on the problem of explanation generation for trained neural networks, and how we can utilize Semantic Web technologies in order to generate non-propositional explanations. This work is at a very early stage, i.e., we will only present the conceptual architecture of the approach and minimal experimental results which are encouraging for continuing the e ort.

The rest of the paper is structured as follows. In Section 2 we introduce notation as needed, in particular regarding description logics which underly the OWL standard, and brie y introduce the DL-Learner tool which features prominently in our approach. In Section 3 we present the conceptual and experimental setup 1 How to go beyond the propositional paradigm in neural-symbolic integration is one of the major challenges in the eld [ 8 ]. for our approach, and report on some rst experiments. In Section 4 we conclude and discuss avenues for future work. 2

Preliminaries

We describe a minimum of preliminary notions and information needed in order to keep this paper relatively self-contained. Description logics [ 1,12 ] are a major paradigm in knowledge representation as a sub eld of arti cial intelligence. At the same time, they play a very prominent role in the Semantic Web eld since they are the foundation for one of the central Semantic Web standards, namely the W3C Web Ontology Language OWL [ 11,12 ].

Technically speaking, a description logic is a decidable fragment of rst-order predicate logic (sometimes with equality or other extensions) using only unary and binary predicates. The unary predicates are called atomic classes,2 while the binary ones are refered to as roles,3 and constants are refered to as individuals. In the following, we formally de ne the fundamental description logic known as ALC, which will su ce for this paper. OWL is a proper superset of ALC.

Desciption logics allow for a simpli ed syntax (compared to rst-order predicate logic), and we will introduce ALC in this simpli ed syntax. A translation into rst-order predicate logic will be provided further below.

Let C be a nite set of atomic classes, R be a nite set of roles, and N be a nite set of individuals. Then class expressions (or simply, classes ) are de ned recursively using the following grammar, where A denotes atomic classes from A and R denotes roles from R. The symbols u and t denote conjunction and disjunction, respectively.

C; D ::= A j :C j C u D j C t D j 8R:C j 9R:C

A TBox is a set of statements, called (general class inclusion) axioms, of the form C v D, where C and D are class expressions { the symbol v can be understood as a type of subset inclusion, or alternatively, as a logical implication. An ABox is a set of statements of the forms A(a) or R(a; b), where A is an atomic class, R is a role, and a; b are individuals. A description logic knowledge base consists of a TBox and an ABox. The notion of ontology is used in di erent ways in the literature; sometimes it is used as equivalent to TBox, sometimes as equivalent to knowledge base. We will adopt the latter usage.

We characterize the semantics of ALC knowledge bases by giving a translation into rst-order predicate logic. If is a TBox axiom of the form C v D, then ( ) is de ned inductively as in Figure 1, where A is a class name. ABox axioms remain unchanged.

DL-Learner [ 6,17 ] is a machine learning system inspired by inductive logic programming [ 20 ]. Given a knowledge base and two sets of individuals from the knowledge base { called positive respectively negative examples { DL-Learner 2 or atomic concepts 3 or properties (C v D) = (8x0)( x0 (C) ! x0 (D))

xi (A) = A(xi) xi (:C) = : xi (C) xi (C u D) = xi (C) ^ xi (D) xi (C t D) = xi (C) _ xi (D) xi (8R:C) = (8xi+1)(R(xi; xi+1) ! xi+1 (C)) xi (9R:C) = (9xi+1)(R(xi; xi+1) ^ xi+1 (C)) attempts to construct class expressions such that all the positive examples are contained in each of the class expressions, while none of the negative examples is. DL-Learner gives preference to shorter solutions, and in the standard setting returns approximate solutions if no fully correct solution is found. The inner workings of DL-Learner will not matter for this paper, and we refer to [ 6,17 ] for details. However, we exemplify its functionality by looking at Michalski's trains as an example, which is a symbolic machine learning task from [ 15 ], and which was presented also in [ 17 ].

For purposes of illustrating DL-Learner, Figure 2 shows two sets of trains, the positive examples are on the left, the negative ones are on the right. Following [ 17 ], we use a simple encoding of the trains as a knowledge base: Each train is an individual, and has cars attached to it using the hasCar property, and each car then falls into di erent categories, e.g., the top leftmost car would fall into the classes Open, Rectangular and Short, and would also have information attached to it regarding symbol carried (in this case, square), and how many of them (in this case, one). Given these examples and knowledge base, DL-Learner comes up with the class

9hasCar:(Closed u Short)

which indeed is a simple class expression such that all positive examples fall under it, while no negative example does. 3

Approach and Experiments

In this paper, we follow the lead of the propositional rule extraction work mentioned in the introduction, with the intent of improving on it in several ways. 1. We generalize the approach by going signi cantly beyond the propositional rule paradigm, by utilizing description logics. 2. We include signi cantly sized and publicly available background knowledge in our approach in order to arrive at explanations which are more concise.

More concretely, we use DL-Learner as the key tool to arrive at the explanations. Figure 3 depicts our conceptual architecture: The trained arti cial neural network (connectionist system) acts as a classi er. Its inputs are mapped to a background knowledge base and according to the networks' classi cation, positive and negative examples are distinguished. DL-Learner is then run on the example sets and provides explanations for the classi cations based on the background knowledge.

In the following, we report on preliminary experiments we have conducted using our approach. Their sole purpose is to provide rst and very preliminary insights into the feasibility of the proposed method. All experimental data is available from http://daselab.org/projects/human-centered-big-data.

We utilize the ADE20K dataset [ 23,24 ]. It contains 20,000 images of scenes which have been pre-classi ed regarding scenes depicted, i.e., we assume that the classi cation is done by a trained neural network.4 For our initial test, we used six images, three of which have been classi ed as \outdoor warehouse" scenes (our positive examples), and three of which have not been classi ed as such (our negative examples). In fact, for simplicity, we took the negative examples from among the images which had been classi ed as \indoor warehouse" scenes. The images are shown in Figure 4.

The ADE20K dataset furthermore provides annotations for each image which identify information about objects which have been identi ed in the image. The annotations are in fact richer than that and also talk about the number of objects, whether they are occluded, and some more, but for our initial experiment we only used presence or absence of an object. To keep the initial experiment simple, we furthermore only used those detected objects which could easily be mapped to our chosen background knowledge, the Suggested Upper Merged Ontology (SUMO).5 Table 1 shows, for each image, the objects we kept. The Suggested Upper Merged Ontology was chosen because it contains many, namely about 25,000 common terms which cover a wide range of domains. At the same 4 Strictly speaking, this is not true for the training subset of the ADE20K dataset, but that doesn't really matter for our demonstration. 5 http://www.adampease.org/OP/ image p1: road, window, door, wheel, sidewalk, truck, box, building image p2: tree, road, window, timber, building, lumber image p3: hand, sidewalk, clock, steps, door, face, building, window, road image n1: shelf, ceiling, oor image n2: box, oor, wall, ceiling, product image n3: ceiling, wall, shelf, oor, product time, the ontology arguably structures the terms in a relatively straightforward manner which seemed to simplify matters for our initial experiment.

In order to connect the annotations to SUMO, we used a single role called \contains." Each image was made an individual in the knowledge base. Furthermore, for each of the object identifying terms in Table 1, we either identi ed a corresponding matching SUMO class, or created one and added it to SUMO by inserting it at an appropriate place within SUMO's class hierarchy. We furthermore created individuals for each of the object identifying terms, including duplicates, in Table 1, and added them to the knowledge base by typing them with the corresponding class. Finally, we related each image individual to each corresponding object individual via the \contains" role.

To exemplify { for the image p1 we added individuals road1, window1, door1, wheel1, sidewalk1, truck1, box1, building1, declared Road(road1), Window(window1), etc., and nally added the ABox statements contains(p1; road1), contains(p1; window1), etc., to the knowledge base. For the image p2, we added contains(p2; tree2), contains(p2; road2), etc. as well as the corresponding type declarations Tree(tree2), Road(road2), etc.

The mapping of the image annotations to SUMO is of course very simple, and this was done deliberately in order to show that a straightforward approach already yields interesting results. As our work progresses, we do of course anticipate that we will utilize more complex knowledge bases and will need to generate more complex mappings from picture annotations (or features) to the background knowledge.

Finally, we ran DL-Learner on the knowledge base, with the positive and negative examples as indicated. DL-Learner returns 10 solutions, which are listed in Figure 5. Of these, some are straightforward from the image annotations, such as (1), (5), (8, (9) and (10). Others, such as (2), (4), (6), (7) are much more interesting as they provide solutions in terms of the background knowledge without using any of the terms from the original annotation. Solution (3) looks odd at rst sight, but is meaningful in the context of the SUMO ontology: SelfConnectedObject is an abstract class which is a direct child of the class Object in SUMO's class hierarchy. Its natural language de nition is given as \A SelfConnectedObject is any Object that does not consist of two or more disconnected parts." As such, the class is a superclass of the class Road, which explains why (3) is indeed a solution in terms of the SUMO ontology. (6) (7) (8) (9) (10)

We have conducted four additional experiments along the same lines as described above. We brie y describe them below { the full raw data and results are available from http://daselab.org/projects/human-centered-big-data.

In the second experiment, we chose four workroom pictures as positive examples, and eight warehouse pictures (indoors and outdoors) as negative examples. An example explanation DL-Learner came up with is

9contains:(DurableGood u :ForestProduct):

On of the outdoor warehouse pictures indeed shows timber. DurableGoods in SUMO include furniture, machinery, and appliances.

In the third experiment, we chose the same four workroom pictures as negative examples, and the same eight warehouse pictures (indoors and outdoors) as positve examples. An example explanation DL-Learner came up with is

8contains:(:Furniture u :IndustrialSupply);

i.e., \contains neither furniture nor industrial supply". IndustrialSupply in SUMO includes machinery. Indeed it turns out that furniture alone is insu cient for distingushing between the positive and negative exaples, because \shelf" is not classi ed as funiture in SUMO. This shows the dependency of the explanations on the conceptualizations encoded in the background knowledge.

In the fourth experiment, we chose eight market pictures (indoors and outdoors) as positive examples, and eight warehouse pictures (indoors and outdoors) as well as four workroom pictures as negative examples. An example explanation DL-Learner came up with is And indeed it turns out that people are shown on all the market pictures. There is actually also a man shown on one of the warehouse pictures, driving a forklift, however \man" or \person" was not among the annotations used for the picture. This example indicates how our approach could be utilized: A human monitor inquiring with an interactive system about the reasons for a certain classi cation may notice that the man was missed by the software on that particular picture, and can opt to interfere with the decision and attempt to correct it.

In the fth experiment, we chose four mountain pictures as positive examples, and eight warehouse pictures (indoors and outdoors) as well as four workroom pictures as negative examples. An example explanation DL-Learner came up with is

9contains:BodyOfWater:

Indeed, it turns out that all mountain pictures in the example set show either a river or a lake. Similar to the previous example, a human monitor may be able to catch that some misclassi cations may occur because presence of a body of water is not always indicative of presence of a mountain. 4

Conclusions and Further Work

We have laid out a conceptual sketch how to approach the issue of explaining arti cial neural networks' classi cation behaviour using Semantic Web background knowledge and technologies, in a non-propositional setting. We have also reported on some very preliminary experiments to support our concepts.

The sketch already indicates where to go from here: We will need to incorporate more complex and more comprehensive background knowledge, and if readily available structured knowledge turns out to be insu cient, then we foresee using state of the art knowledge graph generation and ontology learning methods [ 13,19 ] to obtain suitable background knowledge. We will need to use automatic methods for mapping network input features to the background knowledge [ 7,21 ], while the features to be mapped may have to be generated from the input in the rst place, e.g. using object recognition software in the case of images. And nally, we also intend to apply the approach to sets of hidden neurons in order to understand what their activations indicate.

Acknowledgements. This work was supported by the Ohio Federal Research Network project Human-Centered Big Data.

References

1. Baader , F. , Calvanese , D. , McGuinness , D. , Nardi , D. , Patel-Schneider , P.F . (eds.): The Description Logic Handbook: Theory, Implementation, and Applications . Cambridge University Press, 2nd edn. ( 2010 )

2. Bader , S. , Hitzler , P. : Dimensions of neural-symbolic integration { A structured survey . In: Artemov, S.N. , Barringer , H., d'Avila Garcez , A.S. , Lamb , L.C. , Woods , J . (eds.) We Will Show Them! Essays in Honour of Dov Gabbay , Volume One. pp. 167 { 194 . College Publications ( 2005 )

3. Beckett , D. , Berners-Lee , T. , Prud 'hommeaux, E., Carothers , G.: RDF 1.1. Turtle { Terse RDF Triple Language . W3C Recommendation (25 February 2014 ), available at http://www.w3.org/TR/turtle/

4. Berners-Lee , T. , Hendler , J. , Lassila , O. : The Semantic Web . Scienti c American 284 ( 5 ), 34 {43 (May 2001 )

5. Bizer , C. , Heath , T. , Berners-Lee , T. : Linked Data { The Story So Far . International Journal on Semantic Web and Information Systems 5 ( 3 ), 1 { 22 ( 2009 )

6. Buhmann, L., Lehmann , J. , Westphal , P. : DL-Learner { A framework for inductive learning on the semantic web . Journal of Web Semantics 39 , 15 { 24 ( 2016 )

7. Euzenat , J. , Shvaiko , P. : Ontology Matching, Second Edition . Springer ( 2013 )

8. Garcez , A. , Besold , T., de Raedt , L. , Foldiak, P. , Hitzler , P. , Icard , T. , Kuhnberger, K.U., Lamb , L. , Miikkulainen , R. , Silver , D. : Neural-symbolic learning and reasoning: Contributions and challenges . In: Gabrilovich, E. , Guha , R. , McCallum , A. , Murphy , K . (eds.) Proceedings of the AAAI 2015 Spring Symposium on Knowledge Representation and Reasoning: Integrating Symbolic and Neural Approaches. Technical Report , vol. SS- 15 - 03 . AAAI Press, Palo Alto, CA ( 2015 )

9. d'Avila Garcez , A.S. , Zaverucha , G. : The connectionist inductive lerarning and logic programming system . Applied Intelligence 11 ( 1 ), 59 { 77 ( 1999 )

10. Guha , R.V. , Brickley , D. , Macbeth , S. : Schema. org: evolution of structured data on the web . Commun. ACM 59 ( 2 ), 44 { 51 ( 2016 )

11. Hitzler , P. , Krotzsch, M. , Parsia , B. , Patel-Schneider , P.F. , Rudolph , S. (eds.): OWL 2 Web Ontology Language Primer (Second Edition) . W3C Recommendation (11 December 2012 ), http://www.w3.org/TR/owl2-primer/

12. Hitzler , P. , Krotzsch, M. , Rudolph , S. : Foundations of Semantic Web Technologies . CRC Press/Chapman & Hall ( 2010 )

13. Ji , H. , Grishman , R. : Knowledge base population: Successful approaches and challenges . In: Lin, D. , Matsumoto , Y. , Mihalcea , R . (eds.) The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , Proceedings of the Conference , 19 - 24 June, 2011 , Portland, Oregon, USA. pp. 1148 { 1158 . The Association for Computer Linguistics ( 2011 )

14. Labaf , M. , Hitzler , P. , Evans , A.B. : Propositional rule extraction from neural networks under background knowledge . In: Proceedings of the Twelfth International Workshop on Neural-Symbolic Learning and Reasoning , NeSy' 17 , London, UK, July 2017 ( 2017 ), to appear

15. Larson , J., Michalski, R.S.: Inductive inference of VL decision rules . SIGART Newsletter 63 , 38 { 44 ( 1977 )

16. Lehmann , J. , Bader , S. , Hitzler , P. : Extracting reduced logic programs from articial neural networks . Applied Intelligence 32 ( 3 ), 249 { 266 ( 2010 )

17. Lehmann , J. , Hitzler , P. : Concept learning in description logics using re nement operators . Machine Learning 78 ( 1-2 ), 203 { 250 ( 2010 )

18. Lehmann , J. , Isele , R. , Jakob , M. , Jentzsch , A. , Kontokostas , D. , Mendes , P.N. , Hellmann , S. , Morsey , M., van Kleef, P. , Auer , S. , Bizer , C. : DBpedia { A largescale, multilingual knowledge base extracted from Wikipedia . Semantic Web 6 ( 2 ), 167 { 195 ( 2015 )

19. Lehmann , J. , Volker, J.: Perspectives on Ontology Learning, Studies on the Semantic Web , vol. 18 . IOS Press ( 2014 )

20. Muggleton , S. , Raedt , L.D.: Inductive logic programming: Theory and methods . Journal of Logic Programming 19 /20, 629{ 679 ( 1994 )

21. Uren , V.S. , Cimiano , P. , Iria , J. , Handschuh , S. , Vargas-Vera , M. , Motta , E. , Ciravegna , F. : Semantic annotation for knowledge management: Requirements and a survey of the state of the art . J. Web Sem . 4 ( 1 ), 14 { 28 ( 2006 )

22. Vrandecic , D. , Krotzsch, M.: Wikidata: a free collaborative knowledgebase . Commun. ACM 57 ( 10 ), 78 { 85 ( 2014 )

23. Zhou , B. , Zhao , H. , Puig , X. , Fidler , S. , Barriuso , A. , Torralba , A. : Semantic understanding of scenes through the ADE20K dataset . arXiv preprint arXiv:1608.05442 ( 2016 )

24. Zhou , B. , Zhao , H. , Puig , X. , Fidler , S. , Barriuso , A. , Torralba , A. : Scene parsing through ADE20K dataset . In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition ( 2017 )