Vector space embeddings can be used as a tool for learning semantic relationships from unstructured text documents. Among others, earlier work has shown how in a vector space of entities (e.g. di erent movies) ne-grained semantic relationships can be identi ed with directions (e.g. more violent than). In this paper, we use stacked denoising auto-encoders to obtain a sequence of entity embeddings that model increasingly abstract relationships. After identifying directions that model salient properties of entities in each of these vector spaces, we induce symbolic rules that relate speci c properties to more general ones. We provide illustrative examples to demonstrate the potential of this approach.
In this paper, we consider the problem of how we can learn symbolic rules from unstructured text documents that describe entities of interest, e.g. how we can learn that thrillers tend to be violent from a collection of movie reviews. Obtaining meaningful and interpretable symbolic rules is important in elds like exploratory data analysis, or explaining classi er decisions, as they can be interpreted easily by human users.
A straightforward approach might be to directly learn rules from
bag-ofwords representations of documents. However, such an approach would typically
lead to a large number of rules of little interest, e.g. rules pertaining more to
which words are used together rather than capturing capturing meaningful
semantic relationships. Our approach instead builds on the method from [
nds directions which correspond to interpretable properties in a vector space, labelled using adjectives and nouns that appear in the text collection. In particular, these directions induce a ranking of the entities that re ects how much they have the corresponding property. For example, in a space of wines, a direction may be found that corresponds to the property of being \Tannic", allowing us to rank wines based on the number of tannins.
In order to obtain symbolic rules, we rst derive a series of increasingly general entity embeddings using auto-encoders (see Section 3). To induce rules from embeddings, we link properties derived from those embeddings together.
Copyright © 2016 for this paper by its authors. Copying permitted for private and academic purposes.
As an example, below is one of the rules we have derived using this method:
(1)
Using a set of symbolic rules that qualitatively describe domain knowledge is a
promising approach to generate supporting explanations. Explanations of
classication decisions can give valuable insight into why a system produces a result.
For example, in elds such as medicine it is important for experts to verify the
predictions of a system and justify its classi cation decisions [
The rest of the paper explains how we use unsupervised methods to learn
rules such as (1). In Section 2, we recall the method from [
In this section, we recall the method from [
From bags-of-words to vectors. We use a text collection where each document
describes an entity. For example, if the entities are movies, a collection of movie
reviews. We rst learn a vector space of entities using classical multidimensional
scaling (MDS), which takes a dissimilarity matrix as input. MDS is commonly
used in cognitive science to generate semantic spaces from similarity judgements
that are provided by human annotators. It outputs a space where entities are
represented as points and the Euclidean distance between entities re ects the
given dissimilarity matrix as closely as possible. It was empirically found to lead
to representations that are easier to interpret than the more commonly used
singular value decomposition method [
Identifying directions for frequent terms. To discover terms that correspond
to interpretable properties in the MDS space, the nouns and adjectives that
occur in su ciently many reviews are used as the input to a linear Support
Vector Machine (SVM). The SVM is trained to nd the hyperplane that best
separates the entities that contain the term at least once in their associated
textual description. To accommodate class imbalance, we increase the cost of
positive instances such that their weight is inversely proportional to how many
times the term has occurred. To assess the quality of the hyperplane found by the
SVM, we use Cohen's Kappa score [
Identifying saleint properties by clustering directions. It can sometimes be ambiguous as to what property each term is referring to. For example, it is unclear whether \mammoth" refers to the animal or an adjective meaning large. In this paper, we have chosen the number of clusters equal to the number of dimensions. To determine the cluster centers, we rst select directions whose associated Kappa score is above some threshold T +. We use the highest scoring direction as the center of the rst cluster and nd the most dissimilar direction to the rst cluster's direction to get the centre of the second cluster. Continuing in this way, we repeatedly select the direction which is most dissimilar to all previously selected clusters. By doing so, we obtain a collection of cluster centres that capture a wide variety of di erent properties from the space. We then associate each remaining direction to its most similar cluster centre. In this step, we consider directions whose associated Kappa score is at least T , where typically T < T +. Finally, we take the average of all directions in a cluster to be the overall direction for a cluster. The value of T + should be chosen as large as possible (given that the terms with the highest Kappa scores are those which are best represented in the space), while still ensuring that we can avoid choosing cluster centers which are too similar. Choosing the value of T represents a trade-o . A cluster of terms is often easier to interpret than a single term, which means that we shouldn't choose T to be too high. On the other hand, choosing T to be too low would result in poorly modelled terms being added to clusters. For example, we would not want to term \Bee" to be added to the cluster for \Emotional", even though the direction for \Bee" is closest to that cluster.
Note that as each cluster produced by the above procedure is associated with a direction, it induces a ranking of the entities. This gives us two ways to disambiguate which properties a term is referring to: the rst being examining which terms it shares its cluster with e.g. we know that \Mammoth" refers to the adjective because it is shared with \Epic", \Stupendous", and \Majestic", and the second being examining which entities score highly in the rankings for a cluster direction e.g. \Monster" de nes a ranking in which \Frankenstein" and \The Wolfman" appear among the top ranked movies. 3
In this section, we explain how we obtain a series of increasingly general entity embeddings, and how we can learn symbolic rules that link properties from subsequent spaces together.
To construct more general embeddings from the initial embedding provided
by the MDS method, we use stacked denoising auto-encoders [16]. Standard
autoencoders are composed of an \encoder" that maps the input representation into
a hidden layer, and a \decoder" that aims to recreate the input from the hidden
layer. Auto-encoders are normally trained using an objective function that
minimizes information loss (e.g. Mean Squared Error) between the input and output
layer [
The main novelty of our approach is that we characterize the salient
properties (i.e. clusters of directions) modelled in one space in terms of salient
properties that are modelled in another space. Speci cally, we use the o -the-shelf
rule learner JRip [
We base our experiments on the movie review text collection of the 15,000 top
scoring movies on IMDB1 made available by [
To implement the denoising auto-encoders, we use the Keras [
In Table 1, we illustrate the di erences between clusters obtained using standard auto-encoders and denoising auto-encoders. Layer 1 refers to the hidden representation of the rst auto-encoder, and Layer 4 refers to the hidden representation of the nal auto-encoder. As single labels can lead to ambiguity, in Table 1 we label clusters using the top three highest scoring terms in the cluster. Clusters are arranged from highest to lowest Kappa score.
Both auto-encoders model increasingly general properties, but the properties obtained when using denoising auto-encoder properties are more general. For example, the normal auto-encoder contains properties like \Horror" and \Thriller", but does not contain more general properties like \Society" and \Relationship". Further, \Gore" has the most similar properties \Zombie" and \Zombies" in Layer 1, and has the most similar properties of \Budget" and \E ects" in Layer 4. By representing a category of movie where \Budget" and \E ects" are important, the property is more general. 1 http://www.cs.cf.ac.uk/semanticspaces/ 2 https://github.com/eygrr/RulesFromAuto-encoders 4.3
Our aim in this work is to derive symbolic rules that can be used to explain the semantic relationships between properties derived from increasingly general entity embeddings. We provide examples of such rules in this section. Since the number of all induced rules is large, here we only show high accuracy rules that cover 200 samples or more. Still, we naturally cannot list even all the accurate rules covering more than 200 samples. Therefore we focus here on the rules which are either interesting in their own right or exhibit interesting properties, strengths or limitations of the proposed approach. The complete list of induced rules is available online from our GitHub repository3.
For easier readability, we post-process the induced rules. For instance, the following is a rule obtained for the property \Gore" in the third layer of the network shown in the original format produced by JRip: IF scares-L2 <= 6 AND blood-L2 <= 8 AND funniest-L2 >= 22 => classification=+ (391.0/61.0) In this rule, scares-L2 <= 6 denotes the condition that the movie is in the top 6% of rankings for the property \scares" derived from the hidden representation of the second auto-encoder. We will write such conditions simply as \Scares2". Similarly, a condition such as funniest-L2 >= 22, which indicates that the property is not in the top 22%, will be written as NOT Funniest2. In this simpler notation the above rule will look as follows:
This rule demonstrates an interpretable relationship. However, we have observed that the meaning of a rule may not be clear from the property labels that are automatically selected. In such cases, it is bene cial to label them by including the most similar cluster terms. For example, using the cluster terms below we can see that \Flick" relates to \chick- icks" and that \Amazon" relates to old movies:
Rules derived from later layers use properties described by rules from previous layers. By seeing rules from earlier layers that contain properties in later layers, we can better understand what the components of later rules mean. Below, we have provided rules to explain the origins of components in a later rule:
3 https://github.com/eygrr/RulesFromAuto-encoders
We observe a general trend that as the size of the representations decreases and the entity embeddings become smaller, rules have fewer conditions, resulting in overall higher scoring and more interpretable rules. To illustrate this, we compare rules from an earlier layer to similar rules in a later layer:
Rules in later layers also made e ective use of a NOT component. Below, we demonstrate some of those rules:
As the same terms were used to nd new properties for each space, the obtained rules sometimes use duplicate property names in their components. As the properties from later layers are a combination of properties from earlier layers, the properties in later layers are re nements of the earlier properties, despite having the same term. Below, we provide some examples to illustrate this:
The work presented in this paper di ers from existing works in that it focuses on inducing rules which involve salient and interpretable features from unstructured text documents.
The existing neural network rule extraction algorithms can be categorized as
either decompositional, pedagogical or eclectic [
The algorithm in [
The main di erence in our work is that our method induces rules from properties derived from the layers of a network, rather than learning rules that describe the relationships between units in the network itself. Additionally, we focus on learning increasingly general entity embeddings from hidden representations rather than tuning network parameters such that weights directly relate to good rules.
Another recent topic that relates to our work is improving neural networks
and entity embeddings using symbolic rules [
This is essentially the opposite of the task we consider in this paper: using embeddings to learn better rules. The rules that we derive are not intended to explain how the network functions but rather to describe the semantic relationships that hold in the considered domain. In other words, our aim is to use the neural network representations in the hidden layer as a tool for learning logical domain theories, where the focus is on producing rules that capture meaningful semantic relationships. 6
In this paper, we have shown how we can obtain increasingly general entity embeddings from stacked denoising auto-encoders, and how we can obtain rules from those embeddings that capture domain knowledge. We have qualitatively evaluated the obtained rules to demonstrate the semantic relationships that they capture. The results show the potential of the method for exploratory analysis of collections of unstructured text documents and explaining decisions of classi ers. Acknowledgement. This work was supported by ERC Starting Grant 637277. 16. P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol. Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th international conference on Machine learning, pages 1096{1103, 2008.