The success of the Semantic Web research is dependent upon the construction of complete and reliable domain ontologies. In this paper we describe an unsupervised framework for domain ontology enrichment based on mining domain text corpora. Specifically, we enrich the hierarchical backbone of an existing ontology, i.e. its taxonomy, with new domain-specific concepts. The framework is based on an extended model of hierarchical self-organizing maps. As being founded on an unsupervised neural network architecture, the framework can be applied to different languages and domains. Terms extracted by mining a text corpus encode contextual content information, in a distributional vector space. The enrichment behaves like a classification of the extracted terms into the existing taxonomy by attaching them as hyponyms for the nodes of the taxonomy. The experiments reported are in the “Lonely Planet” tourism domain. The taxonomy and the corpus are the ones proposed in the PASCAL ontology learning and population challenge. The experimental results prove that the quality of the enrichment is considerably improved by using semantics based vector representations for the classified (newly added) terms, like the document category histograms (DCH) and the document frequency times inverse term frequency (DF-ITF) weighting scheme.
The most important prerequisite for the success of the Semantic Web research is the construction of complete and reliable domain ontologies. Building ontologies is still a time consuming and complex task, requiring a high degree of human supervision and being still a bottleneck in the development of the semantic web technology.
The process of domain ontology enrichment has two inputs, an existing ontology –
which plays the role of background knowledge – and a domain text corpus. The aim
of our work is to automatically adapt the given ontology according to a domain
specific corpus. We enrich the hierarchical backbone of the existing ontology, i.e. its
taxonomy, with new domain-specific concepts extracted from the corpus [
Our framework for taxonomy enrichment is based on an extended model of hierarchical self-organizing maps, which represent an unsupervised neural network architecture. The candidates for labels of newly inserted concepts are terms collected by mining a text corpus. The term extraction process is based on recognizing linguistic patterns (noun phrases) in the domain corpus documents. Each term encodes contextual content information, in a distributional vector space. The context features of a term are the frequencies of its occurrence in different documents of the corpus. The classification of the extracted terms into the taxonomy of the given ontology proceeds by associating every term to one target node of the taxonomy, based on a similarity in the distributional vector space. That term becomes a new concept added to the taxonomy, and it is attached as hyponym (successor) under the target node.
Unsupervised hierarchical neural models in general start the growing of a dynamic
tree-like topology from a single initial node. Our neural network model, called
Enrich-GHSOM, is an extension of one of these existent systems, GHSOM [
In the rest of the paper, after a review of related work, section 3 presents the neural network learning solution chosen and adapted in our framework. Then section 4 details the architecture and implementation of the taxonomy enrichment framework and section 5 describes the experimental results. Conclusions and future directions are presented in section 6.
There are two main categories of approaches for taxonomy enrichment [
In the term classification approach, the terms extracted from a domain specific
corpus of text are classified into an existent taxonomy [
The top-down classification behavior in our framework is modeled by a growing
hierarchical self-organizing map (GHSOM) architecture [
A similar, although non top-down approach is [
Our extended model of hierarchical self-organizing maps – Enrich-GHSOM – represents the unsupervised neural network based learning solution adopted by our taxonomy enrichment framework. This choice is suitable to the knowledge structure to be enriched – a taxonomy, i.e. an is-a hierarchy of concepts.
GHSOM is an extension of the Self-Organizing Map (SOM, also known as Kohonen
map) learning architecture [
The SOM map learns by a self-organization process. There is no initial knowledge about the membership of any input data item in a particular class or about the number of classes. The training proceeds with unlabeled input data like any unsupervised learning. Clusters (classes) are discovered and described by gradually detected characteristics during the training process. These gradually adjusted characteristics play the role of weights in the weight vector associated to each neuron. The role of a completely trained map is to represent all the available observations – the whole input data space – with optimal accuracy by using a restricted set of weight vectors associated to the map neurons.
The initial values for the weight vectors of the neurons can either be chosen depending on the problem domain or they can be taken randomly. Every iteration of the learning algorithm processes one input (training) vector as follows. Like usually for unsupervised neural networks, some form of competitive learning takes place: the winner neuron index c, which best matches the current input vector, is identified as the neuron whose weight vector is most similar to the current input vector in some metric. Then all the weight vectors or a subset of them that correspond to neurons centered around the winner neuron c – i.e. neurons in the neighborhood area of c –, including the winner itself, are updated in the direction of the input vector. This adaptation renders a globally ordered map in the process of learning. A neuron has four immediate neighbors in a rectangular map topology, which is our chosen map topology. This is merely a rectangular lattice type of the two-dimensional grid of neurons, and the SOM map is kept as a planar rectangle.
Data spaces contain some latent structuring in the form of clusters. SOM maps can discover and illustrate this clustering. However, some hierarchical structures are also latent in data sets. To give an interesting example in the present context, a thesaurus is a data space consisting of terms in a language, represented as a lexical database. The main relation between the terms in a thesaurus is the taxonomic relation. However, because of their essentially flat topology, SOM maps have a limited capability to discover and illustrate hierarchical clusters in data sets. A solution for this problem is represented by the hierarchical SOM maps.
The growing hierarchical self-organizing map model consists of a set of SOM
maps arranged as nodes in a hierarchy and it is able to discover hierarchical clusters
[
The SOM’s in the nodes can also grow vertically during the training, by giving rise to successor nodes. Each neuron in the SOM map could be a candidate for expansion into a successor node SOM map (see Fig. 1). The expansion takes place whenever the data deviation on the current neuron is over a threshold 2. This sounds like a zoom into the data subspace mapped into the parent neuron, because the successor SOM map is trained merely with data items in that subspace. Further node expansions continue recursively on successor nodes, and the training of the whole GHSOM model finally stops (converges) when both thresholds are satisfied. The training begins with a single-neuron SOM map having the whole input data set mapped into its only neuron. This becomes the root of the final, completely trained GHSOM model.
The thresholds 1 and 2 control the granularity of the hierarchy learned by GHSOM in terms of depth and branching factor. A low 1 with a much lower 2 leads to a deep hierarchy with an increased number of neurons into the SOM nodes, and consequently an increased branching factor also. A high 1 with a lower 2 leads to deep hierarchies with small SOM nodes (with few neurons), and consequently a reduced branching factor corresponding to the reduced number of neurons in SOM nodes. When both thresholds are low and comparable, then the hierarchy is flat with a high branching factor. If both thresholds are high and comparable, then the hierarchy is flat with a low branching factor.
Each level in a learned GHSOM model displays a more detailed clustering of the data space as compared to the parent level. This corresponds to a top-down process of hierarchical clustering of the input data space items.
The growth of a GHSOM is a completely unsupervised process, being only driven by the unlabeled input data items themselves together with the two thresholds and some additional learning parameters. There is no way to suggest from outside any initial paths for the final learnt hierarchy. We have extended the GHSOM model with the possibility to force the growth of the hierarchy along with some predefined paths of a given hierarchy. Our new extended model, Enrich-GHSOM, is doing a classification of the data items into an existing taxonomic structure. This initial tree plays the role of an initial state for the tree-like neural network model. The classical GHSOM model grows during the training by only starting from a single node. The top-down growth in our extended model starts from a given initial tree structure and inserts new nodes attached as successors to any of its intermediate and leaf nodes.
In Enrich-GHSOM, the nodes of the predefined hierarchy are labeled with some data item labels from the input data space used for training. The training data items propagate top-down throughout the given tree hierarchy structure. When the propagation process hits a parent SOM of a tree node, then the weight vector of the corresponding parent neuron in that parent SOM is initialized with the data item vector of that successor node label. The weight vectors of the SOM neurons with no successor are initialized with random values. Then the training of that SOM proceeds by classifying the training data items against the initialized neurons. Training data items that are similar (distributionally similar as vectors) to the predefined initialized neurons are propagated downwards to the associated successor SOM nodes to continue the training (recursively) on that predefined successor SOM. Data items that are not similar to the initialized neurons are mapped to other, non-initialized, neurons in the same SOM, and they are not propagated downwards into the predefined hierarchy. They remain as mapped into that SOM, and are considered as classified into the parent neuron of that SOM, i.e. as successor of that parent.
For instance, consider the parent neuron of a current SOM node is labeled mammal, and there are two predefined successor nodes labeled feline and bear, which correspond to two predefined initialized neurons in the current SOM. Then the training data item vector dog is not similar to any of the two neuron initializer weight vectors associated to feline and bear (see Fig. 2, where the neuron initializers are marked with bold). So dog will remain as classified into that SOM – mapped on another, non-initialized neuron – i.e. as successor (hyponym) of mammal and twin of the existent nodes feline and bear. Also, a data item labeled tiger – similar with the weight vector of the predefined “feline” neuron – will be propagated into the associated predefined successor SOM map together with other terms that correspond to felines, which will all become direct or indirect hyponyms of the concept feline. The process continues top-down for all the SOM nodes in the predefined initial tree hierarchy, ending at the leaves. The data item vector representations of the labels of the given initial tree play the role of predefined initializer weight vectors of our neural model.
The architecture of our framework is implemented as a pipeline with several linguistic and machine learning processing stages. The whole processing can be divided in two main steps: the term extraction step and the taxonomy enrichment step.
The candidates for the labels of new concepts inserted during the taxonomy
enrichment are terms representing noun phrases, identified by mining the domain text
corpus. In order to identify the terms by a linguistic analysis of the corpus documents,
our framework relies on several processing resources offered by the ANNIE module
for analyzing English texts in the GATE framework [
The terms extracted from the domain text corpus are mapped to classes (concepts) of the existing taxonomy. The taxonomy enrichment algorithm proceeds by “populating” the given taxonomy with the terms collected from the corpus. The Enrich-GHSOM neural network drives a top-down hierarchical classification of the terms along with the given taxonomy branches and inserts new nodes (concepts) corresponding to these classified terms. Every new concept is attached as successor of an intermediate or a leaf node of the given taxonomy and becomes a hyponym of that node.
In order to use our Enrich-GHSOM neural network to induce such a taxonomy enrichment behavior, a symbolic-neural translation is first done by parsing a textual representation of the initial taxonomy (is_a(concept, superconcept) assertions or OWL format). The result of this parsing is the initial internal tree-like state of the neural network. In order for the initialized network to be able to classify terms into this initial taxonomic structure, apart from the vector representation of the classified terms, a representation as a numerical vector is also needed for each node in the initial taxonomy. This vector plays the role of initial weight vector for the neural network (see section 3.3). It is the vector representation for the noun phrase concept label associated to the node, computed as will be described in section 4.3. The acquisition of this vector takes place in the same way as the acquisition of the vector representation of the classified terms (section 4.3).
We assume that the concept labels of the initial taxonomy are terms – noun phrases – extractable from the domain text corpus from which the classified terms themselves have also been extracted. Their vectors are then computed in the same way as the vectors of all the corpus extracted terms which are classified during the enrichment. Using the same corpus from a specialized domain to acquire the feature vectors of the concepts in the initial taxonomy and the terms to be classified is a reasonable choice, since it will reduce the problems with ambiguous (multiple) senses of one and the same term.
Since Enrich-GHSOM is a connectionist system, the terms classified by
EnrichGHSOM and the concepts of the given taxonomy have to be represented as vectors. In
our framework, the attributes (features) of the vector representation of a term or
concept encode contextual content information, in a distributional vector space.
Specifically, the context features are the frequencies of the occurrence of the term –
classified term or concept label term – in different documents of the corpus. The
number of component attributes of such a term vector coincides with the number of
documents of the text corpus out of which all the terms have been extracted. Every
attribute in the vector of a term is essentially the number of occurrences of the term in
one document. This representation is inspired from the latent semantic analysis [
The vector representation in the current framework satisfies Harris’ distributional
hypothesis [
The framework allows multiple ways to encode the frequencies of occurrence,
starting from simple flat counts of occurrences. Another variant is the DF-ITF
weighting scheme, which means “document frequency times inverse term frequency”.
We propose this weighting scheme, which is a transposed of TF-IDF [
A third way to encode the vector representation is one in which we propose the
vector to be a document category histogram (DCH). Specifically, first a SOM [
Data Sparseness. The dimensionality reduction achieved by using the document
category histogram (DCH) representation is important since it removes the semantic
noise caused by minor differences in semantic content for different corpus documents.
Such documents now belong together to the same semantic category. This intuition is
already confirmed by our experiments reported in previous work [
A source of data sparseness is represented by terms with very few occurrences in the text corpus. Among such terms are the most generic terms that label the roots of the main trees in a given initial taxonomy and usually the concepts which are very high in a taxonomy. When in the Enrich-GHSOM neural network such an overly generic term with a very sparse vector labels the concept of one of the roots, and also when using the flat count vector representation instead of the histogram representation, then the main tree rooted by that concept is unable to attract and classify a relevant quantity of training terms. Thus the top-down search during the classification is misled. It is the case of the root concepts spatial_concept, intangible, and thing in the ontology of the “Lonely Planet” tourism dataset used in the present experiments. Some of the branches of these main trees are populated by no training term, which leads to the starvation of the neural network. Starvation means that the neural network enters an infinite loop when trying to tune the quantization error on a neuron below the thresholds (see section 3.2). Many of our experiments which used a flat count vector representation failed by starvation. As opposed, all the experiments using the reduced, histogram vector representation (DCH) converged to a result.
A way of reducing the number of zeros in the vector representation of the generic
terms that label the generic concepts in the initial taxonomy is the centroid vector [
In order for the corpus extracted terms to actually become domain specific concepts,
they have to be noun phrases with enough frequency of occurrence in the domain
specific corpus. In the term extraction process, we have set a threshold for the
extracted noun phrases to occur in at least 0.5% of the number of documents in the
corpus. Having set this threshold, we have extracted and acquired the corresponding
numerical vector representations for 1241 noun phrases. These extracted terms are
classified against the taxonomy of a tourism ontology consisting of 73 concepts,
which is proposed in the PASCAL ontology learning and population challenge [
The evaluation of the enrichment means evaluating the quality of the mapping
from corpus extracted terms into target concepts of the given initial taxonomy. An
extracted term becomes a new concept added to the taxonomy, and it is attached as
hyponym (successor) under its associated target node. In order to evaluate the
taxonomy enrichment, we followed a cross-validation strategy [
The most appropriate measure for evaluating the taxonomy enrichment task is the
learning accuracy, defined and evaluated in [
For a given classified term i, if pi is the target concept assigned (predicted) by the system, and ci the correct target concept according to the given initial taxonomy, the learning accuracy is the average over all the classified terms i of the function LA(pi, ci), where the function LA is defined as
(top, a) + 1 LA(p, c) = ------------------------------------
(top, a) + (a, c) + (a, p) + 1
top is the root of the taxonomy, and a is the least common subsumer of the concepts p
and c (i.e. the most specific common hypernym of p and c). (a, b) is the taxonomic
distance between the concepts a and b, i.e. the number of taxonomy edges to be
traversed when going from the taxonomy node labeled a towards node b. This is the
most used formula to compute the learning accuracy. In the context of the Pascal
ontology learning and population challenge, it is actually called symmetric learning
accuracy, and the term learning accuracy is used for a historically initial version of
the learning accuracy measure, as introduced by [
According to formulae (1) and (2) to compute both variants of the learning accuracy, the same number of edges in the taxonomic distance between the predicted and the correct target concept means a better accuracy when the edges are lower in the taxonomy. This is due to the intuition that the same number of edges between two concrete (lower in the taxonomy) concepts means an increased similarity (a reduced semantic distance), as compared to the same number of edges between two abstract concepts (higher in the taxonomy).
Another quantitative evaluation measure similar in spirit to the learning accuracy is the edge measure. It actually counts the average deviation (in terms of taxonomic distance) between the system predicted target concept and the correct one according to the given initial taxonomy. Consequently, as opposed to the first two learning accuracy measures (formulae (1) and (2)), the edge measure means a better classification for a lower edge measure value.
A first set of experimental runs is based on a document category histogram (DCH)
vector representation for the extracted terms and concept label terms. Also, the
concept label terms of the given initial taxonomy are represented using the centroid
method for the whole sub-tree of a given concept node, as described in section 4.3.
The improvements gained by using DCH and centroid are already confirmed
qualitatively by our experiments reported in [
In a second set of experiments, we first applied the DF-ITF weighting scheme on the flat count term vectors of 1801 attributes. The result vectors were then converted into DCH histograms, thus reducing the term vector dimension to 179.
[
We evaluated the three learning accuracy measures on placing the 43 concepts in
their actual position in the given initial ontology from the Pascal challenge [
Another finding is that limiting the depth of the sub-concepts for the computation
of the centroid vector representation for taxonomy concepts leads to a slight
degradation of the learning accuracy. The experiments in [
Named Entity Classification. In a last set of experiments, instead of classifying terms represented by common noun phrases extracted from the “Lonely Planet” corpus, we rather classified noun phrases for proper names – i.e. named entities – extracted from the same corpus. The majority of the named entities occur few times in the corpus, and many of them only occur once, in a singe document. This is why, in the experiments reported in what follows, we have reduced the frequency threshold to zero. It was 0.5% in the preceding experiments (see section 5.1).
Having no more frequency threshold for the corpus extracted noun phrases, we
found and extracted a total of 43006 noun phrases, compared to 1241 in the preceding
three taxonomy enrichment experiments. Some of them are common nouns and the
other are named entities. We will refer in what follows to this experiment as the
maximal experiment. To reduce the dimensionality of the data, and consequently the
inherent noise, one of our experiments was trying to keep only what is absolutely
necessary for the classification. We kept a minimum of common noun phrases
corresponding to the concept labels in the taxonomy, and a minimum of proper noun
phrases representing the set of named entities asked to be classified in the PASCAL
ontology learning and population challenge [
We evaluated these last experiments automatically by using the PASCAL
challenge site online evaluation system1. This evaluation system is based on a gold
standard, i.e. an ontology populated with the set of named entities that are asked to be
classified in the PASCAL challenge. In other words, the PASCAL competition target
set of named entities are considered as correctly mapped to the different concepts in
the gold standard ontology. In the maximal experiment, a number of 623 named
entities extracted from the “Lonely Planet” corpus are classified against an ontology
consisting of 74 concepts, which is proposed in the PASCAL challenge [
There are two explanations for the lower classification quality values in the maximal experiment as compared to the minimal one. First, the minimal experiment uses the DCH histogram vector representation as compared to the flat counts of the maximal experiment, and second is the noise caused by the much bigger quantity of noun phrases classified in the maximal experiment – 43006 versus 631. Also, an explanation for an overall degraded quality of the named entity classification as 1 http://olc.ijs.si/eval.html. compared to the taxonomy enrichment in the preceding experiments is that the classified named entities have very low frequency of occurrence as compared to the classified terms (common nouns) from the taxonomy enrichment, and consequently they have a very sparse vector representation. This misleads their classification. We have presented an unsupervised top-down neural network based approach and framework for taxonomy enrichment. The framework can be applied to different domains and languages. The experimental results obtained in the “Lonely Planet” tourism domain prove that our contributed semantics based vector representations, i.e. the document category histograms and the DF-ITF weighting scheme are suitable for the task of taxonomy enrichment.
The comparison of taxonomy enrichment systems (and of named entity classifiers)
is problematic. Different systems use different domains and, even for the same
domain, they use different corpora of different sizes and different ontologies. [
Another interesting point is that sometimes given taxonomic structures are not reflecting correctly some fine-grained meanings. For instance, in the initial taxonomy used in our experiments, forest is hyponym of area. However the context in which the term forest occurs in the corpus are rather specific to plants (plant concept), which is far in the taxonomy from area. Our system “incorrectly” classified forest as plant.
The data sparseness remains a problem for the task of taxonomy enrichment.
Terms (or named entities) represented by sparse vectors have an increased chance to
be wrongly classified, because of the reduced power of attraction towards the correct
branches and nodes of the taxonomy. Thus the top-down search during the
classification is misled, and this phenomenon is mostly encountered in the case of
named entity classification, where named entities have very sparse vector
representations. Consequently, as further work, we will try to change the statistical
distributional vector representation of the terms to further reduce the dimensionality
of the vectors. We will try using pseudo-syntactic dependencies as representation of
the terms, in the spirit of [