AI applications often receive their input in the form of natural language text, or as the transcription of spoken text. A commonsense inference system should transform such input to a formal representation with limited vocabulary in order to be able to process them. In this paper, we present a method based on neural word embeddings that automatically assigns semic features to words of natural language. These features either describe the ontological category of a given word or provide some characterization or additional information. We show that our method has high coverage and performs well for English and Hungarian, and can easily be extended to other languages as well.
One of the most natural representations of commonsense knowledge is natural language. What people think or know about the world is expressed in either spoken or written language. Due to the popularity and accessibility of on-line media, crowds of people put their knowledge into written texts, either in the form of very short comments on social media sites or in the form of longer posts in addition to the writings of professional journalists. These texts, which are produced in a daily manner, adapt to changes in language use, and not only general knowledge, but facts and beliefs about the actual state of the world is also represented in them. Moreover, not only standard language, but slang and words used in informal contexts and special domains are also present in texts collected from the Web. In addition, more and more books representing a wide range of domains and styles are digitized. Large written corpora consisting of these resources are available as raw material for research, and can be exploited as a source of knowledge.
A more structured form of knowledge representation is
hand-crafted ontologies, such as WordNet
Whatever its source, a knowledge base is an
essential component of a commonsense inference system. Even
though recent results achieved by applying deep neural
systems on raw textual input have been significant, traditional
inference systems first transform their input written in
natural language into a formal representation using features
extracted from one or more knowledge bases, then they try
to solve the given task based on this formal representation.
In order to be able to process arbitrary input, the coverage
of the knowledge bases used should be as high as possible
In this paper, we present an automatic method that is able
to assign semantic features or atomic predicates to
practically any (even non-standard/slang or misspelled) word
form in a text in a language-independent manner. As we
apply morphological analysis and lemmatization to the corpus
both at the time of generating the embedding models and at
query time, all forms of a single lemma are covered instead
of only those explicitly present in the original corpus. This
is essential to achieve a good coverage for an agglutinating
language like Hungarian where a single lexeme may have
hundreds of possible word forms, only few of which are
actually present even in a huge corpus. Instead of constructing
another static knowledge base of fixed vocabulary, we
propose a dynamic tool that can be retrained or fine-tuned at
any time using an up-to-date, possibly domain-specific
corpus appropriate to the task at hand. The target formalism or
set of semantic features to be used is also an
interchangeable parameter of the proposed method. The set of features
and predicates presented in this paper is derived from
formalized definitions of a subset of the headwords (including
the defining vocabulary) of the Longman Dictionary of
Contemporary English (LDOCE)
Before we present the structure of the paper, let the following example illustrate the kind of semantic annotation automatically assigned by the model to words in the sentence The cow gives milk to her calf.: cow: mammal, at_farm, produce_milk, HASffour(legs)g, animal gives: =AGT.CAUSEf=DAT.HAS.=PATg, give, offer, communicate milk: food, sweet, drink, liquid calf: young, mammal, animal, has_wool. HASffour(legs)g
The paper is structured as follows: first, a brief introduction to neural word embeddings is presented. This is followed by the description of the lexical resource that we used when creating our models. In the following section, the method of building the model is described. In this paper, the method is demonstrated for English. However, existing semantic resources can also be mapped to word embedding spaces over the vocabulary of other languages. We have performed experiments with Hungarian, an agglutinative language with scarce semantic resources, but the method can easily be applied to other languages as well. Finally, we present both qualitative and quantitative evaluation of the models.
Traditional models of distributional semantics build word
representations by counting words occurring in a fixed-size
context of the target word
When training embedding models, a fixed-size context of each word in the vocabulary is used as the input of a neural network. This network is used to predict the target word from the context by using back-propagation and adjusting the weights assigned to the connection between the input neurons (each corresponding to an item in the whole vocabulary) and the projection layer of the network. This weight vector can finally be extracted and used as the embedding vector of the target word. Since similar words are used in similar contexts, these vectors optimized for prediction from the context will also be similar for similar words. There are two types of neural networks used for this task. One of them is the so called CBOW (continuous bag-of-words) model in which the network is used to predict the target word from the context, while the other model, called skip-gram, is used to predict the context from the target word. For both models, the embedding vectors can be extracted from the middle layer of the network and can be used alike as a dense vector representation of the meaning of the words in both cases.
The vectors thus obtained point to certain locations in the
semantic space consistently so that semantically and/or
syntactically related words are close to each other, while
unrelated ones are more distant. Moreover, it has been shown
that vector operations can also be applied to these
representations, thus the semantic relatedness of two words can be
quantified as the algebraic difference of the two vectors
representing these words. Similarly, the meaning of the
com1https://code.google.com/archive/p/word2vec/
position of two (or more) words is generally well
represented by the sum of the corresponding embedding vectors
As the words are represented as dense real-valued vectors, the similarity of two words can easily be defined as the angle between the vectors of the words, i.e. the most similar words for a query word can be retrieved by finding its nearest neighbours in the vector space according to cosine distance.
One of the main drawbacks of building such a model from raw corpora, however, is that by itself it is not able to handle polysemy and homonymy, because one representational vector is built for one lexical element regardless of the number of its different senses. We applied a simple method to alleviate this problem, at least in cases where the homonyms have different PoS. In order to assign different vectors to the same word with different parts-of-speech, we applied PoS-tagging and lemmatization to the training corpora before building the model. The main PoS tag of each word was attached to the word as a suffix in the form lemma#PoS, thus a different embedding vector was created for homonymous lemmas with different parts-of-speech.
We trained an English word embedding model on the English Wikipedia dump2 of 2.25 billion tokens (8.24 M token types) that was annotated using the Stanford tagger (Toutanova et al. 2003). Since the CBOW model has proved to be more efficient for large training corpora, we used this model architecture for training with the radius of the context window set to 5 and the number of dimensions to 300 and using a token frequency limit of 5.
Figure 1 illustrates how the words pianist, teacher, turner,
maid and their three nearest neighbors are arranged in the
English word embedding space3. The original vectors
consist of 300 dimensions, but these were mapped to a 2D
representation using the t-sne algorithm
Our goal was to create a model that can assign semantic
features and elementary predicates to words in an arbitrary
text. Thus, first, the set of features to be used had to be
defined. The Longman Dictionary of Contemporary English
(LDOCE)
We further transformed this format so that we have some category labels (here: unary and binary predicates) and listed examples. This was achieved by segmenting the formal descriptions into elementary predicates (by splitting at commas), but we did not segment predicates into further parts, so e.g. HAS[four.(legs)] remained an atomic feature. Each such token was treated as a category label. Then, all words that had the particular token in their definition were listed as an example for that label. This resulted in 1489 category labels and 12,507 words listed as examples for them. Then, in order to make this resource compatible with the word embedding model built from the Wikipedia corpus, its vocabulary was intersected with that model. Even though the vocabulary of this resource consists mostly of frequent words used in LDOCE definitions, it also includes some affixes, inflected forms, and a few multiword items, which are not present in the lemmatized Wikipedia model, so the intersection resulted in 11,039 words. Table 1 shows some examples words for some features derived from the 4lang resource.
However, some categories were too broad and the set of
words listed for them was too heterogeneous. To handle this
problem, a hierarchical agglomerative clustering algorithm
was applied to the set of words in those categories that
contained at least five words. The reason for applying a
hierarchical clustering rather than k-means is based on the
argument of
Even though we present our method using only the 4lang dictionary as a lexical resource, the system can be built from any dictionary that can be transformed to a similar format.
Our objective was to create a model with high lexical coverage that can also return the most relevant semantic features for words not present in 4lang. In order to achieve this goal, the semantic features from this controlled set were projected into the embedding space containing the representation of the words. Nearest feature neighbors for each word can be retrieved from the model using the cosine distance metric.
For each indexed semantic predicate label output by the clustering algorithm, we iterated the list of example words annotated with their part-of-speech (the crude PoS tags used in the 4lang resource had to be mapped to the more finegrained PTB tags returned by the Stanford tagger) and retrieved their embedding vectors from the word embedding model built from the PoS-tagged Wikipedia corpus. As a simple but effective method for rendering a representation vector for a set of words with their corresponding word embeddings we took the mean of these vectors, and used that as the embedding vector of that particular semantic feature. Thus a representation of each predicate used in the definitions was obtained in the semantic space created from the English PoS-tagged corpus. These semantic feature vectors were kept separated from the word vectors in the original embedding model in order to be able to restrict lookup to either words or features derived from each lexical resource. To find the relevant features for a query word tagged with its appropriate part-of-speech, its representational vector is retrieved from the word embedding model and its nearest neighbors are taken from the model containing the semantic predicates. Since instead of exact matching, nearest neighbors are searched for, out-of-vocabulary words (with respect to the original lexical resources) can also be assigned semantic labels. The only requirement is that the word must be present in the word embedding model.
We also carried out some experiments to apply our method
to another language, Hungarian. Hungarian is an
agglutinative language with very few lexical semantic resources.
As the original 4lang dictionary contained the Hungarian
translation of the vocabulary included (3477 words), it was
straightforward to create a similar model for Hungarian as
well. For this, we had to create a Hungarian word embedding
model, which was built from a web-crawled corpus of 3.18
billion tokens (27.49 M token types) that was annotated
using the PurePos
The aim of this research was to investigate the possibility of providing a high coverage tool for assigning a semantic representation to words of a natural language input dynamically instead of using a static knowledge base with a limited vocabulary. Thus, first we investigated the performance of the tool for some example input, then we also performed a quantitative analysis.
Table 2 shows an example: Laika likes eating fried onion with cucumber. First, using the Stanford parser, the input is annotated with part-of-speech tags and each word is lemmatized. Then, for each lemmatized content word (i.e. omitting the function word with) with corresponding part-of-speech, the top 10 nearest features are retrieved from the model and ordered by their distance from the vector representing the target word in the embedding space. Note that the number of top n features generated for each word is a free parameter, but moving further in the semantic space results in less and less appropriate features for the target word. Table 3 shows the WordNet hypernyms assigned to each content word in the same sentence (the representation of the adjective fried and the proper name Laika is missing from WordNet).
As it can be seen in the example, our model is able to assign two types of features to words. Ontological/taxonomic categories, such as carnivorous, mammal for the word Laika vegetable, food for the words onion and cucumber appear together with characteristic features of the given concept, such as faithful, HASffour(legs)g, hAT/2744.farmi or round and CAUSEffood.HAS.tasteg. While the first type of features can be extracted from traditional ontologies, the latter type of characteristics can not. However, we believe that the latter type of features form an important part to common sense knowledge, because if people are asked to describe a concept, they will rather use such characteristics. Moreover, an inference system can also benefit from such descriptions. It can also be seen from the example, that the model “knows” that Laika is a dog by returning semantic features characterizing dogs. In addition, the feature EAT.flesh emphasizes the contrast of Laika being a dog and eating cucumber and onion.
Another benefit of our model, as mentioned above, is that it is able to generate features for all the words that are present in the original corpus the word embedding was built from, not only for the extremely limited set of words included in the 4lang dictionary. WordNet or other hand-made resources are limited only to the words and the classification that the designer of the resource had in mind. Our model, in contrast, is able to assign features to proper names, slang words or mistyped word forms as well as long as these are represented in the corpus the word embedding model was created from. In addition to the above example containing the dog name Laika, the following examples show some of the nearest features for two more proper names and two slang words: IBM: information.IN, computer, equipment, electric, group Facebook: information.ON, ABOUT.recent(events), computer hype: fame, fun, idea, popular, surprise numpty: bad, lazy, stupid, lack(work), dull
A weakness of our method is that in some cases it also adds noise in the generated features. For example, features such as sleep or sing generated for the verb eat are not ones we would expect to be part of the definition of eat (even if in a broader sense they might be related). Inappropriate features like this may be eliminated manually from the representations generated by the model. The model can thus also be used as an aid in a semi-automatic semantic resource creation/extension process proposing an initial representation that can be cleaned manually for applications that require a high-precision lexical semantic representation. Otherwise, the generated semantic features can be used in models performing some downstream task even without filtering out the noise. In that case, the added semantic features may improve the performance of the downstream tool providing mostly useful features for words that otherwise would completely lack semantic representation.
We also carried out two kinds of quantitative analysis of the performance of our model. First, we checked the robustness of the model by performing a sanity check. For each word present in the original 4lang dictionary, we calculated how many of the semantic features present in the original definition were retrieved among the top N features returned by the model (feature recall, Rf ) and the percentage of words for which all features were retrieved (word recall, Rw). The results are shown in Table 4 as a function of N (numbers are percentages). Recall was also calculated ignoring words having more than N features (Rwp) and discounting features jfj over the N limit for words having more than N features (Rfp ). As no definition contained more than 10 terms, Rwp is identical to Rw and Rfp is identical to Rf for N 10. The definitions are terse and contain a minimal description for each word: for half of the words containing only a single term, and for almost all words not more than 5, see column jf j N ). Feature precision (P (f )) apparently decreases quickly as the number of features retrieved increases if we blindly accept only terms present in the original definitions as correct. See, however, further discussion below. The last column of the table shows the mean average precision (MAP) of features (terms) present in the original definitions.
In the other experiment, we selected 280 words not present in the original dictionary randomly from a predefined list of Hungarian words in which each word was assigned to one of 28 semantic domains (e.g. food, vehicles, locations, occupations, etc.). From each domain 10 words were chosen randomly and were translated to English. Then, for these words, the 10 nearest features were generated and two human annotators checked whether each feature was adequate for each given word. The same evaluation was performed for Hungarian. The agreement ratio between the annotators was 0.798 for English and 0.734 for Hungarian according to Cohen’s kappa, which is substantial in both cases. The results are shown in Table 5.
The table shows feature accuracy (acc: the ratio of correctly assigned features) in each domain. We also automatically computed feature “domain accuracy” (d-acc): here we ignored feature assignment errors where the same feature was marked adequate for another test word in the same domain. The number of different features that appeared in this evaluation and the number of features marked wrong at least once are shown in the last two columns. Note that the feature accuracy (precision) for 10 features retrieved turned out to be much higher (75.13%) than in the sanity check experiment (only 32.70%) even though this list contained words not in the original resource. The reason for this is that the model returns many features which, while not explicitly present in the original terse definitions, correctly follow from the knowledge embodied in the feature model. E.g. While the definition of dog in 4lang contains only 3 terms: animal, faithful and carnivorous, the top 10 features retrieved from the model also include mammal, HASffour(legs)g, hairy and companion. The sanity check experiment thus grossly underestimated the precision of the model.
We have presented an automatic method that is able to assign semantic features to words of natural language. This approach exploits the representative power of neural word embeddings by mapping features derived from formal definitions of words to the vector space of the given language. In addition to some illustrative examples, we have presented the evaluation of the models demonstrating that the method works with relatively high accuracy. Although there is a moderate amount of noise in the set of generated features, the method has a very high coverage, being able to process proper names or non-standard words as well, which cannot all be included in hand-made static knowledge bases. As such, our automatic method can be used as the base of a manually constructed resource, or can provide valuable input for downstream applications, such as commonsense inference systems.
This research has been implemented with support provided by grant FK125217 of the National Research, Development and Innovation Office of Hungary financed under the FK17 funding scheme.