English. We propose a new method for unsupervised learning of embeddings for lexical relations in word pairs. The model is trained on predicting the contexts in which a word pair appears together in corpora, then generalized to account for new and unseen word pairs. This allows us to overcome the data sparsity issues inherent in existing relation embedding learning setups without the need to go back to the corpora to collect additional data for new pairs.
Italiano. Proponiamo un nuovo metodo per l’apprendimento non supervisionato delle rappresentazioni delle relazioni lessicali fra coppie di parole (word pair embeddings). Il modello viene allenato a prevedere i contesti in cui compare uns coppia di parole, e successivamente viene generalizzato a coppie di parole nuove o non attestate. Questo ci consente di superare i problemi dovuti alla scarsita` di dati tipica dei sistemi di apprendimento di rappresentazioni, senza la necessita` di tornare ai corpora per raccogliere dati per nuove coppie di parole.
In this paper we address the problem of unsupervised learning of lexical relations between any two words. We take the approach of unsupervised representation learning from distribution in corpora, as familiar from word embedding methods, and enhance it with an additional technique to overcome data sparsity.
Word embedding models give a promise of
learning word meaning from easily available text
data in an unsupervised fashion and indeed the
resulting vectors contain a lot of information about
the semantic properties of words and objects they
refer to, cf. for instance Herbelot and Vecchi
(2015). Based on the distributional hypothesis
coined by Z. S. Harris (1954), word embedding
models, which construct word meaning
representations as numeric vectors based on the
cooccurrence statistics on the word’s context, have
been gaining ground due to their quality and
simplicity. Produced by efficient and robust
implementations such as word2vec
On the other hand, lexical knowledge includes
not only properties of individual words but also
relations between words. To some extent, lexical
semantic relations can be recovered from the word
representations via the vector offset method as
evidenced by various applications including analogy
solving, but already on this task it has multiple
drawbacks
Just like a word representation is inferred from the contexts in which the word occurs, information about the relation in a given word pair can be extracted from the statistics of contexts in which the two words of the pair appear together. In our model, we use this principle to learn high-quality pair embeddings from frequent noun pairs, and on their basis, build a way to construct a relation representation for an arbitrary pair.
Note that we approach the problem from the viewpoint of lerning general-purpose semantic knowledge. Our goal is to provide a vector representation for an arbitrary pair of words w1; w2. This is a more general task than relation extraction, which aims at identifying the semantic relation between the two words in a particular context. Modeling such general relational knowledge is crucial for natural language understanding in realistic settings. It may be especially useful for recovering the notoriously difficult bridging relations in discourse since they involve understanding implicit links between words in the text.
Representations of word relations have
applications in many NLP tasks. For example, they could
be extremely useful for resolving bridging,
especially of the lexical type
Our project is related to the task of relation
extraction that has been in focus of various
complex models
There are numerous studies dedicated to
characterizing relations in word pairs abstracted away
from the specific context in which the word pair
appears. Much of this literature focuses on one
specific lexical semantic relation at a time. Among
these, lexical entailment (hypernymy) has
probably been the most popular since Hearst (1992)
with various representation learning approaches
specifically targeting lexical entailment
The kind of relation representations we aim at
learning are meant to encode general relational
knowledge and are produced in an unsupervised
way, even though it can be useful for
identification of specific relations like hypernymy and for
relation extraction from text occurrences
We propose a simple solution to the scalability problem inherent in word relation embedding learning from joint cooccurrence data, which also allows the model to generalize to word pairs that never occur together in the corpus, or occur too rarely to accumulate significant relational cues information. The model is trained in two steps.
First, we apply the skip-gram with negative sampling algorithm to learn relation vectors for pairs of nouns n1; n2 with high individual and joint occurrence frequencies. In our experiments, all word pairs with pair frequency more than 100 and its individual word frequency more than 500 are considered as frequent pairs. To estimate the SkipRel vector of the pair, we adapted the learning objective of skip-gram with negative sampling, maximizing log (vc0T :un1:n2 )+ ik=1 Eci Pn(c)[log ( vc0T :un1:n2 )] i (1) where un1:n2 is the SkipRel embedding of a word pair, vc0 is the embedding of a context word occurring between n1 and n2, and k is the number of negative samples.
High-quality SkipRel embeddings can only obtained for noun pairs that co-occur frequently. To allow the model to generalize to noun pairs that do not co-occur in our corpus, we estimated an interpolation u~n1:n2 of the word pair embedding u~n1:n2 = relU (Avn1 + Bvn2 ) (2) where vn1 ; vn2 are pretrained word embeddings for the two nouns and the matrices A; B encode systematic correspondences between the embeddings of a word and the relations it participates in. Matrices A; B were estimated using stochastic gradient descent with the objective of minimizing the square error for the SkipRel vectors of frequent noun pairs n1; n2
We call u~n1:n2 the generalized SkipRel embedding (g-SkipRel) for the noun pair n1; n2. RelWord, the proposed relation embedding, is the concatenation of the g-SkipRel vector u~n1:n2 and the Diff vector vn1 vn2 . 4
We trained relation vectors on the ukWAC corpus
The experimental setup is built and
maintained on GPU clusters provided by GRID5000
If our relation representations are rich enough in
the information they encode, they will prove
useful for any relation classification task regardless
of the nature of the classes involved. We evaluate
the model with a supervised softmax classifier on
2 labeled multiclass datasets, BLESS
g-SkipRel RelWord Random Majority 81.15 59.07 80.94 12.5 24.71 relata spanned across 8 classes of semantic relation and EVALuation1.0 has 7.5k datasets spanned across 9 unique relation types. From EACL 2017 dataset, we used a list of 4062 noun pairs.
Since we aim at recognizing whether the information relevant for relation identification is present in the representations in an easily accessible form, we choose to employ a simple, one-layer SoftMax classifier. The classifier was trained for 100 epochs, and the learning rate for the model is defined through crossvalidation. L2 regularization is employed to avoid over-fitting and the l2 factor is decided through empirical analysis. The classifier is trained with mini-batches of size 16 for BLESS & EVALuation1.0 and 8 for EACL 2017. SGD is utilized for optimizing model weights.
We prove the efficiency of RelWord vectors, we contrast them with the simpler representations of (g-)SkipRel and to Diff, the difference of the two word vectors in a pair, which is a commonly used simple method. We also include two simple baselines: random choice between the classes and the constant classifier that always predicts the majority class. 6
All models outperform the baselines by a wide margin (Table 1). RelWord model compares favorably with the other options, outperforming them on EVAL and EACL datasets and being on par with the vector difference model for BLESS. This result signifies a success of our generalization strategy, because in each dataset only a minority of examples had pair representations directly trained from corpora; most WordRel vectors were interpolated from word emeddings.
Now let us restrict our attention to word pairs that frequently co-occur (Table 2). Note that the composition of classes, and by consequence the majority baseline, is different from Table 1, so the accuracy figures in the two tables are not di
SkipRel RelWord Random Majority 77.13 73.37 83.27 12.5 33.22 44.61 48.40 54.47 11.11 26.37 rectly comparable. For these frequent pairs we can rely on SkipRel relation vectors that have been estimated directly from corpora and have a higher quality; we also use SkipRel vectors instead of gSkipRel as a component of RelWord. We note that for these pairs the performance of the Diff method dropped uniformly. This presumably happened in part because the classifier could no longer rely on the information on relative frequencies of the two words which is implicitly present in Diff representations; for example, it is possible that antonyms have more similar frequencies than synonyms in the EACL dataset. For BLESS and EVAL, the drop in the performance of Diff could have happened in part because the classes that include more frequent pairs such as isA, antonyms and synonyms are inherently harder to distinguish than classes that tend to contain rare pairs. In contrast, the comparative effectiveness of RelWord is more pronounced after frequency filtering. The usefulness of relation embeddings is especially impressive for the EACL dataset. In this case, vanilla SkipRel emerges as the best model, confirming that word embeddings per se are not particularly useful for detecting the synonymy-antonymy distinction for this subset of EACL, getting an accuracy just above the majority baseline, while pair embeddings go a long way.
Finally, quantitative evaluation in terms of classification accuracy or other measures does not fully characterize the relative performance of the models; among other things, certain types of misclassification might be worse than others. For example, a human annotator would rarely confuse synonyms with antonyms, while mistaking has a for has property could be a common point of disagreement between annotators. To do a qualitative analysis of errors made by different models, we selected the elements of EVAL test partition where Diff and RelWord make distinct predictions that are both different from the gold standard label. We manually annotated for each of the 53 examples of this kind, which model is more a acceptable according to a human’s judgment. In a majority of cases (28) the WordRel model makes a prediction that is more human-like than that of Diff. For example, WordRel predicts that shade is part of shadow rather than its synonym (gold label); indeed, any part of a shadow can be called shade. The Diff model in this case and in many other examples bets on the antonym class, which does not make any sense semantically; the reason why antonym is a common false label is probably that it is simply the second biggest class in the dataset. The examples where Diff makes a more meaningful error than RelWord are less numerous (6 out of 53). There are also 15 examples where both system’s predictions are equally bad (for example, for Nice,France Diff predict isa label and WordRel predicts synonym) and 4 examples where the two predictions are equally reasonable. For more examples, see Table 3. We note that sometimes our model’s prediction seems more correct than the gold standard, for example in assigning hasproperty rather than isa label to the pair human, male. 7
The proposed model is simple in design and
training, learning word relation vectors based on
cooccurrence with unigram contexts and extending
to rare or unseen words via a non-linear
mapping. Despite its simplicity, the model is
capable of capturing lexical relation patterns in vector
representations. Most importantly, RelWord
extends straightforwardly to novel word pairs in a
manner that does not require recomputing
cooccurrence counts from the corpus as in related
approaches
In our evaluation, we observed that learning word pair relation embeddings improves on the semantic information already present in word embeddings. With respect to certain semantic relations like synonyms, the performance of relation embedding is comparable to that of word embeddings but with an additional cost of training a representation for a significant number of pair of words. For other relation types like antonyms or hypernyms, in which words differ semantically but share similar contexts, learned word pair relation embeddings have an edge over those derived from word embeddings via simple subtraction. While in practice one has to make a choice based on the task requirements, it is generally beneficial to combine both types of relation embeddings for best results in a model like RelWord.
Our current model employs pretrained word
embeddings and learns the word pair embeddings
and a word-to-relation embedding mapping
separately. In the future, we plan to train a version
of the model end-to-end, with word embeddings
and the mapping trained simultaneously. As
literature suggests
This research is supported by CNRS PEPS grant ReSeRVe. We thank Roberto Zamparelli, Germa´n Kruszewski, Luca Ducceschi and anonymous reviewers who gave feedback on previous versions of this work.
Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, Jun Zhao, et al. 2014. Relation classification via convolutional deep neural network. In COLING, pages 2335–2344.