Word Embeddings can capture lexico-semantic information but remain flawed in their inability to assign unique representations to different senses of polysemous words. They also fail to include information from well-curated semantic lexicons and dictionaries. Previous approaches that obtain ontologically grounded word-sense representations learn embeddings that are superior in understanding contextual similarity but are outperformed on several word relatedness tasks by single prototype words. In this work, we introduce a new approach that can induce polysemy to any pre-defined embedding space by jointly grounding contextualized sense representations learned from sense-tagged corpora and word embeddings to a knowledge base. The advantage of this method is that it allows integrating ontological information while also readily inducing polysemy to pre-defined embedding spaces without the need for re-training. We evaluate our vectors on several word similarity and relatedness tasks, along with two extrinsic tasks and find that it consistently outperforms current state-of-the-art.
Distributed representations of words
While these embeddings capture meaningful lexical
relationships, they come with their own set of drawbacks. For
instance, complete reliance on natural language corpora
amplifies existing vocabulary bias that is inherent in datasets.
Vocabulary bias is caused by words not seen in the training
corpora and also extends to bias in word usage where some
words, often morphologically complex words, are used less
frequently than other words or phrases with the same
meaning. Thus embeddings suffer from inaccurate modeling of
less frequent words which is evident in the relatively lower
performance of word embeddings on the rare word
similarity task
Polysemy is an important feature of language which causes words to have a different meaning or “sense” based on the context in which they occur. For instance, the word bank can refer to a financial institution or land on either side of a river. A large body of work has gone into developing word sense disambiguation systems to identify the correct sense of a word based on its context. Word embeddings, on the other hand, assign a single vector representation to a word type, irrespective of polysemy. The availability of disambiguation systems coupled with the growing reliance of NLP systems on distributional semantics has led to an increasing interest in obtaining powerful sense representations.
Some of the previous work that has gone into learning
sense representations includes unsupervised learning
techniques to cluster contexts and learn multi prototype
vectors(
Previous work combining distributional semantics and
knowledge bases include
In this work, we present a novel approach that uses
knowledge bases and sense representations to directly induce
polysemy to any pre-defined word embedding space. Our
approach leads to interpretable, ontologically grounded sense
representations that can easily be used with powerful
disambiguation systems. The main contributions of this
paper are a) Obtaining ontologically grounded sense
representations that perform well on both similarity and
relatedness tasks b) Automatic sense induction and integration of
knowledge base information into any predefined embedding
space without re-training c) Our embeddings also show
performance benefits when used with transfer learning
methods like CoVE
In our approach we thus rely on a) Sense tagged corpora to obtain contextualized sense representations. The objective of which is to capture sense relations and interactions in naturally occurring corpora. The sense representations are interpretable and have lexical mappings to a knowledge base. We use them to induce polysemy in word embedding spaces. b) Pretrained word embeddings to capture beneficial lexical relationships that are inherent on account of being trained on large amounts of data. Sense representations do not adequately capture these relationships due to the limited size of sense-tagged corpora which is used to train them. c) Lastly, to account for the vocabulary bias in corpora which causes similar meaning words to be farther apart in embedding spaces, we use a knowledge base to jointly ground word and sense representations.
We thus describe our approach in three parts a) Lexicon building b) Sense-Form Representations and c) Multi
For our Knowledge Base, we rely on WordNet
Thesaurus Inclusion The structure of WordNet(WN) is such that it labels semantic relations among different synsets. While this structure helps determine the degree of similarity between synsets, it leads to a restricted set of synonyms that represent a synset. To best combine information from both resources, we augment the synonyms in a WordNet synset using a Thesaurus.
1https://www.thesaurus.com/
Unlike WordNet(WN), the thesaurus does not have distinct labels for senses. Senses are instead represented by a group of words. Given a query word, the thesaurus returns clusters of words where each cluster represents some sense. Given a WN synset(s), we use the synset’s headword to query the thesaurus and use a simple algorithm to map the most appropriate cluster to the corresponding WN synset by computing each cluster’s probability with respect to (s).
Probabilities are assigned based on the words in a cluster and the WN structure. Thus if a thesaurus cluster has more words that are “closer” based on WN structure to the synset(s), it receives a higher probability. To measure “closeness”, we use the path-similarity(p) metric of WN. Path-similarity(p) measures the similarity between two synsets by considering the distance between them. It ranges from 0 1 with scores towards 1 denoting “closer” synsets. Since path-similarity(p) calculates similarity between two synsets, thus given a word (w) in a thesaurus cluster queried using the headword of the WN synset(s), we find the distance-based similarity dw;s between s and w by first obtaining all of the synsets(Sw) in WN for w and use it to calculate dw;s as follows.
dw;s
maxfp(s; si)8si 2 Swg If a word is not found in WN, we assign dw;s to 0:1 which is the lowest distance-based similarity implying it is ”farthest” from the synset(s) in WN.
To account for varying cluster sizes in the thesaurus
Input: WordNet Synset (s), corresponding synonym set(Sw) Output: Most probable cluster for a word Cwn out of all possible clusters Cw found in Thesaurus for a word. 1: Cw Thesaurus(w) 2: if length(Cw) = 1 3: n = 0 4: else 5: 6: 7: end if 9: return Cwn pc(w) fp(cluster)8cluster 2 Cwg n index(pc(w) , max(pc(w)) and prevent larger clusters from invariably having bigger scores, we divide words in each cluster(c) into ten discrete bins(bins) based on each word’s d score. The bins are in an incremental range of 0:1(( [0-0.1 , 0.11- 0.2 ,...,0.91-1.0]), with the highest score bin being 1. We then obtain cluster scores, scorecluster as : scorecluster = wbin
count(bin)
X bin2bins We then get the probability of a cluster(pcluster) from scorecluster by passing it through a sigmoid function. pcluster =
exp(scorecluster) exp(scorecluster) + 1
The words in the thesaurus cluster with the highest probability is then picked to augmented into the synonym list of the respective WN synset(s) . We’ve outlined the procedure in Algorithm 1.
In the Table 1 we denote the vocabulary and synset cluster changes brought about by this step. The last column records the average number of synonyms linked with a synset in WordNet. Originally, owing to WordNet’s stringent relation structure we see there are an average of approximately 2 synonyms within a synset. This number drastically increases using a thesaurus for augmentation.
Phrases WordNet 147307 Thesaurus(Introduced) 4026
WordNet Form Extension To obtain representations that cater to both similarity and relatedness, we modify the synset nodes in WordNet. A synset in WordNet is represented by a set of synonyms. We observe that these synonym sets include words of the same meaning without differentiating between their syntactic forms. For instance, consider the synset operate.v.01, defined as “direct or control; projects, businesses” , it has both run and running in its synonym sets. In practice, each syntactic form of a word has different semantic distributions. For instance, for this sense, run is found to most likely occur with words such as lead and head as compared to its alternate form running which is more likely to appear with words such as managing, administrating , leading. To account for this difference in semantics, we extend WordNet nodes to include the syntactic form information and call a synset, syntactic form pair “senseform.” To obtain different sense-form nodes, we make use of the OMSTI corpus and record different forms of a synset based on the different syntactic forms of words associated with the synset. Each “sense-form” is then linked to the corresponding syntactic form of synonyms. The extended WordNet(Ext-WN) sense-form nodes and synonyms are depicted in Figure 1.
To obtain sense-form representations, we use a sense-tagged
corpus, OMSTI
We initialise each sense-form node in WN using the representations obtained from the sense-tagged corpora. Then, for each sense-form and the respective augmented synonym set, we obtain unique multi word-sense representations by jointly grounding the word and sense-form embeddings to WordNet. For a word(w) in synonym set of a sense(s), we obtain multi word-sense representations as follows: vw;s =
w;s([uw; vs;form(s)])
Where, uw is the pre-trained word embedding , vs;form(s) is the contextualized sense-form representation of the node learned from sense-tagged corpora. For grounding, we use WordNet’s synset rank information and graph structure to obtain the scaling factor, w;s for grounding as follows: w;s = 1
clog(x); where x = ranks;w + d(w; s)
For word (w) in the w; s pair, WN which gives the list
of senses(Sw) in decreasing order of likelihood. We use this
to obtain the rank ranks;w of a senses with respect to w.
The sense with rank 1 in Sw for a word is thus the most
likely sense of the word. As outlined in our previous
sections, we use an augmented synonym set by adding from a
thesaurus for each synset node which means there are many
word-sense pairs in our extended-WN not found in WN. For
example, the extended-WN includes “hold” as a synonym
for sense “influence.n.01”. This word and sense pair(hold,
influence.n.01) is not found in WN. Thus “influence.n.01” is
not part of Shold in the original WN. If a word(w),sense(s)
pair from our extended-WN is present in Sw, we use the
rank directly. If not, we use the rank of the synset in Sw that
is “closest” to the sense s in the word-sense pair. The WN
path-similarity(p) metric is used to denote “closeness”. We
would also like to penalise senses s found in our
extendedWN pairs more if they are farther in the WN graph structure
to the original senses Sw given by WN for word w. The
intuition is, the closer a sense is to a word in the WN graph,
the more relevant it is to the word. The same intuition is
followed in retrofitting vectors to lexicons as well
p(s; x)8x 2 Sw])
Recall p(s; x) is the path-similarity score with a higher
score denoting closer pairs, implying closer pairs get
assigned a lower penalizing distance. We use a monotonically
decreasing distribution 1 clog(x) with c as some constant
in our probability distribution as found by
We thus get grounded representations with the scaling factor w;s reflecting likelihood and ontology graph structure.
In this section, we describe the experiments done to evaluate our multi word-sense word embeddings. We use an array of existing word similarity and relatedness datasets to conduct intrinsic evaluation and 4 datasets across 2 tasks for extrinsic evaluation.
We test our embeddings intrinsically on similarity, relatedness and contextual similarity datasets.
Word Representations To run our experiments,we
pick two different embeddings of 300 dimension
GLoVE
Similarity Measures Given a pair of words w with M
senses and w0 with N senses, we use the following two
metrics proposed by
AvgSim(w; w0 ) = M axSim(w; w0 ) =
1 M N
M N X X(cos(vw;i; vw0 ;j )) i=1 j=1 max 1 i M;1 j M cos(vw;i; vw0 ;j ) AvgSim computes word similarity as the average similarity between all pairs of sense vectors. Whereas M axSim computes the maximum over all pairwise sense vector similarities.
Baselines We denote two baselines in Table 2. and Table 3., in addition to the baseline score of the single prototype word embeddings themselves. The first baseline we denote is to measure performance on concatenating sense embeddings learned from the OMSTI corpus along with word embeddings using WordNet to retrieve senses for a word. This baseline is to indicate scores on concatenating embeddings from two different sources. This is denoted as +Synset(WN) in the table. The second baseline, +CoKE(Ext-WN) is to track performance changes when splitting senses to sense-forms and grounding them to extended-WN. Finally, we show scores with +CoKE(Thes+Ext-WN) which reflects performance of grounded word-sense representations using sense-forms, extended-WordNet and the thesaurus.
Word Similarity We evaluate our embeddings on
several standard word similarity datasets namely, SimLex
Each dataset contains a list of word pairs with an individual score generated by humans of how similar the two words are. We calculate the Spearman correlation between the labels and the scores generated by our method. For similarity, we use M axSim as a metric to find the most similar pair among different senses of a word. The results are outlined in Table 2.
We observe that the lower performance for Synset(WN), obtained by concatenating word with sense embeddings to get word-sense embeddings, is because of the limited number of synonyms for a synset recorded in WordNet along with the limited size of the dataset used to learn these embeddings.
The average improvement column in the table(Avg
Improvement), shows a significant improvement in performance on
splitting senses to sense-forms and
grounding(CoKE(ExtWN)). The benefits of this approach are reflected mainly
on the SimVerb-3500 dataset. This is not a surprising result
since words tend to have more syntactic forms when they
occur as verbs. With distributional semantics, syntactic forms
of verbs often remain close making it hard to capture
differences. However drastic improvements can be seen through
76.96
-25.76
-24.64
+0.21
74.97
-11.85
-7.96
+10.84
76.15
-10.34
-4.23
+11.6
50.33
-28.24
-27.7
+1.72
45.78
-23.03
-25.38
+1.51
+0.59
+4.04
+17.69
40.82
+0.48
+6.96
+18.29
55.89
+5.41
+11.75
+11.69
57.08
+0.26
+7.02
+11.8
78.80
-11.44
-9.48
+5.98
78.60
-10.24
-6.19
+7.27
+1.1
+6.71
+13.51
28.32
+0.47
+8.06
+17.59
-10.02
-6.75
+8.80
-9.35
-5.12
+9.75
thesaurus inclusion(CoKE(Thes+Ext-WN)), this is because
using WordNet alone leads to limited lexemes on account
of words being represented by fewer senses as opposed to
a large number of senses captured for a word by word
embeddings, as a result of being trained on large datasets. On
including a thesaurus and augmenting the synonym set for
synsets in WordNet, we see that the number of senses that
represent a word drastically changes leading to more
lexemes that closely reflect all possible senses of a word.
We also note that the improvements for WS-S is relatively
lower; we suspect this is because the dataset is designed
based on association rather than similarity alone. We also
observe that as baselines of embedding spaces get higher for
datasets, the performance gains reduces since most of the
information is captured in the embedding spaces. The same
trend is also observed in
The baselines we use are the same as for word similarity as described above. We notice how performance improvements through sense-form splitting are not as drastic as for word similarity. This could be on account of word relatedness tasks more frequently checking for relatedness of objects rather than verbs; sense-form splitting is more beneficial to verbs than nouns on account of more varying forms of words as verbs.
We are not sure why the overall performance gains are not
as high as for similarity, but the scores do reflect gains as
opposed to retrofitting directly to lexicons which leads to a
serious drop in relatedness. The big performance gains on
SGS
SCWS dataset introduced by (Huang et al. 2012), where word pairs are chosen to have variations in meanings for polysemous and homonymous words. We compare our method with other state-of-the-art multi-prototype models.We find that our model performs competitively with previous models. We use the Skip-Gram(SG) word embedding with our method to allow for fair comparison, since previous work uses Skip-Gram for retrofitting to WordNet.The Spearman correlation between the labels and scores are indicated in Table 4.
A lot of the prior work on obtaining sense embeddings show
performance improvements in intrinsic tasks, but leave out
testing them on downstream tasks. It is thus difficult to judge
the effectiveness of these representations. To bridge this
gap, we run experiments on two tasks(Sentiment Analysis
and Question Classification) across 4 datasets to provide
some insight on the usefulness of our representations.
Datasets For sentiment analysis we use the Stanford
Sentiment Treebank dataset
CoKE by representing words as an average of their respective sense embeddings. It is a known fact that words are a weighted sum of their senses. Thus the intuition of using averaged embeddings is that having grounded word-sense representations should lead to better word representations through averaging.
Recent trends have also lead to an increasing interest
in transfer learning for obtaining superior word
representations. CoVE
Due to the promising performance of CoVE and ELMo as
word sense disambiguation systems and increasing interest
in using them in NLP tasks, we use them as disambiguation
systems in our experiments to sense tag the four benchmark
datasets. To get the disambiguated sense tags using CoVE
or ELMo, we use the same approach as outlined in
In our experiments, we use the CoKE word-sense embeddings obtained by using GLoVE with the thesaurus and extended-WordNet for grounding.We pick CoKE with GLoVE embeddings to be fair in comparison with CoVE which is obtained by concatenation with GLoVE embeddings.
We thus compare performance using GLoVE, CoVE and ELMo independently, using an average of CoKE representations to get word representations, and also using ELMo/CoVE as disambiguation systems with sense-tagged words represented with CoKE embeddings(CoKE+(CoVE), CoKE(+ELMo)). Note if a word is not sense-tagged we use vanilla GLoVE vectors concatenated with an unknown vector.
Training Details To test for performance of different
embeddings on datasets, we implement a single-layer
LSTM
In this section, we look at some visualisations of senses induced and show how they are easily interpretable. Since sense tags have lexical mappings to an ontology, they can be looked up to find meanings. Moreover, the semantic distribution of the word-senses also plays a role in obtaining meaningful sense clusters. We analyse two things 1) Sense clusters induced 2) How using different sense forms affect representations and sense interactions in their respective word forms. For all our analysis, we use the concatenated version of CoKE + GLoVE embeddings and use Principle Component Analysis to perform dimensionality reduction.
We look at the sense clusters formed by our word specific senses embeddings for the word “rock”.
The clusters for the word ”rock” is depicted in Figure 2. The multiple fine-grained word-sense embeddings for the word “rock” cluster to form 5 basic senses. We see three distinct clusters that dominate. “Cluster#2” can be interpreted as all synsets that speak of rock as a ”substance”. In, “Cluster#3”, the synsets cluster together to speak of rock as “music”. An interesting property can be observed comparing “Cluster#1” and “Cluster#5”. The senses found in both of these clusters interpret “rock” as “movement/motion”. However, the two distinct clusters also capture the kind of motion. For instance ,the senses roll:v:13 and rock:v:01 in “Cluster#5” map specifically “sideways movement”. While the senses in “Cluster#1” map to glosses “sudden movements”(convulse , lurch,move , tremble) and “back and forth movements(wobble , rock)”. Another interesting property is depicted by “Cluster#4”, although they are more synonymous in meaning to rock as a “substance”, the senses for gravel cluster very closely to senses mapping to gloss “jerking” movements capturing deeper relations between senses.
In this section, we analyse how different sense-form representations interact for synonyms within a synset. We do so by considering the word-forms “plan” and “planning” both of which are synonyms of their respective sense-forms of “mastermind.v.01” (Gloss: plan and direct, a complex undertaking).
In order to observe the difference in sense-form relationships of word-forms, we consider only common synsets in “plan” and “planning” for visualisation and observe the interactions with each other. For the word “plan” as shown in Figure 3.a), we observe that the synset “mastermind” is closer in proximity to synsets that map to words like “plan”, “sketch”, “prepare”. In contrast, the same synset in the embedding space for ”planning” as shown in Figure 3.b) interacts closely with synsets that are analogous to “project planning”, “scheduling”, “organising”. This shows how using different sense-form representations, leads to different and unique interactions among the same group of synsets for each word.
In our work, we explore the possibility of obtaining multi word-sense representations and sense induction to embedding spaces by using distributional semantics and a knowledge base. The prototypes allow ease of use with WSD systems, can easily be used in downstream applications since they are portable and are flexible to use in a wide variety of tasks. Previous work on obtaining sense representations falls under three distinct clusters - Unsupervised methods, Supervised resource-specific methods and ontology grounding. By using pre-trained unsupervised embeddings, supervised sense embeddings and jointly grounding them in an ontology, ours is the first approach that lies in the intersection of all three approaches. The code and vectors will be made available publicly as well.