In this paper, we propose an approach for enhancing word representations twice based on large-scale knowledge bases. In the rst layer of enhancement, we use the knowledge base as another contextual form corresponding to the corpus and add it to the training of distributed semantics including neural network based and matrix-based. In the second layer, we utilize local features of the knowledge base to enhance the word representations by mutual reinforcement between the keyword and the strongly associated words. We evaluate our approach not only on the well-known datasets but also on a brand-new dataset, IQ-Synonym-323. The results show that our approach compares favorably to other word representations.
Word representations as the fundamental tool of NLP become increasingly
important research. Currently, distributional semantics models that follow the
distributional hypothesis represent the most popular approach of word
representations. They commonly refer to the statistics derived from a large text corpus.
Meanwhile, it is proved that the larger the corpus is, the better the model
performs in most tasks [
As another approach of word representations which attracts increasing attention, the knowledge-based approaches mainly rely on the external structured databases. The abundant and explicit lexical relationships between lexical items in databases can just make up for the blurring of contexts in the large corpus.
In this work, we propose an approach with double enhancement based on the
large lexical database. Firstly, we take the related words in knowledge base as
the additional accurate context in comparison with the large corpus. Afterwards,
inspired from Kiela et al. [
The knowledge base we use in our approach is composed of a large number of
oneto-many relationship structures, i.e. given a keyword, the knowledge base will
list its closest semantically related words. Therefore, in the rst enhancement
of our approach, we take the related words provided by knowledge base as the
relative accurate context of keywords and inject them in the existing
representative distributional semantics models. For comparison, we select skip-gram [
Neural Network Based The original skip-gram is the neural network framework with a single hidden layer. Its basic idea is to predict the maximum probability of words appearing near the keyword. After the rst step of the original training in large text corpus, our added step follows the formula (1). The objective of the second step is to maximize the following average log probability. w1, w2,... wT is a sequence of training words. For keyword wt, Awt is the set of its related words. And the length of the set is regarded as the context window size in the additional training step. We name this approach SG-KB-I. 1 XT T t=1 wa2Awt
X log p (wajwt) (1) Matrix Based GloVe is an unsupervised learning approach which emphasizes the superiority of ratio in words' relevance and train log-bilinear regression model based on a global word-word co-occurrence matrix.
The cell of original matrix is the co-occurrence frequency of words in the xed-length context window of the text corpora. In our approach to attaching knowledge base as accurate context, we add the co-occurrence frequency of keyword-related word in knowledge base to the original matrix. In this way, we use the modi ed cell values to adjust the degree of association between words. Then the original algorithm is applied to the new matrix to promote the word representations. We called this approach GloVe-KB-I. 2.2
Within our extracted knowledge base, some pairs of words are mutual related. For instance, for the keyword \people", \human" is one of its related words in knowledge base. Meanwhile, \people" is also in the related word set of keyword \human". We consider \people" and \human" as a strongly associated words pair. For these word pairs, we attempt to tune their representations by mutual reinforcement. In the formula (2), Wsnr is the set of strongly associated words of keyword w, the vsr is the vector of the elements in this set, n is the length of the set. Wcmr is the set of commonly associated words of keyword w, vcr is their vector, the number is m. Wsnr and Wcmr together form the related words set of keyword w in knowledge base. We set a weight value to the strongly associated words, so that to pull keyword closer to the strongly associated words than the commonly ones.
vw = n
0 1
X vsr2Wsnr vsr +
X vcr2Wcmr
1 vcrA (2)
Afterwards, we use the SG-KB-I, GloVe-KB-I, as the initial vectors vi. We tune each keyword's vector vt by vw and vi. and are the weight coe cients. vt = vw + vi 2.3
Compared with raw corpus data, the knowledge base demonstrates more clari ed relations between words. We choose two large lexical databases as our sources, namely WordNet and ConceptNet, which contain adequate concepts and a very broad range of word relationships. We extract more than 155 thousand keywords from WordNet, and 766 thousand from ConceptNet. After combining the two parts with mutual lexical items, we nally get 777 thousand keywords with related words, to constitute our lexical relation knowledge base. 3 3.1
We evaluate our representations not only on the well-known dataset but also on the brand-new dataset we build. We construct a new dataset by collecting 323 synonym questions from related real IQ test books and websites for testing human intelligence, and name it IQ-Synonym-323. The questions we collected in our 323 synonym dataset have several types, like \Choose the word most similar in meaning to X?", or \Which word is closest to the X?" , etc. But all these types can be included as the keyword and candidate words then we reorganized them. Our dataset will be available as an open source. Table 1 shows a sample. 3.2
We choose a 11G dumps of English Wikipedia as text corpus. Table 2 shows
the performances of all comparison approaches, including skip-gram and GloVe
as the starting points, ConceptNet [
GloVe GloVe-KB-I
SG-KB-I GloVe-KB-II
SG-KB-II
ConceptNet Counter- tting
SimLex-999 IQ-Synonym-323 0.39 60.14% 0.35 59.61%
In this paper, we propose a double enhancement approach relying on knowledge base. Since knowledge base can specify more accurate related words of keywords as context information, we use it to compensate for the noises generated by multiple domains covered by the large corpus. Utilizing the features of knowledge base twice brings two signi cant improvements, as shown in Table 2. We evaluate our approach on the well-known SimLex-999, and the brand-new dataset, IQ-Synonym-323. The outstanding performance explains the advantage of our approach in embracing more accurate semantic similarity between similar vocabularies under large-scale corpora.