We propose a novel method for learning Chinese word embeddings. Different from previous approaches, we investigate the effectiveness of the Chinese stroke-level information when learning Chinese word embeddings. Empirically, our model consistently outperforms several state-of-the-art methods, including skipgram, cbow, GloVe and CWE, on the standard word similarity and word analogy tasks.
Wikipedia is one of the most important knowledge bases for many semantic web tasks
[
Most existing research efforts on learning word embeddings have conducted
experiments on the English Wikipedia data, which were shown effective [
Existing works such as CWE [
As shown in Fig. 1, “(P (timber)” and “î (forest)” are two semantically related words. There is no information that is shared across them at the character level. However, these two words are semantically related, and this can be captured by their stroke-level n-gram information that yields the common semantically meaningful component “( (wood)”.
In general, Chinese strokes can be summarized in five different types, i.e., horizontal stroke, vertical stroke, left-falling stroke, right falling stroke and turning stroke. We convert a word into stroke n-grams with four steps: 1. break down a word into characters; 2. obtain stroke sequence of each character by character-stroke mapping table; 3. concatenate such sequences of characters in a word; 4. use a sliding window with size n to generate stroke n-grams. 2.2
word2vec [
X
q2S(w)
q c
where sim(w; c) is the similarity score between current word w and a context word
c, q is an entry of collection S(w), where q is the embedding of q. Inspired by NCE
(noise-contrastive estimation) technology [
X w2D c2T (w)
log (sim(w; c)) + Ec0 P [log ( sim(w; c0))] where L is the objective function, D is the collection of words, and T (w) is the collection of context words of w within a sliding window. Ec0 P [ ] denotes the expectation, where c0 follows word unigram distribution P . In the euqation, c0 denotes a negative sample that does not occur in the context but randomly selected from the whole word vocabulary, and represents the number of negative samples. In addition, is sigmoid function described as (x) = (1 + exp( x)) 1. 3
We download Chinese Wikipedia dump3 as our training corpus, and preprocess the dataset as follows: 1. convert original XML format into plain text by a script4 in gensim toolkit; 2. change traditional characters into simplified Chinese using opencc5; 3. segment Chinese words in sentences by ansj toolkit6.
We crawl the stroke sequences from Xinhua Dictionary7 so as to build our stroke n-gram table. 3 https://dumps.wikimedia.org/zhwiki/latest/ 4 https://radimrehurek.com/gensim/corpora/wikicorpus.html 5 https://github.com/BYVoid/OpenCC 6 https://github.com/NLPchina/ansj seg 7 http://xh.5156edu.com/ (1) (2)
Following [
Word similarity. The task is a common approach to evaluate word embedding methods,
which measures the semantic relatedness between two words. We conduct the
experiments on two human-annotated datasets, including wordsim-240 and wordsim-296 [
In order to validate the effectiveness of our proposed model, we compare the results to the following state-of-the-art algorithms as baselines.
– word2vec [
In order to have a fair comparison, we set both window size and negative samples as 5 and fix the embedding size as 100 for all the models, and then remove the rare words that occur less than 10 times in the corpus. 3.4
As shown in Table 1, CWE overall outperforms skipgram and cbow on word similarity and word analogy tasks, since it leverages certain subword information – character-level information. Interestingly, GloVe, as a count-based approach, is better than these three models on the word analogy task. We can observe that, however, our model achieves overall the best results, thanks to the stroke n-gram information incorporated. 4
We investigated the use of the stroke-level information when learning Chinese word embeddings. We demonstrated the effectiveness of such information. In our future work, we would like to investigate other application scenarios where stroke-level n-gram information could be used. For example, the task of joint learning of word embeddings and concept (or mention) embeddings for entity linking. 8 code available: https://code.google.com/archive/p/word2vec/ 9 code available: http://nlp.stanford.edu/projects/glove/ 10 code available: https://github.com/Leonard-Xu/CWE
The authors would like to thank the three anonymous reviewers for their valuable comments, as well as Ziqi Liu for his helpful suggestions on the work.