Authors of Word2Vec claimed that their technology could solve the word analogy problem using the vector transformation in the introduced vector space. However, the practice demonstrates that it is not always true. In this paper, we investigate several Word2Vec and FastText model trained for the Russian language and find out reasons of such inconsistency. We found out that different types of words are demonstrating different behavior in the semantic space. FastText vectors are tending to find phonological analogies, while Word2Vec vectors are better in finding relations in geographical proper names. However, we found out that just four out of fifteen selected domains are demonstrating accuracy more that 0.8. We also draw a conclusion that in a common case, the task of word analogies could not be solved using a random word pair taken from two investigated categories. Our experiments have demonstrated that in some cases the length of the vectors could differ more than twice. Calculation of an average vector leads to a better solution here since it closer to more vectors.
The basic point for the semantic space of natural language words was the paper [
In order to keep the important lexical information, the FastText model was
introduced in 2017 [
In this paper, we investigate the reasons of such deviations in accuracy. In the
Section 2, we state the problem of word analogies as a vector transformation problem. The
Section 3 gives a short review of existing vector transformation methods for the
problem of word analogies. Sections 4 and 5 describe the used data set and the numerical
evaluation of free language models for the Russian language. The Section 6 analyses
the reasons of low accuracy for some categories of word analogies. The Section 7
concludes the article. In this paper, we do not consider systems of the BERT and ELMO
families [
In common words, the main question of the problem of word analogies could be stated as “Is there a word c which relates to the word b as the word a' relates to the word a?” Answering this question, Word2Vec uses vector representation of the word. Let va' and
We can reformulate the question for word groups. Let us consider a set of word pairs (w11:w12), (w21:w22), …, (wN1:wN2) that have the same semantic or lexical relation, and their corresponding vectors v11, v11, v21, v21, …, vN1, vN1. In this case, the task of word analogies could be formulated as following: if there is a vector x that makes an affine transformation of w11, w21, …, wN1 to w12, w22, …, wN2, then x is such that 2 = ′
( ′, 1 + ),
Let us denote the fact that the word a relates to the word b in the same sense as the word c relates to the word d by the following equation: (a:b) :: (c:d). For example, (apple:fruit) :: (cucumber:vegetable), (apple:apples) :: (cucumber:cucumbers), and, classical, (king:queen) :: (man:woman). In this case, a request to find an analogy looks like (king:?) :: (man:woman) or (man:woman) :: (king:?).
= + ′ − . ′ = ∉{ , ′, } ( , + ′ − ) (1) (2) (3) (4)
Evaluation of Vector Transformations for Russian Word2Vec and FastText Embeddings 3 va be vectors corresponding to the words a' and a respectively; in this case, the vector difference va' - va expresses the semantic relation (or in other words, the semantic difference) between the words a and a'. Thus, in order to find an analogue, we should find the word x and its corresponding vector y such that y - vb = va' - va, or
However, the probability of existence of a word having exactly the same vector as vx is extremely small. That is why Word2Vec finds vector y' that is the closest word to the vector y: 3
As it was mentioned above, the Word2Vec system uses an algorithm that finds the
vector nearest to the calculated one [
( , + ′ − ).
Such model is called 3CosAdd. As it was demonstrated in [
That is the why researchers have to find other methods to solve the problem of word
analogies. One of them is the Only-b [
( − , ′ − ).
The authors of 3CosMul method show in the paper [
The authors use an example (London:England)::(Baghdad:?). The component cos(v, vEngland) responses to the similarity of the answer with the country, the component cos(v, vBaghdad) responses to the cultural and geographical similarity, and the component cos(v, vLondon) responses to the cultural and geographical difference. The correct answer is Iraq, but the component cos(v, vBaghdad) dominates all other components and the resulting answer becomes Mosul. In order to eliminate such domination, the authors introduce a new formula that keeps balance among the various components: ′ = ∉{ , ′, } ( , ′) ( , )+ ( , ),
We suppose here that if va and vb in (1) are very close, then va - vb = 0 and y = vb.
The next method is Ignore-a [
∉{ , ′, } ′ =
( , ′ + )
As it was shown in [
The task of word analogies is very sensitive to the noise in the input data. A word
can be homonymous; this means that it should be presented as two or more separate
vectors representing different meanings of this word. In case of Word2Vec, such a word
will be represented only by a vector that will be a superposition of all its meanings.
Moreover, the resulting vectors of similar entities could express differences in their
occurrence with other words. For example, a dog and a cow are both animals, but dog
is a carnivore and a human’s friend, while a cow is an herbivore and gives milk; thus,
the analogy is not complete here. In order to eliminate such influence, the authors of
the 3CosAvg method [
( , + ∑ =1 ′ − ∑ =1 )
Evaluation of Vector Transformations for Russian Word2Vec and FastText Embeddings 5 4
We used several pre-trained models from the site RusVectōrēs (http://rusvectores.org/ru/models/): Araneum Upos Skipgram 2018, Ruwikiruscorpora Upos Skipgram 2018, Ruwikiruscorpora Upos Skipgram 2019, Tayga Upos Skipgram 2019, News Upos Skipgram 2019, Ruscorpora Upos CBOW 2019, Araneum Fasttext Skipgram 2018, Facebook FastText CBOW 2018 [29, 30]. The first models were trained using Word2Vec, the two later ones were trained using FastText.
For semantic analogies, we used the Russian versions of Google analogy test set [
For the purpose of evaluation, we calculated the accuracy metrics for all category examples for the given language model. Fig.1 demonstrates the results for the 3CosAdd method; Fig.2 demonstrates the results for the 3CosAvg method. Dark blue shows results with higher accuracy, up to 1; light blue shows results closer to zero.
The best results for the 3CosAdd method were 0.9 for the category Country → Adjective taught at the Russian Wikipedia. Note that this category demonstrates the best results for any used language model. It is not surprising that Country categories achieve better results on Wikipedia text model since there is more information on this topic.
The worst results are demonstrated at reflective verbs (A14) and verbs with prefixes (A15). In our experiments, we included not only the initial forms of the verb but a variety of forms: the past tense, the imperative mood, etc. That is why the resulting affine transfer vector expresses variety of characteristics, but not the only one, and failed to find one preferential direction.
Evaluation of Vector Transformations for Russian Word2Vec and FastText Embeddings 7 Some words were rare in the learning corpus, and the model failed to represent a solid vector representation for these words. That is true, for example, for the category A10 Possessive Adjective → Comparative Adjective.
The FastText model works better for categories with flexies: A8, A9, A11, A12, A13. Originally, the FastText model was created to work with character n-grams and learn grammatical features of the language. So, FastText is better situated to learn the ‘sense’ of prefixes and affixes.
The results for 3CosAvg are much better. The category A1 Famous capital → Country demonstrates 100% accuracy on Wikipedia data. However, the results for some categories are not so impressive. The categories A14 and A15 still demonstrate the same results by the same reasons. The category A3 Country → currency cannot be solved better than 37%. We can suppose that the reason is that different countries have the same name for their currency, but their vectors are different; thus, their transition vectors should be different as well. Finally, the category A7 Singular → Plural faces to the variety of Russian homonymous postfixes denoting several tags. 6
In order to find out the reasons of success and fail, we conducted a visual analysis of the Word2Vec and FastText vectors. First of all, we projected 300-dimensional vectors into 2D-space using Principal Component Analysis (PCA). Instead of t-SNE and UMAP, PCA does not create areas with non-linear skewing. As a result, parallel vectors keep their parallelism. On the other hand, there is a non-zero probability that two vectors on parallel planes could become parallel on the projection. The later case makes some distortion in the data, but not as critical as the former one.
For our experiments, we used two pre-trained language models: Araneum Upos Skipgram 2018 and Araneum Fasttext Skipgram 2018. They were trained on the same Araneum corpus. We randomly selected five word pairs for several categories presented in Fig. 3-8. All vectors are directed from a blue to a red point.
Fig. 3-8 helps us make the following conclusion. The main reason of low accuracy in the task of word analogies is the bias between vectors. It is easy to see that Word2Vec vectors in Fig. 3 are mostly parallel, excluding slightly bias for Рим-Италия (RomeItaly). The length of the vectors is also almost the same. Averaging among the beginning and ending points of vectors helps adjusting these small biases in the Word2Vec model. However, the FastText model is oriented mostly on word parts; that is why its vectors are almost randomly oriented and have no preferential direction. The same is true for Fig. 4. However, in Fig. 6 the situation is opposite. The difference between the starting and finishing points here is the prefix не- (non-, ir-). The FastText model, which vectors are parallel and have quite the same length, processes such situation better than Word2Vec. Quite the same is true for the category A13 Verb → Noun with -тель (-er, -or) (Fig. 8) where the FastText model achieves much better results.
The worst situation is shown in Fig. 5 and Fig. 7 for prefix при- and reflexive verbs, which differ from the main form by the suffix -ся or -сь. These parts of word are very homonymous. So, prefix при- could mean approaching, connection, proximity, partiality of an action, finalization of an action, etc. According to these meanings, the corresponding vectors could be perpendicular or antiparallel. This results in very low accuracy for all used models. We used just initial forms for Fig. 5 and Fig. 7 to eliminate the influence of grammatical features. 7
In this article, we found several reasons why the vector transformation does not work on some categories of word analogies. 1. A used language model should be taught on texts that have enough occurrences of words for which the task of word analogies is solved. As we can see in Fig. 1 and 2, categories including countries, their capitals, and other related words are better analyzed using corpora like Wikipedia, since such corpora have enough information for inference of logical relations among these words. Moreover, these words are allocated in the same context in the Wikipedia text; thus, their vectors become more similar. Other model, which was trained on fiction or news texts, does not have the same context. That is why categories A1-A5 demonstrate the best results on the Wikipedia corpus. Note that this conclusion needs to be proved in an independent investigation. 2. In a common case, the task of word analogies could not be solved using a random word pair taken from two investigated categories. At least, some words are homonymous, and their vectors are out of the common systems for words of the selected category. If these words are chosen as the basis in such methods as Word2Vec, then such basis will be shifted and word analogies will be found incorrectly. For example, Moscow and Berlin are respectful representatives of their countries in case of international or cultural affairs; these cities are used as synonyms of Russian and German government and culture. However, Bogota and Kampala are rather a capital city than a government. Moreover, Russian prefixes and suffixes could have different meanings. Thus, there will be several preferential directions for different prefix or suffix values. If someone misuses just one word in a pair, he or she will get a wrong transition vector.
Averaging of the starting and finishing points helps to eliminate this problem. This helps more in the case of factual information (about 30% increment for accuracy), but less for grammatical information (less than 10% for Masculine → Feminine transition and several percent for other categories in the case of Word2Vec, 1-30% depending the category for FastText). Moreover, averaging over extremely homonymous prefixes and suffixes makes the results worse (categories A11, A15 for FastText over the Araneum corpus).
Averaging over a word list has some drawbacks. One should have a list of words in a given category to average their vectors; this is not always possible. On the other hand, sometime such a list is available, and one could use it to tune vectors in order Evaluation of Vector Transformations for Russian Word2Vec and FastText Embeddings 11 to predict out-of-list words. But previously he or she should be sure that this list does not contain several preferred semantic directions.
This list could be used separately, but not for calculating pairs only. Any group will have its own center coordinates. If we have a pair a:b, we can use the coordinates of the nearest centers instead of the vectors of these words. 3. The main idea of an affine transition is that there is one vector that could be added to the word a to find its analogy b. That means that all vectors for a:b should be equal, i.e. have approximately equal length and angles of orientation. At least, the value of the bias between this transition vector and the vector of the correct answer should be less than half the distance to the nearest neighbor. As we have found out, it is not always true. In case of homonymous prefixes and suffixes, some vector groups could be oppositely directed. This means that the word analogies task could not be solved using only one transition vector. Our experiments have demonstrated that in some cases the length of the vectors could differ more than twice. Such biases lead to the situation, when the software module has to generate several outputs, and the user should find some extra methods to find the correct answer.
In our future research, we are going to investigate these drawbacks in more detail. One of the solutions here is to construct an interpretable vector space where each axis corresponds to a semantic feature. LSA-based methods successfully mark domains building a hierarchical representation of language semantics. However, these methods are not able to construct a vector space. In our opinion, incorporation of these two methods (LSA and vector space) could help achieve reasonable results. On the one hand, the vector space of Word2Vec-inherited methods allows the vector arithmetic in a continuous semantic space; on the other hand, the application of a hierarchy of terms to such a continuum makes it more interpretable.