Word embeddings are real-valued word representations capable of capturing lexical semantics and trained on natural language corpora. Word embedding models have gained popularity in recent years, but the issue of selecting the most adequate word embedding evaluation methods remains open. This paper presents research on adaptation of the intrinsic similarity and relatedness task for the Lithuanian language and the evaluation of word embedding models, testing the quality of representations independently of specific natural language processing tasks. 7 different evaluation benchmarks were adapted for the Lithuanian language and 50 word embedding models were trained using fastText, GloVe, and Word2vec algorithms and evaluated on syntactic and semantic similarity tasks. The obtained results suggest that for the intrinsic similarity and relatedness task, the dimension parameter has a significant impact on the evaluation results, with larger word embedding dimension yielding better results.
The development of natural language processing tools influenced a growing need for word embeddings as real-valued representations of words for text analytics, generated by applying distributive semantic models. While word embeddings have become one of the most widely used tools in modern natural language processing (NLP) applications, their limitations have not yet been fully explored. The problem of assessing word embedding consistency and quality is one of the most relevant questions in distributive semantics research.
The idea of word embeddings is not new, but it gained popularity after Mikolov et al. [
There are two types of word embedding evaluation: intrinsic and extrinsic. Intrinsic evaluation tests the representation quality independent of specific natural language processing (NLP) tasks, while extrinsic evaluation uses word embeddings as input features to an NLP task and measures changes in corresponding performance metrics. We focus on intrinsic evaluation methods, based on human annotated datasets, because datasets can be adapted for different languages by translating and reevaluating human annotated scores.
The method of word semantic similarity, based on correlation with human judgment of how closely words are related among themselves, was one of the first intrinsic evaluation metrics for distributional meaning representations. According to this method, words smart and intelligent should be closer in the vector space than smart and dumb, since smart and intelligent are intuitively better semantically related.
There are gold-standard benchmarks for evaluating distributive semantic models such as SimLex999
[
The word analogy method aims to identify words based on operation prediction in a word vector
space. The method tries to predict a missing word in a word pair based on a known relationship in
another word pair. Thus, for a dataset a–b, c–d, the task is to identify an unknown word d based on the
known relationship between words a and b. For example, given the words, a (brother), b (sister), and c
(father), this method should correctly predict the value mother for word d [
The word clustering method evaluates a word embedding space by applying the word clustering
approach. It is aimed at splitting a given word set into groups of words corresponding to different
categories based on word vectors. For example, words dog and cat belong to one cluster, while words
car and plane – to another [
The situation with word embeddings for the Lithuanian language is influenced by its specifics. The
Lithuanian language is a morphologically rich Baltic language, being considered one of the most archaic
living Indo-European languages [
The goal of this research was to adapt selected intrinsic similarity benchmarks for the Lithuanian language and to apply them for experimental evaluation of fastText, Word2vec, and GloVe embedding models with different hyperparameters.
In order to reach this goal, we perform the following tasks: related work analysis (Section 2), corpus building for embedding training (Section 3), methodology for the adaptation of evaluation benchmarks for the Lithuanian language (Section 4), experimental evaluation of different embeddings based on the derived benchmarks (Section 5), conclusions and future plans (Section 6).
In recent years, there have been several critical articles on intrinsic assessment methods: some
researchers address the subjectivity of human judgments, the vagueness of instructions for particular
tasks, and terminology confusions [
There have been successful attempts to adapt intrinsic evaluation benchmarks to other languages.
Research has shown that when monolingual vector space models were translated into German, Russian,
and Italian, it became clear that their predictions did not always correlate well with human decisions
made in the language used for model training [
Another study attempted to translate the SymLex999 benchmark into Estonian and discovered that,
unlike in the original research, computational word embedding models better correlate with noun scores
rather than adjective scores [
A few studies on the evaluation of Lithuanian word embeddings have been carried out. In the first
study, word embeddings for different models and training algorithms were evaluated against a limited
implementation of the Lithuanian WordNet [
The second study compared traditional and deep learning approaches for sentiment analysis using word embeddings, finding that deep learning performed well only when applied to small datasets, and that traditional methods performed better in all other contexts [16].
The third study was conducted with Transformer models using GloVe word embeddings [17]. The study concluded that multilingual transformer models can be fine-tuned to word vectors, but still perform much worse than specifically trained embeddings.
In conclusion, we see that the Lithuanian language lacks word embedding evaluation benchmarks.
Semantic intrinsic similarity benchmarks cover many different types of test domains, such as geography, languages, currency, etc. Therefore, we need to have a wide variety of data for embedding training. Research has shown that for a larger corpus we get better word embeddings [18]. For this reason, it is important to build an extensive corpus for embedding training, that will be further used for evaluation.
Wikipedia texts are usually a typical approach for building a corpus for embedding training. In order to expand our experimental corpus, we used articles from Lithuanian news portals, mainly from the largest one, Delfi.lt, the collected articles covering different topical areas such as news, cars, fitness, culture, food, and so on.
In order to obtain better word embeddings, we also included texts from the Corpus of Contemporary Lithuanian Language (CCLL) [19], texts in a variety of genres and topics.
Statistics for our combined experimental corpus is presented in Table 1.
The pre-processing phase consists of two steps: 1) breaking text into tokens, lowercasing text, removing special symbols, numbers, non-Lithuanian words and stop-words 2) removing short documents, less than 50 characters in size; 3) lemmatizing the texts, this being important for rich morphology languages [20]. Lemmatization was performed using lexical and morphological analysis tools from the Lithuanian language technology infrastructure built in the Semantika2 [21] project.
Alternatively, text could have been stemmed instead, but lemmatization was preferred, as all our documents were in normative spelling and punctuation. Stemming is more favorable in case of social texts with lots of out-of-dictionary words. Also, stemming has its limitations, e.g. over-stemming and under-stemming problems [22].
The statistics for the final version of our experimental corpus are presented in Table 2.
The methodology part covers the methods applied in this research: (1) benchmark dataset adaptation method for semantic similarity based intrinsic embedding model evaluation; (2) semantic similarity based embedding evaluation using the adapted benchmarks.
As a result of a brief analysis, the following English-language benchmarks were selected for the adaptation to the Lithuanian language, their popularity being the main criteria: 1. MEN (Marco, Elia and Nam), 3 000 pairs [23]. 2. WordSim-353, 353 pairs assessed by semantic similarity [24]. 3. WordSim-353-REL, 252 pairs [25]. 4. WordSim-353-SIM, 203 pairs [26]. 5. SimLex-999, 999 pairs [27]. 6. MTurk-287, 287 pairs [28].
7. RG-65, 65 pairs [29].
There are 5160-word pairs (2191 unique words) in total across all datasets.
The following algorithm was applied for the dataset adaptation to the Lithuanian language: 1. Automated translation of datasets (by applying the Google Cloud Translation API) [30]. 2. Inconsistency checking (manual examination), discarding inconsistent word pairs. 3. Word lemmatization. 4. Re-evalution of the score that was initially assigned to the English language word pairs was done by two independent persons (manual procedure). An average score was calculated.
( , ) =
As mentioned in Chapter 1, the semantic similarity datasets are based on correlation with human judgments of how closely words are related.
The similarity benchmark datasets consist of a certain number of word pairs. Each pair is determined
by its similarity and relatedness. The values are in the range [
The word embedding models are represented by corresponding vectors for each word in the dictionary. If a word is missing in the trained word embedding model, it is replaced by the mean of all vectors. In order to calculate the similarity between vectors, we can use the cosine similarity formula (see Eq. 1), where a and b are vectors in the word embedding vector space.
. (1)
|| || × || ||
here ai and bi are vectors of N-dimension. The result of cosine similarity is a value in the range [
A human-annotated benchmark dataset consists of n triplets containing pairs of words and their corresponding similarity scores ⟨wi , wj , hij⟩, where wi , wj are dictionary words, and hij is the score.
Let h = (hi1, hi2, . . . , hiN ) be a vector of human annotated benchmark datasets, and m = (mi1, mi2, . . . , miN ), correspondingly, a vector of similarity scores calculated from word embeddings.
Then, the evaluation score for the corresponding embedding model, based on the selected benchmark, is calculated as Spearman’s correlation ρ (see Eq. 2) between h and m.
Spearman value can be any value satisfying−1 ≤ ≤ 1, and the interpretation is that p values close to +1 indicate stronger relationship, while those closer to -1 indicate weaker relationship.
The Spearman correlation formula is: where n – dataset length, d – difference between ranks of h and m.
The aggregated score of one-word embedding model is calculated (see Eq. 3) as: where is Spearman correlation value of specific benchmark, n – number of benchmarks.
In order to compare different embedding model types (Word2vec, fastText, GloVe). We can calculate an average score of all models’ embeddings (see Eq. 4). where – an average score of all t word embeddings, n – number of t word embedding models.
Experiments were carried out in a series of tasks: (2) (3) (4) accumulated experimental corpus. evaluation benchmarks.
hyperparameters on benchmark evaluation results.
Firstly, 7 selected evaluation benchmark datasets were adapted for the Lithuanian language. Secondly, 50 word embeddings with different hyperparameter sets were trained on the Thirdly, the obtained word embedding models were evaluated using the adapted intrinsic Finally, the resulting data were examined in order to determine the effect of different Excerpt of the adapted of SimLex999 benchmark for the Lithuanian language
The selected 7 (see Chapter 4.1) evaluation benchmark datasets were adapted from English to Lithuanian language. There were 5610 word pairs at the beginning. After the adaptation process, 5573 word pairs remained. A total of 37 word pairs were discarded. The following problems were observed during the adaptation process: 1.
Multiple words – in some cases, one-to-one word translation is not possible, when a two-word expression in the Lithuanian language is a correspondence to a single word in the English language. For example, for the word pair “computer – software”, the Lithuanian translation would be “kompiuteris – programinė įranga”. As we use vector-to-vector comparison, such word pairs were
The meaning of certain words has been shaped by American culture, e.g. words like soccer, football, and FBI. These are words that are commonly used in the US. Such words were replaced discarded. with Lithuanian synonyms were discarded.
A few older words have undergone semantic changes as their meanings evolved. For example, the word pair “Arafat – terror”, had a greater similarity back in history than it does now. Such pairs
In some cases, both English words have the same meaning in the Lithuanian language, for example, the following pairs: “smart – intelligent”, “happy – cheerful”, “fast – rapid”. Such word pairs as a result contained two equal words, and their scores were set to 10 (maximum similarity). An excerpt of the adapted SimLex999 dataset for the Lithuanian language is presented in Table 3. word1 steak nail band book meat thumb orchestra bible
The first two columns contain an English word pair in its original form. The third column contains the human-generated similarity score. The fourth and fifth columns contain Lithuanian translations of English words and revalued Lithuanian word scores.
The following tools were used for word embedding training: python genism wrapper of Word2vec1, fastText – official python library2, GloVe - official library3. We used similar training parameters in order to be able to compare different word embeddings (see Table 4).
All the trained embedding models were evaluated using the Spearman ρ correlation coefficient between human benchmark scores and vector space model scores. The results were grouped by different vector model types, characterized by different hyperparameter sets (see Table 4).
The best 4 and the worst 4 models ranked by the benchmark result average are presented correspondingly in Table 5 and Table 6. The first column in these tables indicates model name together with hyperparameter indication. The following labels are used: N – negative sampling, S – SoftMax, CBOW – Continuous Bag of Words, SKIP – Skipgram, d – dimension, w – window size, m – minimum count threshold, i – iteration count. The rest are benchmark names and Spearman ρ correlation scores. The last column shows aggregated Spearman correlation score of all the benchmarks. 1 https://radimrehurek.com/gensim/models/word2vec.html 2 https://github.com/facebookresearch/fastText/ 3 https://github.com/stanfordnlp/GloVe
300d 5w 5m 5i
300d 5w 2m 5i
100d 5w 5m 5i
GloVe 300d 10w 1m 5i GloVe 100d 10w 2m 5i
MEN
Comparison between different types of embeddings (Word2vec, fastText, GloVe) was done by averaging by embedding type (see Eq. (4).
To be able to do score comparison, only embedding models with the same hyperparameters were used: dimensions (100, 300), window size (5), and minimum count (2, 5) (Figure 1).
GloVe's word embedding model scores were on average lower than those of fastText and Word2vec. The previous two were nearly identical, with a difference of only 0,001 between them.
Additionally, the experiment results were analyzed to determine whether a particular hyperparameter had a significant effect on the results. Following a thorough examination of all the hyperparameters, we discovered a correlation between the dimension value and the correlation results. (Figure 2).
Different dimension values for various embedding types had a significant effect on the results. The larger the dimension of the word embedding, the more accurate the results. As illustrated in Figure 3, as vector size increases, the model correlation score also increases.
between vector size (dim) hyperparameter and benchmark We can use Pearson correlation score (see Eq. 5) to see if there is correlation between values. =
(∑ ) − (∑ )(∑ ) √[ ∑ 2 − (∑ )2] [ ∑ 2 − (∑ )2] (5) where n – number of models, x – dimension value. y – Spearman correlation value for a model. = 0.918, this indicates strong relationship between values.
This was the first attempt to adapt the most popular intrinsic similarity and relatedness benchmark datasets for the Lithuanian language. Despite reported challenges when adapting benchmarks to other languages, we proved, that this can be done even for morphology rich languages like Lithuanian.
The application of the adapted benchmark datasets for the evaluation of the embedding models, trained on an experimental corpus, showed, that GloVe model performed worse than fastText and Word2vec, judging by average benchmark results.
We also conclude, that for the intrinsic similarity and relatedness task, the dimension hyperparameter has a significant impact on the evaluation results, with larger word embedding dimension yielding better results.
In the future, we plan to adapt other types of embedding evaluation benchmarks, such as categorization and analogy testing, as well as extrinsic evaluation with POS tagging, named entity recognition (NER), and other NLP tasks. This would allow us to compare intrinsic and extrinsic evaluation methods. Also, we will continue to expand our corpus for future tests.
[16] Kapočiūtė-Dzikienė, Jurgita, Robertas Damaševičius, and Marcin Woźniak. "Sentiment analysis of lithuanian texts using traditional and deep learning approaches." Computers 8.1 (2019): 4. [17] Stankevičius, Lukas, and Mantas Lukoševičius. "Testing pre-trained Transformer models for
Lithuanian news clustering." arXiv preprint arXiv:2004.03461 (2020). [18] Lai, Siwei, et al. "How to generate a good word embedding." IEEE Intelligent Systems 31.6 (2016): 5-14. [19] Vytauto Didžiojo universitetas. Kompiuterinės lingvistikos centras. Dabartinės lietuvių kalbos tekstynas. 1998-2016, http://tekstynas.vdu.lt/tekstynas. [20] Kutuzov, Andrey, and Elizaveta Kuzmenko. "To lemmatize or not to lemmatize: how word normalisation affects ELMo performance in word sense disambiguation." arXiv preprint arXiv:1909.03135 (2019). [21] VDU vykdo ES finansuojamą projektą „SEMANTIKA 2“, 2020. URL: https://www.vdu.lt/lt/vduvykdo-es-finansuojama-projekta-semantika-2/ [22] Jivani, Anjali Ganesh. "A comparative study of stemming algorithms." Int. J. Comp. Tech. Appl 2.6 (2011): 1930-1938. [23] Bruni, Elia, Nam-Khanh Tran, and Marco Baroni. "Multimodal distributional semantics." Journal of artificial intelligence research 49 (2014): 1-47. [24] Finkelstein, Lev, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. (2002) Placing Search in Context: The Concept Revisited. ACM Transactions on Information Systems, 20(1):116-131. [25] Agirre, Eneko, et al. "A study on similarity and relatedness using distributional and wordnet-based approaches." (2009). [26] Agirre, Eneko, et al. "A study on similarity and relatedness using distributional and wordnet-based approaches." (2009). [27] Hill, Felix, Kyunghyun Cho, and Anna Korhonen. "Learning distributed representations of sentences from unlabelled data." arXiv preprint arXiv:1602.03483 (2016). [28] Radinsky, Kira, et al. "A word at a time: computing word relatedness using temporal semantic analysis." Proceedings of the 20th international conference on World wide web. 2011. [29] Rubenstein, Herbert, and John B. Goodenough. "Contextual correlates of synonymy."
Communications of the ACM 8.10 (1965): 627-633. [30] Fast, dynamic translation tailored to your content needs, 2021. URL: https://cloud.google.com/translate