English. In this work we analyze the performances of two of the most used word embeddings algorithms, skip-gram and continuous bag of words on Italian language. These algorithms have many hyper-parameter that have to be carefully tuned in order to obtain accurate word representation in vectorial space. We provide an extensive analysis and an evaluation, showing what are the best configuration of parameters for specific analogy tasks.
The distributional hypothesis of language, set
forth by Firth (1935) and Harris (1954), states
that the meaning of a word can be inferred from
the contexts in which it is used. Using the
cooccurrence of words in a large corpus, we can
observe for example that the contexts in which client
is used are very similar to those in which customer
occur, while less similar to those in which
waitress or retailer occur. A wide range of algorithms
have been developed to exploit these properties.
Recently, one of the most widely used method in
many natural language processing (NLP) tasks is
word embeddings
In Mikolov et al. (2013), the authors propose two different models that seek to maximize, respectively, the probability of a word given its context (Continuous bag-of-word model), and the probability of the surrounding words (before and after the current word) given the current word (Skip-gram model). In this work we seek to further explore the relationships by generating word embedding for over 40 different parameterizations of the continuous bag-of-words (CBOW) and the skip-gram (SG) architectures, since as shown in Levy et al. (2015) the choice of the hyperparameters heavily affect the construction of the embedding spaces.
Specifically our contributions include:
Word embedding. The analysis of how
different hyper-parameters can achieve
different accuracy levels in relation recovery tasks
Word embeddings have demonstrated to capture semantic and syntactic properties, we compare two different objectives to recover relational similarities for semantic and morph-syntactical tasks.
Qualitative analysis. We investigate problematic cases.
The interest that word embedding models have
achieved in the NLP international community has
recently been confirmed by the increasing
number of studies that are adopting these algorithms
in languages different from English. One of the
first example is the Polyglot project that produced
word embedding for 117 languages
However, these optimistic results are not confirmed by more recent studies. Indeed the performance of word embedding are not directly comparable in the accuracy test to those obtained in the English language. For example, Attardi and Simi (2014) combining the word embeddings in a dependency parser have not observed improvements over a baseline system not using such features. Berardi et al. (2015) found a 47% accuracy on the Italian versus 60% accuracy on the English. The results may be a sign of a higher complexity of Italian with respect to English as we will see section 4.1.
Similarly, recent work that trained word
embeddings on tweets have highlighted some
criticalities. One of these aspects is how the morphology
of a word is opaque to word embeddings. Indeed,
the relatedness of the meaning of a lemma’s
different word forms, its different string representations,
is not systematically encoded. This means that in
morphologically rich languages with long-tailed
frequency distributions, even some word
embedding representations for word forms of common
lemmata may become very poor
For this reason, some recent contribution on Italian tweets have tried to capture these aspects. Tamburini (2016) trained SG on a set of 200 million tweets. He proposed a PoS-tagging system integrating neural representation models and a morphological analyzer, exhibiting a very good accuracy. Similarly, Stemle (2016) proposes a system that uses word embeddings and augment the WE representations with character-level representations of the word beginnings and endings.
We have observed that in these studies the
authors used either the most common set-up of
parameters gathered from the literature
Previous results on the word analogy tasks have
been reported using vectors obtained with
proprietary corpora
The hyper-parameters used to construct the different embeddings for the SG and the CBOW models are: the size of the vectors (dim), the window size of the words contexts (w), the minimum number word occurrences (m) and the number of negative samples (n). The values that these hyperparameters can take are shown in Table 1. 4
The obtained embedding1 spaces are evaluated
on an word analogy task, using a enriched
version of the Google word analogy test
To recover these relations two different
methods are used: 3COSADD (Eq. 1)
With respect to the size of the context (w) used to create the words representations we do not observe a clear difference between the 18 pairs both in the SG and in the CBOW. On the contrary a clear trend can be observed varying the n hyperparameter, with n = 1 the accuracy was significantly lower than the one we obtained with n = 5 or n = 10. Increasing the number of negative samples constantly increases the accuracy.
These results support also the claim put forward
by
If we restrict the error analysis to the most macroscopic differences in figure 1 we can compare three different parametrizations: SG-200 w5-m5n1, SG-500 w5-m5-n1, SG-500 w5-m5-n10. In this way we can analyze the results obtained changing the number of dimensions of the vectors and the role played by n. In Table 4 the total number of errors and the number of different words that have not been recovered by each parametrization are presented. From this table we can see that most of the errors are done one a relatively small set of words. This phenomenon can be studied analyzing the most problematic cases. In Table 5 we can see the list of the most common errors ranked by frequency for each method. As we can SG-200-w5-m5-n10 # SG-500 w5-m5-n1 # SG-500 w5-m5-n10 # california 328 california 349 california 287 texas 223 texas 224 texas 165 arizona 164 arizona 164 arizona 145 florida 144 ohio 142 florida 124 ohio 135 florida 140 ohio 112 see from these lists the errors are done on the same words and this because they are the most common in the dataset (e.g.: in the dataset there are 217 queries that require Florida as answer compared to the 55 of Italia). However if we compare the frequency of these errors in the analogy test within the three parametrisation we can observe an improvement of approximately 15% in accuracy with SG-500 w5-m5-n10. Indeed, despite many errors are not recovered for any of the parametrisation, we can observe that approximately 21% of the errors are recovered under certain parametrizations (Table 6). To further investigate these improvements related to the aforementioned parametrisation we focused on one of the most frequent errors in the analogy test, the word California. As we can see from the list of the analogy test solved (Table 7) different parametrizations are helpful to solve different types of analogies. For example an increase in the dimensionality increases the accuracy, but mainly in analogy test with words that have a representation in the training data related to a wider set of contexts (Houston:Texas; Chicago:Illinois). The best parametrisation is obtained increasing the negative sampling. As we can see from the examples provided, the analogies are resolved thanks to a contextual similarity between the two pairs (Huntsville:Alabama; Oakland:California). In these cases the negative sampling could help to filter out from each representation those words that are not expected to be relevant for the words embeddings.
Similar types of improvement are noticed on analogy tests that contain a challenging word predire (predict). The results of this analysis are presented in Table 9 where it is possible to see that an higher dimensionality improves the accuracy of analogical tests containing open domain verbs (e.g.: descrivere, vedere). Similarly to the previous case, an higher dimensionality allows for fine grained partitions improving the correct associations between terms. However, also in in this case, the best parametrizations are obtained increasing the negative sampling or both the parameters. As we can see here both the present participle and the past tense pairs are correctly solved. These example provide a preliminary evidence of how negative sampling, filtering out non informative words from the relevant context of each word, is able to build representation by opposition that are beneficial both for semantic and syntactic associations.
Examples of words that almost always are not recovered correctly are presented in Table 10. A selected list of words problematic for all parametrizations is shown in Table 8. It contains plurals, feminine, currencies, superlatives and ambiguous words. The low performances on these cases can be explained by the poor coverage of these categories in the training data. In particular, it would be interesting to study the case of feminine and to analyze if it is due to a gender bias in the Italian Wikipedia, as a preliminary analysis of the most frequent errors that persist in all the parametrization seems to suggest. The words that have been benefited by the increase of n are: the errors that have been introduced increasing this parameter are related to the words in Table 11. It is interesting to notice that given an error in an analogy test, it is possible to find the correct answer in the top five most similar words to the query. Precisely we observed this phenomenon in 26% of the cases for SG-200-w5-m5-n10, in 27% of the cases for SG-500-w5-m5-n1 and in 25% for SG-500w5-m5-n1. Furthermore, approximately in 50% of these cases the correct answer is the second most similar. Most of the recovery errors are due to vocabulary issues. In fact, many words of the test set have no correspondence in the developed embedding spaces. This is due to the low frequency of many words that are not in the training corpus or that have been removed from the vocabulary because of their (low) frequency. For this reason we kept the m hyper-parameter very low (e.g., 1 and 5), in counter-tendency with recent works that use larger corpora and then remove infrequent words setting m with high values (e.g., 50 or 100). In fact, with increasing value of m the number of not given answers increases rapidly. It passes from 300 (m = 1) to 893 (m = 5).
Some of the words that are not present in the vocabulary with m = 1 include plural verbs (1st person), that probably are not used by a typical Wikipedia editor and remote past verbs (1st person), a tense that in recent years is disappearing from written and spoken Italian. Some of these verbs are: giochiamo affiliamo rallentiamo zappiamo implementai rallentai mescolai nuotai
In Berardi et al. (2015) the number of not given answer is 1.220. The accuracy of their embeddings, obtained using a larger corpus and using the hyper-parameters that perform well on English language, is always lower than those obtained with our setting, in both the morphosyntactic and the semantic tasks. This confirms that the regularization of the parameters is crucial for good representation of the embeddings, since the Berardi et al. (2015)’s model has been trained on a much larger corpus and for this should outperform ours. Furthermore, this model seems to have some tokenization problem. 5
We have tested two word representation methods: SG and CBOW training them only on a dump of the Italian Wikipedia. We compared the results of the two models using 12 combinations of hyperparameters.
We have adopted a simple word analogy test to evaluate the generated word embeddings. The results have shown that increasing the number of pilotesse pere real kwanza aquila migliore colori meloni matrigna figliastra sua lev yen mamma vantaggiosissimo urlano stimano eroina programmato impossibilmente dim = 500 & n = 10 solo n = 10 dire detto predire predetto cantare cantato predire predetto mescolare mescolando predire predicendo correre correndo predire predicendo predire predicendo generare generando generare generando predire predicendo rallentare rallentando predire predicendo predire predicendo programmare programmando scoprire scoprendo predire predicendo scrivere scrivendo predire predicendo solo dim = 500 descrivere descritto predire predetto vedere visto predire predetto dimensions and the number of negative examples improve the performance of both the models.
These types of improvement seems to be beneficial only for the semantic relationships. On the contrary the syntactical relationship are negatively affected by the low frequency of many of its terms. This should be related to the morphological complexity of Italian. In the future it would be helpful to represent the spatial relationship regarding specific syntactical domain in order to evaluate the contribution of hyper-parametrization to syntactical relationship accuracy. Moreover future work will include the testing of these word embedding parametrizations in practical applications (e.g. analysis of patents’descriptions and books’ corpora).
Part of this work has been conducted during a collaboration of the first author with DocFlow Italia. All the experiments in this paper have been conducted on the SCSCF multiprocessor cluster system at University Ca’ Foscari of Venice.