We present our systems and findings on unsupervised lexical semantic change for the Italian language in the DIACR-Ita shared-task at EVALITA 2020. The task is to determine whether a target word has evolved its meaning with time, only relying on raw-text from two time-specific datasets. We propose two models representing the target words across the periods to predict the changing words using threshold and voting schemes. Our first model solely relies on part-of-speech usage and an ensemble of distance measures. The second model uses word embedding representation to extract the neighbor's relative distances across spaces and propose “the average of absolute differences” to estimate lexical semantic change. Our models achieved competent results, ranking third in the DIACR-Ita competition. Furthermore, we experiment with the k neighbor parameter of our second model to compare the impact of using “the average of absolute differences” versus the cosine distance used in (Hamilton et al., 2016).
Lexical semantic change has recently gained
interest in the intersection of natural language
processing and historical linguistics1, therefore
several datasets have been proposed for different
languages
We present two perspectives to approach the problem, regarding how we represent target words and estimate the lexical-semantic change across datasets. (1) uses the POS distribution of target words as representation, and employee an ensemble of distance measures for the estimation. (2) uses the target words neighbor similarities as representation and one (of two proposed) similarity measure for estimation.
The following three sections describe the previous works, modeling, and results we obtained using these approaches. Following that, section 5 (Discussion) focuses on examine the second approach to illustrate the impact of the k parameter in similarity measures and the discriminatory performance of our embedding-based model. 2
Previous works have employed similar approaches
to address the unsupervised
lexical-semanticchange task, mostly for the English language
On the other hand, our second approach follows
the idea of “embedding-based models”
Our second approach align the space locally using the nearest neighbors of target words as shared feature. 3
In this section we provide a detailed description of our systems, each of them composed of two stages, the model and the voting scheme. 3.1
We represented the target words as vectors for each time of period using two perspectives that originate our submitted systems: the POS-model and the embedding-model. The word representations are comparable across spaces, and serve to estimate the lexical semantic change through similarity and distance measures, from which we finally predict the changing words using thresholds and voting schemes.
POS-model: we simply analyzes the Part-OfSpeech distribution as the relative frequency over the datasets taking the top 4 most common POStags, namely ADJ, NOUN, PROPN and VERB. The produced four-dimensional vector pairs are then used to assess the lexical semantic change of each target word from the perspective of their Euclidean, Manhattan and Cosine distances3.
Embedding-model: We lowercase and
concatenate each word form with its corresponding
POS to build embedding models for each dataset
T , namely T 0 and T 1. Specifically, we used
Word2Vec models
Given that DIACR-Ita is an unsupervised task we experiment with different threshold and voting schemes to aggregate the measure ranks and determine which target words have underwent a lexical semantic change. As a result, we propose three voting schemes from which we derive our results.
System1: Upper-third of distance ranks (used for POS model): we sorted the target words in descending order and rank their positions according to the Euclidean, Manhattan and Cosine distances. We then sum all these ranks and sort in descending order again. Finally we label the first upper-third part of this list as changing words.
System2: Half intersection (used for the embedding model): We sort the target words in descending and ascending order for the linealdifference scores (1) and the cosine-similarity (2) respectively. Then we take the top 50% of each group, and intersect them to obtained the words that we predicted as changing words.
tersection: This is just the union of results from System1 and System2.
3we noticed that at this point Kulkarni et al. (2015) uses Jenssen-Shannon divergence measure 4https://radimrehurek.com/gensim/models/word2vec.htm 5Unlike Hamilton et al. (2016) that takes the top-k neighbors from each model and union them (Nk = NkT 0 [ NkT 1). 6Hamilton et al. (2016) only uses cosine distance.
In this section we employee the gold-standard labels of the target words to analyze at Figure 1 the capabilities of our neighbor-based embeddingmodel using several settings. To this end, we divide the Figure 1 into vertical and horizontal views. The vertical view defines 3 groups (from top to bottom), that serves to compare the three proposed measures to estimate the lexical semantic change, namely the average of absolute differences, cosine similarity and cosine distance. At the same time, the horizontal view serves to compare the strategy of only use T0 (at left), versus the union of T0 and T1 (at right), to define the top nearest neighbors Nk.
7https://github.com/ajason08/ evalita2020_diacrita
Next, each of the charts shows an analysis of the model for the given measure across the k parameter. The area charts represent by color regions the ranges that discriminate the lexical semantic change of target words: “changing words” (orange region) and “non-changing words” (purple region). The yellow region in the middle marks the intersection of these ranges, thus, words falling into the yellow region are difficult to estimate, according to the used measure. We also identified the threshold that best discriminate changing and non-changing target words, and draw a dashed line at that point. On the other hand, the line charts throw light on all the possible performance that the model could obtain by changing the k parameter while using the best possible discriminator threshold.
These results suggest that the “average of absolute difference” is the best proposed measure because it obtains a better performance for a larger number of k values as displayed in the line charts. Moreover, the “average of absolute difference” offers a larger range for possible discriminator thresholds (as shown in the area charts), and it is tolerant to the Nk election, since it remains almost unchanged while using either the union of T0 and T1, or only T0. One can also note that the area charts for the cosine similarity versus cosine distance mirror each other, as expected, and their performance is the same when using Nk only from T0 (at left), but slightly differ when using Nk as the union of T0 and T1 (at right). 6
We tackle the problem of unsupervised lexical semantic change on two time-specific datasets for 18 target words in Italian language. Our two approaches focus on the representation of target words across the provided diachronic datasets, they use part-of-speech usage and nearest neighbors respectively, and a number of measures between these representation to estimate the lexical semantic change. Then, this estimation serves to decide which target words underwent a change by the use of proposed threshold and voting schemes. Afterward, in the last part of this work, we analyzed the nearest neighbor model through the impact of deciding the k parameter and the similarity measure that estimates the lexical semantic change. Our results for the DIACR-Ita datasets suggest that the estimations of “the average of absolute differences” measures have a better performance for a larger number of k values than the cosine similarity and the cosine distance used in Hamilton et al. (2016).
As for future work, we plan to investigate different mechanism for deciding the threshold, and explore other diachronic datasets for other languages such as English, German and Spanish. We also believe that further experiments on a larger number of target words will benefit the reliability of models to judge the lexical semantic change in an unsupervised fashion.
The authors thank CONACYT for the computer resources provided through the INAOE Supercomputing Laboratory’s Deep Learning Platform for Language Technologies. Web), page 625–635, Florence, Italy, May. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland.