In this paper, we present our results related to the EVALITA 2020 challenge, DIACR-Ita, for semantic change detection for the Italian language. Our approach is based on measuring the semantic distance across time-specific word vectors generated with Compass-aligned Distributional Embeddings (CADE). We first generate temporal embeddings with CADE, a strategy to align word embeddings that are specific for each time period; the quality of this alignment is the main asset of our proposal. We then measure the semantic shift of each word, combining two different semantic shift measures. Eventually, we classify a word meaning as changed or not changed by defining a threshold over the semantic distance across time.
Semantic change detection is the task of detecting
if a word has shifted in meaning between different
periods of time
The task is described as follows: for training, two corpora t1 and t2, consisting of text coming from different periods are given, for testing, a set of unlabeled target words is given, where for each of them a binary scores has to be predicted: 1 identifies lexical change between t1 and t2 while 0 does not.
In this paper, we present our approach to semantic change detection that is based on two components: 1) an alignment procedure to generate distributional vector spaces that are comparable for t1 and t2 and 2) the use of distance metrics to compute the degree of semantic change for a given word. Our alignment procedure is based on Compass Aligned Distributional Embeddings (CADE) proposed by Bianchi et al. (2020) (note the approach was introduced as Temporal Word Embeddings with a Compass by Di Carlo et al. (2019), but the name was changed to enforce the idea that the embeddings can be used to align more general corpora and not just diachronic ones). Given the aligned embeddings, we use two measures to compute the degree of change based on the similarities of the vectors in the embedded space. Our results show that our methodology for aligning spaces can be useful in detecting lexical semantic change. 2
Our approach is based on measuring the
semantic distance across time of time-specific word
vectors generated with CADE and on the use of two
measures for detecting semantic shifts i.e., the
semantic distance between word vectors across time.
This distance can be interpreted as a function of
the words’ self-similarity across time, where the
similarity is measured by a linear combination of
cosine and second-order similarity
Finally, a threshold over this self-similarity is used to classify a word as changed or not changed.
This methodology was applied also in the
semantic shift detection challenge presented at
SemEval2020
Word2vec
There are different approaches to generate these
aligned embeddings (see for example the work by
Given in input a set of slices of text, where each slice corresponds to text coming from a specific period of time, the alignment procedure is as follows:
First, the text from all the slices is concatenated and CBOW is run on this corpus in order to obtain a “compass” model, i.e., a model defining the embedding space. The CBOW model uses two matrices to generate the embeddings (U and C in Figure 1), one for the context words and one for the target words. The target word matrix of the compass is then used to initialize the target matri1Check the belerico entry in the challenge leaderboard at https://competitions.codalab.org/ competitions/20948#results ces for each new CBOW model fitted on each of the slices. During training, these new target matrices are frozen, i.e., they are not updated during the training on the slice. This ensures that at the end of the training process, the various temporal embeddings are all aligned in the same embedding space, making them comparable without losing their individual temporal distinctions. We use the publicly available online implementation of CADE.2 2.2
Once the embeddings are aligned, we need measures to evaluate the degree of semantic change. We compute the semantic shift of each word, i.e. the semantic distance between word vectors across time using the combination of two different measures: Local Neighbors (ln), introduced by Hamilton et al. (2016a) and cosine similarity (cos), merging them with a weighted linear combination into a new measure called Move.
Local Neighbors ln is based on the similarity between a word and its neighbor words in the two different time periods. Essentially we compute the degree of semantic change of the word w in two slices by first collecting the nearest neighbors (NNs) of wt and wt+1 in the two respective slices, then given the embeddings at time t the similarities between the vector of wt and the vectors of all the neighbors are computed.3 The same process is run for time t + 1 with wt+1, eventually giving us two vectors of similarity scores. These two vectors are again compared using cosine similarity. The higher the value of this measure the less the vector has changed with respect to its neighbors and thus the less the word should have shifted in meaning. Cosine Similarity The second measure we use is simply the cosine similarity of the vectors of a word in two different time periods. Similarly as before , the higher the value the less the vector has changed and thus the less the word should have shifted in meaning.
The Move Measure We merge these measures together using a weighted linear combination, that is: s(wt; wt+1) = (1
) ln(wt; wt+1) +
cos-sim(wt; wt+1) 2http://github.com/vinid/cade 3When a neighbor is missing in one time slice, we replace it with the average vector of the space.
Di Ci
Dn
Cn
U 1) Train the compass from the concatenation
C
U 2) Initialize and freeze each CBOW target matrix with the same U matrix 3) training 3) training 3) training 3) training 3) training U
U
U
U with 2 [0; 1]. In particular express the usage strength of the two measures: a high will shift Move towards the cosine similarity, while a low one towards the ln measure. As introduced before we classify if the meaning has changed by defining a threshold over s (more details about this are presented in the next Section). 3
The dataset provided by the challenge’s organizers
For the training data we used the flat version
with only the lemmas, obtained by the
organizers’ script
We use the embeddings aligned with CADE and the move measure. The parameters of the moving average we need to consider are: the number of nearest neighbors (NNs) to be collected by ln, for the moving average and the threshold for the similarity. We set the threshold to decide if a word is stable or not is set to 0.7, with the decision given by: (0 if s(wt; wt+1)
0:7
Essentially, the less changed are the two vectors of the words (for cos) and the neighbors (for ln) the more the word has been stable between the two time periods. As heuristics we chose 2 f0:3; 0:5; 0:7g to evaluate the relationship between the two measures used to build move, and we set to 22 the number of nearest neighbors to be considered by the ln; this is the general setup that gave the results that have been submitted to the challenge.
We trained CADE for 10 epochs to learn 100dimensional vectors, with the window size set to 5, 10 negative examples for every positive one, with the initial learning rate set to 0.025 and decreased linearly during training.
As other models, in the post evaluation we also considered one that only uses the cos (CADE (cos)) similarity measure and one that uses only the ln metric CADE (ln)) (again with 0.7 as threshold and with the number of NNs for ln set to 22).
As baselines, the authors propose to use
baseline-freq, that is the absolute value of the
difference between the words’ frequencies and
baseline-colloc, where the Bag-of-Collocations of
the two words in the two different periods is built
and then cosine similarity is applied. A
threshold is used on both metrics to define semantic
change
CADE (move)y
CADE (ln) / / / 0.3 / / / / / / / / The evaluation metric used in this challenge is the accuracy, that is, the number of correct predictions over the target data. Table 1 shows the results. Our model was the third most accurate. However, in the post-evaluation we discovered that just using the ln metric and ignoring the use of cos (this is equivalent to using = 0 in our move measure) improves the performance leading to the second best accuracy score in the leaderboard. 4
Our results show that CADE
A
We report in Tables 2 and 3 CADE’s misclassifications with the two best metrics, namely CADE (move) with = 0.3 and CADE (ln). Eventually, we also show in Tables 4 and 5 some examples of neighborhood for the target words.
Wrong predictions done by CADE
= 0.3.
Table 4 shows the top 10 nearest neighbors of the target word “pacchetto” and we think CADE classifies its meaning as changed because during time t1 the meaning is more focused in the economic area, as one can see from neighbors like “azionario”, “obbligazione” or “contante” (translated to “stock” as referred to the market, “bond” and “cash” resp.); while at time t2 shifts to a more political sense, as shown by words such as “decreto” or “emendamento” (“decree” and “amendment” resp.). azionario obbligazione azionista azionano edison casseforte contante siap shell prestire maxiemendamento finanziaria decretone decreto ddl emendamento liberalizzazioni decretere maxidecreto ecobonus
The same it seems to happen for the target word “piovra”, as one can see from Table 5, where at time t1 CADE gathers senses from both considering it as the animal, for example from the word “tentacle”, or as someone tied to crime in general, given words such as “profittatore” or “ruberia” (“profiteer” and “robbery” resp.); while at time t2 captures a shift towards the Italian crime TV series “La piovra”, as emerge from words such as “fiction”, “camorra” or “retequattro”, which is an Italian television channel.