Semantic change detection task in a relatively low-resource language like Italian is challenging. By using contextualized word embeddings, we formalize the task as a distance metric for two flexible-size sets of vectors. Various distance metrics like average Euclidean Distance, average Canberra distance, Hausdorff distance, as well as Jensen-Shannon divergence between cluster distributions based on K-means clustering and Gaussian mixture model are used. The final prediction is given by an ensemble of top-ranked words based on each distance metric. The proposed method achieved better performance than a frequency and collocation based baselines.
Lexical Semantic Change detection aims at
identifying words that change meaning over time; this
problem is of great interest for NLP,
lexicography, and linguistics. A semantic change detection
task in English, German, Latin, and Swedish was
proposed by Schlechtweg et al. (2020). Recently,
Basile et al. (2020a) organized a lexical
semantic change detection task in Italian called
DIACRIta at EVALITA 2020
Some previous approaches for semantic change
modelling were based on static word embedding,
where word vectors were trained for each
timestamped corpus and then were aligned, e.g. by
orthogonal projections
By using contextualized word embedding, each word in a specific sentence is represented as a vector depending on the neighboring words which form the context of the word; a word appearing many times in a corpus is therefore represented as a set of vectors since one vector corresponds to each occurrence). In this paper, semantic change detection is addressed by computing the distance between two flexible-size sets consisting of vectors with respect to two time-stamped corpora. We investigated several distance metrics: average Euclidean Distance, average Canberra distance, and Hausdorff distance. Our methodology also relies on a clustering algorithm (e.g. Kmeans clustering and Gaussian Mixture Model) on the joint set and calculates a Jensen–Shannon divergence between cluster distributions in the two sub-corpora. We aggregate top-ranked words based on each distance metric as the final prediction. The proposed method achieved better performance than frequency and collocation based baselines and finally ranked the 8-th among 9 participanting teams. 2
Unlike the static word embedding like Word2vec
Let us denote a corpus with m sentences as C. In this paper, C is related to a time span t because of the task characteristics; however, the corpus can be tailored to any specific aspect, e.g. a specific domain such as news or books. For a word wi appearing in C, its contextualized word representation in the k-th sentence 2 is denoted by ei(;Ck). The word representation in the corpus is a set iC = fei(;C1); ei(;C2); ei(;Ck); ; ei(;Cm) g (1)
To examine whether a word wi exhibits a semantic change between two corpora C1 (in t1) and C2 (in t2), we check the difference between two sets iC1 and iC2 . Let li be a human-annotated label indicating the semantic change degree; li usually ranges from 0 to 1, where 1 denotes a full semantic change. Let D be the dimension of the word vector. We define the distance metric as a function
f : fRDgm; fRDgn ! R: to obtain a semantic change degree based on the representation of a word in two corpora denoted as iC1 ; iC2 . When labels are binary, one may simply use a threshold on the values of f ( ; ) to predict the binary label. Let be a function to generate a binary output, e.g., based on a hand-crafted threshold. We can predict whether wi exhibits a semantic change between C1 and C2 as follows li = (f ( iC1 ; iC2 )) where li is the predicted binary label.
In conclusion, in our work the semantic change detection task is formalized as follows arg max X f; wi (f ( iC1 ; iC2 )) == li (2) (3) (4) 1An overview on word vectors is in Wang et al. (2019). 2If a word appears in a sentence more than once, we take the average.
Since this is a closed task, we may not have enough annotated samples to train a f using gradient descent. Therefore, a well-selected f will be crucial. 3 3.1
Using contextualized word embeddings like ELMO and BERT has be shown to improve performance in various downstream tasks due to its expressive power for words. In this paper, we use a multilingual-BERT3. Uncased models are adopted since we assume that semantic change detection is insensitive to word case. All models are in base settings with 12 layers, 12 heads, and a hidden state dimension of 768. Only last-layer output of BERT is used as word representation. 3.2 3.2.1
In this section, we introduce various methods to calculate the semantic change degree.
metric Distance (AGD) (also can be seen in
AGD( iC1 ; iC2 ) =
1 mn
X
x2 iC1 ;y2 iC2
d(x; y)
The distance function d( ; ) can be the Euclidean
Distance 4, the Canberra distance
Hausdorff distance. Hausdorff distance
HD( iC1 ; iC2 ) = max( sup x2 iC1 y2infiC2 jjx x2 iC2 y2infiC1 jjx sup yjj2; yjj2) (5) 3https://storage.googleapis.com/bert_ models/2018_11_03/multilingual_L-12_ H-768_A-12.zip.
4Euclidean Distance : d(x; y) = jjx yjj2 5Canberra distance is a normalized version of the Manjxi yij hattan distance, d(x; y) = PiD=1 jxij+jyij
By clustering the union set between iC1 and iC2
in K clusters/categories, we obtained the
category distributions p; q for iC1 and iC2 ,
respectively. We adopted two commonly used
clustering methods: the K-means clustering method
and the Gaussian Mixture Model method. As
for the distance between distributions, we adopted
the Jensen–Shannon Divergence (JSD), which is
a symmetrized and smoothed version of the
Kullback–Leibler divergence:
where KL(p; q) = PiK=1 pi log pqii .
We took the top-K ranked target words of each
metric and aggregated them for the final
submission. The K was decided when the aggregated
target words reached the half of total words
numbers, since we assumed that the annotated labels
are balanced. See
Labels in this task are binary and the task is considered as a binary classification problem. The evaluation is based on accuracy: T; F refers to ‘True’ and ‘False’, P; N refers to ‘positive’ and ‘negative’. For example, T P is the number of Truly-predicted Positive samples.
The task735680 organizers provided two
baselines: Frequencies: the absolute value of the
difference between the words’ frequencies is
computed; Collocations: for each word, it
computes the cosine similarity between two
Bag-ofCollocations (BoCs) vector representations related
to C1 and C2. In both baseline models, a threshold
is used to predict if the word has changed its
meaning.
In this section, we will provide a bi-dimensional
visualization of word representation to intuitively
understand how the contextualized word vectors
work. For each word, we get all contextualized
word vectors (with a dimension of 768) based on
its context. To visualized word in a 2D plane,
we used a typical dimension reduction algorithm
called T-SNE
For example, ‘rampante’ and ‘palmare’ are the predicted positive samples while ‘cappuccio’ and ‘campanello’ are predicted negative samples. As shown in Figure 1, the predicted semanticallyshifted words exhibit a clear difference between red points an blue points with respect to two timestamped corpora. For the predicted semanticallyunshifted words (see Figure 2), it looks slightly indistinguishable. 5
In
There are typically two kinds of methods to use
contextualized embeddings for semantic change
detection: embedding-based distance metrics and
clustering-based distance metrics
patterns? Many typical patterns of semantic
shifts have been investigated
Normalization. Most of distance metrics are not normalized except for negative cosine similarity. Absolute values of unnormalized distance metrics may differ a lot among individual words; they are sometimes unexpectedly affected by the number of samples, leads to that the values of metrics may not be comparable among words.
Outliers. Some distance metrics (e.g., Hausdorff distance) are sensitive to outliers. For example, since the calculation of Hausdorff distance is based on infimum and supremum, an outlier point may largely affect the final Hausdorff distance. As seen in Table 3, frequently-appearing words e.g., ‘campionato’ and ‘unico’ have the highest Hausdorff distance between C1 and C2, this is probably biased by the fact that the two words appear frequently (see Table 1) and therefore likely have more unexpected outliers.
Model Fine-tuning. The contextualized word embedding that is based on pre-trained language models like BERT achieved much better results compared to static word embedding with a twostage training paradigm, where the two stages are pre-training in language model (e.g., mask language model) and fine-tuning in downstream tasks (e.g., classifications). However, in the semantic change detection task, fine-tuning in downstream tasks is currently impossible because the annotated labels are insufficient to this aim; to some extent, the lack of fine-tuning stage may harm the performance of the pre-trained language models. 5.2
After clustering, we used the Jensen–Shannon divergence (JSD) which is affected by the issues mentioned in Section 5.1 like other distance metrics. Plus, the clustering algorithm may introduce some errors of semantic change detection. First, typical clustering algorithms may not necessarily converge to an identical clustering result when the seed centroids are changed. Moreover, the number of clusters is crucial since the optimal number of clusters cannot easily be decided before clustering. This paper formalizes semantic change detection as a distance metric between two variable-sized sets of vectors. The final prediction is based on an ensemble of different distance metrics. The proposed method outperformed weak frequency and collocation baselines, but it performed less well than SOTA baselines. As a future work, this task may be largely improved via a supervised task in a unified multi-lingual framework; thus, any human-annotated labels in other languages could be used in this task since currently the number of annotated semantically-shift words in a single language is limited.
This work is supported by the Quantum Access and Retrieval Theory (QUARTZ) project, which has received funding from the European Union‘s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 721321.
A
word matematica dettagliato sanita` senatore istruzione egemonizzare lucciola campanello trasferibile brama polisportiva palmare processare pilotato cappuccio pacchetto ape unico discriminatorio rampante campionato tac piovra