This paper introduces Kronos-it, a dataset for the evaluation of semantic change point detection algorithms for the Italian language. The dataset is automatically built by using a web scraping strategy. We provide a detailed description about the dataset and its generation, and four state-of-the-art approaches for the semantic change point detection are benchmarked by exploiting the Italian Google ngrams corpus.
Computational approaches to the problem of
language change have been gaining momentum over
the last decade. The availability of long-term and
large-scale digital corpora, and the effectiveness of
methods for representing words over time, are the
prerequisite behind this interest. However, only
few attempts have focused on the evaluation, due
to two main issues. First, the amount of data
involved limits the possibility to perform a manual
evaluation and, secondly, to date no open dataset
for the diachronic semantic change has been made
available. This last issue has roots in the
difficulties of building a gold-standard for detecting the
semantic change of terms in a specific corpus or
language. The result is a fragmented set of data
and evaluation protocols, since each work in this
area has used different evaluation datasets or
metrics. This phenomenon can be gauged from
The rest of the article is organised as follows: Section 2 describes how our dataset is built, while Section 3 provides details about the approaches under analysis and the evaluation. Finally, Section 4 closes the paper and provides possible future work. 2
The main goal of the dataset is to provide for each lemma a set of years which indicate a semantic change for that lemma. Some dictionaries provide historical information about meanings, for example the year in which each meaning is observed for the first time. The main problem is that generally these dictionaries are not digitally available or they are in a format that is not machine readable.
Regarding the Italian language, the dictionary “Sabatini Coletti”1 is available on-line. It provides for some lemmas the year in which each meaning was observed for the first time. For example, taking into account the entry for the word “imbarcata” from the dictionary, we capture its original meaning “Group of people who gather to find each other, to leave together”, and other two meanings: 1) “Acrobatic manoeuvre of an air-plane” introduced in 1929; and 2) “fall in love” introduced in 1972.
1https://dizionari.corriere.it/ dizionario_italiano/
We setup a web scraping algorithm able to extract this information from the dictionary. In particular, the extraction process is composed of several steps: 1. Downloading the list of all lemmas occurring in the online dictionary with the corresponding URL. We obtain a list of 34,504 lemmas; 2. For each lemma, extracting the section of the web page containing the definition with the list of all possible meanings. We obtain a final list of 34,446 definitions; 3. For each definition, extracting the year in which that meaning was introduced. For a given lemma, we are not able to assign the correct year to each of its meaning, but we can only extract a year associated with the lemma. This happens because the dictionary does not follow a clear template for assigning the year to each meaning. Although associating the year of change to the definition of the meaning is not useful for the purpose of our evaluation, it could help to understand the reason behind the semantic change. We plan to fix this limitation in a further release of the dataset. In the rest of the paper we call change point (CP) each pair (lemma, year); 4. Removing those change points that are expressed in the form “III sec.” (III century) because they refer to a broad period of time rather than to a specific year.
The final dataset2 contains 13,818 lemmas and 13,932 change points. The average change points for lemma is 1.0083 with a standard deviation of 0.0924. The maximum number of change points 2https://github.com/pippokill/ kronos-it for lemma is 3 and the number of lemmas with more than one change point is 113. The oldest reported change point is 1758, while the most recent one is 2003; this suggests that the dictionary is outdated and it does not contain more recent meanings.
The dataset is provided in textual format and reports for each row the lemma followed by a list of years, each one representing a change point. For example: enzima 1892 monopolistico 1972 tamponare 1886 1950 elettroforesi 1931 fuoricorso 1934
The low number of change points for lemma reflects the fact that generally, the first meaning has no information about the year it first appeared in or that its time period is expressed in the form of century. This means that all the other meanings are additional meanings introduced after the main one. However, there are some more recent words for which the first year associated with that entry corresponds to the year in which the word is observed for the first time. Unfortunately, it is not easy to automatically discern the two cases.
Finally, we report the distribution of change points over time in Figure 1. The years with a peak are 1942, 1905 and 1869 with respectively 404, 352 and 322 change points. 3
For the evaluation we adopt our dataset as
goldstandard and the Italian Google n-grams
Google n-grams provides n-grams extracted from the Google Books project. The corpus is composed of several compressed files. Each file contains tab-separated data, each line has the following format: ngram TAB year TAB match count TAB volume count NEWLINE. For example: parlare di pace e di 2005 4 4 parlare di pace e di 2006 3 3 parlare di pace e di 2007 7 7 parlare di pace e di 2008 2 2 parlare di pace e di 2009 4 4 The first line tells us that in 2005, the 5-grams “parlare di pace e di” occurred 4 times overall, in 4 distinct books.
3http://storage.googleapis.com/books/ ngrams/books/datasetsv2.html
In particular, we use the 5-grams corpus and we limit the analysis to words that occur at least in twenty 5-grams. Moreover, we lowercase words and filter out all words that do not match the following regular expression: [a-ze´e`a``ıo`u`]+. We limit our analysis to the period [1900-2012].
In order to build the context words by
using 5-grams, we adopt the technique described
in
We investigate four systems for representing words over time and then we apply a strategy for extracting change points from each technique. Finally, we evaluate the accuracy of each approach by using our dataset as gold standard. 3.1
We adopt four techniques for representing words
over time. The first strategy is based only on word
co-occurrences, the other three exploit
Distribution Semantic Models (DSM). In particular, the
techniques are:
Collocation. This approach is very simple and it
is used as baseline. The idea is to extract
for each word and each time period the set
of relevant collocations. A collocation is a
sequence of words that co-occur more often
than what would be expected by chance. We
extract the collocation by analysing the word
pairs extracted from 5-grams and score each
word pair using the Dice score:
dice(wa; wb) =
2 fab
fa + fb
(1)
where fab is the number of times that the
words wa and wb occur together and fa and
fb are respectively the number of times that
wa and wb occur in the corpus. Since the
Dice score is independent of the corpus size,
it is possible to build for each word and each
time period a list of collocations by
considering only the collocations occurring in a
specific period of time. In order to consider only
a restricted number of collocations, we take
in account only the collocations with a Dice
value above 0.0001. For each word and each
time period we obtain a list of collocations
with the associated Dice score. For
example, a portion of the list of collocations for the
word pace (peace) in the period 1980-1984 is
reported as follows:
pace guerra 0.007223173
pace giustizia 0.0068931305
pace trattati 0.0067062946
pace trattative 0.006033537
Temporal Random Indexing (TRI). TRI
Temporal Word Analogies (TWA). This
approach is able to build diachronic word
embeddings starting from independent
embedding spaces for each time period. The
output of this process is a common vector
space where word embeddings are used for
computing temporal word analogies: word
w1 at time ti is like word w2 at time tj . We
build the independent embedding spaces by
using the C implementation of word2vec
with default parameters
Procrustes (HIST). This approach aligns the
learned low-dimensional embeddings by
preserving cosine similarities across time
periods. More details are available in
All approaches are built using the same vocabulary and the same context words generated starting from the 5-grams as previously explained. 3.2
In order to track how the semantics of a word changes over time we need to build a time series. A time series is a sequence of values, one for each time period, that indicates the semantic shift of that word in the specific period. In our evaluation, we split the interval [1900-2012] in time periods of five years each.
The time series are computed in different ways according to the strategy used for representing the words. In particular, the values of each time series (wi) associated to the word wi is computed as follow:
Collocation: given two lists of collocations
related to two different periods, we compute
the cosine similarity between the two lists by
considering a list as a Bag-of-Collocations
(BoC). In this case each point k of the
series (wi) is the cosine similarity between
the BoC at time Tk 1 and the BoC at time
Tk;
TRI: we use two strategies, (point-wise and
cumulative), as proposed in
We obtain seven time series as reported in Tables 1 and 2. In particular: BoC is build on temporal collocations; T RIpoint and T RIcum are based on TRI by using respectively point-wise and cumulative approach; T W Aint and T W Auni are built using TWA on words that are common (intersection) to all the periods (T W Aint) and on the union of words (T W Auni). The same procedure is used for HIT S obtaining the two time series HISTint and HISTuni.
For finding significant change points in a time
series, we adopt the strategy proposed in
We compute the performance of each approach by using Precision, Recall and F-measure. However, assessing the correctness of the change points generated by each system is a not easy task. A change point is defined as a pair (lemma, year). In order to adopt a soft match, when we compare the change points provided by a system with respect to the change points reported in the gold standard, we take into account the absolute value of the difference between the year predicted by the system and the year provided in the gold standard.
As a first evaluation (exact match), we impose the difference between the detected year and the gold standard to be less or equal than five, which is the time period span of our corpus. As a second evaluation (soft match), we impose only that the predicted year is greater or equal than the change point in the gold standard. This is a common methodology adopted in previous work.
For a fairer evaluation, we perform the following steps:
We remove from the gold standard all the change points that are outside of the period under analysis ([1900-2012]); We remove from the gold standard all the words that are not represented in the model under evaluation. This operation is necessary because (1) the previous filtering step can exclude some words;(2) there are words that do not appear in the original corpus.
Since the gold standard contains lemmas and
not words, we perform a lemmatization of each
output by using Morph-it!
Results of Precision (P), Recall (R) and F-measure (F) are reported in Table 1. We can observe that generally we obtain a low F-measure. This is due to a large number of false positive change points detected by each system.
BoC T RIpoint T RIcum T W Aint T W Auni HISTint HISTuni BoC T RIpoint T RIcum T W Aint T W Auni HISTint HISTuni
The best approach in both evaluations is T RIcum. Considering the exact match evaluation, the difference in performance is remarkable since generally TRI has a high recall. In the soft match evaluation, T W Auni obtains the best precision, while the simple BoC method is able to achieve good results compared with more complex approaches such as T W Aint and HIST .
The results of the evaluation prove that the task of semantic change detection is very challenging; in particular, the large number of false positive drastically affects the performance.
Further analyses are necessary to understand which component affects the performance. In this preliminary evaluation, we adopt a unique approach for detecting the semantic shift. An extended benchmark is necessary for evaluating several approaches for detecting semantic change points.
The systems are built on a vocabulary that is larger than both the original dictionary and the gold standard. For that reason, we provide an additional evaluation in which we perform an ideal analysis by evaluating only lemmas that are common to the gold standard and the system output. The goal of this analysis is to measure the ability of correctly identifying change points for those lemmas that are represented in both the gold standard and the system. Results of this further evaluation are provided in Table 2
For the exact match evaluation, T RIcum obtains the best F-measure as in the first evaluation, while T W Auni achieves a very good performance in the soft match evaluation.
The plot in Figure 2 reports how the F-measure increases according to the time span that we adopt in the soft match. In particular, the X-axis reports the maximum absolute difference between the year in the gold standard and the year predicted by the system. We can observe that under 20 years T RI provide better performance than T W A, and after 60 years all the approaches reach a stable Fmeasure value. 4
In this paper, we provide details about the construction of a dataset for the evaluation of semantic change point detection algorithms. In particular, our dataset focused on the Italian language and it is built by adopting a web-scraping strategy. We provide a usage example of our dataset by evaluating several approaches for the representation of words over time. The results prove that the task of detecting semantic shift is challenging due to a large number of detected false positive. As future work, we plan to investigate further methods for building time series and detecting semantic shifts in order to improve the overall performance. Moreover, we plan to fix some issues of our extraction process in order to improve the quality of the dataset itself.
This work was supported by the ADAPT Centre for Digital Content Technology, funded under the Science Foundation Ireland (SFI) Research Centres Programme (Grant SFI 13/RC/2106) and is co-funded under the European Regional Development Fund and by the European Unions Horizon 2020 (EU2020) research and innovation programme under the Marie Skodowska-Curie grant agreement No.: EU2020 713567. The computational work has been executed on the IT resources made available by two projects, ReCaS and PRISMA, funded by MIUR under the program “PON R&C 2007-2013”.