English. In this paper, we propose several methods for the diachronic analysis of the Italian language. We build several models by exploiting Temporal Random Indexing and the Google Ngram dataset for the Italian language. Each proposed method is evaluated on the ability to automatically identify meaning shift over time. To this end, we introduce a new dataset built by looking at the etymological information reported in some dictionaries.
by the need to reflect the continuous changes of the world. The evolution of word meanings has been studied for several centuries, but this kind of investigation has been limited by the low amount of data on which to perform the analysis. Moreover, in order to reveal structural changes in word meanings, this analysis has to explore long periods of time.
Nowadays, the large amount of digital content
opens new perspectives for the diachronic analysis
of language. This large amount of data needs
efficient computational approaches. In this scenario,
Distributional Semantic Models (DSMs) represent
a promising solution. DSMs are able to
represent words as points in a geometric space,
generally called WordSpace
Since its first release, the Google Ngram dataset
In this paper, we propose an evolution of our
previous work
The paper is structured as follows: Section 2 provides details about our methodology, while Section 3 describes the dataset that we have developed and the results of a preliminary evaluation. Section 4 reports final remarks and future work. 2
Our method has its roots in a previous model based
on Temporal Random Indexing (TRI)
The idea behind TRI is to build different WordSpaces for each time period that we want to analyse. The peculiarity of TRI is that word vectors over different time periods are directly comparable because they are built using the same random vectors. In particular TRI works as follows: 1. Given a corpus C of documents and a vocabulary V of terms1 extracted from C, the method assigns a random vector ri to each term ti 2 V . A random vector is a vector that has values only in f-1, 0, 1g and it is sparse with few non-zero elements distributed randomly along its dimensions. The set of random vectors assigned to all terms in V are near-orthogonal; 2. The corpus C is split in different time periods Tk using temporal information, for example the year of publication; 3. For each period Tk, a WordSpace W Sk is built. All the terms of V occurring in Tk are represented by a semantic vector. The semantic vector svik for the i-th term in Tk is built as the sum of all the random vectors of the terms co-occurring with ti in Tk. When computing the sum, we weigh the random vector; in this case we adopt a formula based on inverse document frequency. Formally, the weight is computed as w(ri) = log #Ctkik , where Ck is the total number of occurrences in Tk and #tik is the occurrences of the term ti in Tk. The idea is to give less weight to the most frequent words.
In this way, the semantic vectors across all time periods are comparable since they are the sum of the same random vectors.
RRI can be implemented by repeating the steps 2 and 3 several times. Where at each iteration random vectors are replaced by the semantic vectors built in the previous step. The idea is to model implicit connections between terms that never cooccur together, but that could occur frequently with other shared terms.
The next two sub-sections provide details about the Google Ngram dataset and the method used to automatically detect word meanings shift. 2.1
Google Ngram is a very large dataset containing all the n-grams (up to five) extracted from Google Books. It is built by analysing over five millions books spanning the years from 1500 to 2012, but the developers estimate that the most reliable period is from 1800 to 2012. The dataset covers several languages including Italian. For each 1The terms that we want to analyse. Usually, the most n frequent terms are extracted. language, several compressed files are released. Each file contains for each line the following information: Ngram <TAB> year <TAB> match count <TAB> volume count. For example, the line “analysis is often described as 1991 104 5” means that the ngram “analysis is often described as” occurs 104 times in 5 books in the 1991 .
We modify TRI for building the WordSpaces directly from the Google Ngram dataset. In particular, we need a pre-processing step in which we split the n-grams in several files according to the time periods we want to analyse. For example, if we fix the dimension of a time period to ten years from 1850 to 2012, we build several files for each period: T1 = 1850-1859; T2 = 1860-1869; : : : ; T16 = 2000-2009; T17 = 2010-2012. Each file contains only the n-grams that occur in the specific time period. We remove information about the year and the book count since they are not useful in the subsequent steps. Considering the previous example, the line “analysis is often described as 104” will be stored in the file 1990-1999.
After this pre-processing step, we can easily run TRI and RRI, where RRI can be repeated multiple times. 2.2
To track the word meaning change over time, for each term ti we build a time series (ti) taking into account several methods. A time series is a sequence of values, one value for each time period, that indicates the semantic shift of that term in the specific period. We adopt several strategies for building time series. The first strategy is based on term log-frequency; each value in the series is defined as: k(ti) = log( #Ctkik ).
In order to exploit the ability of our methods in computing vectors similarity over time periods, we define two strategies for building the time series: point-wise: k(ti) is defined as the cosine similarity between svik and svik 1. In this way, we want to capture vector changes between two time periods; cumulative: we build a cumulative vector sviCk 1 = Pjk=01 svij and compute the cosine similarity with respect to the vector svik. The idea is that the semantics at point k 1 depends on the semantic of all the previous time periods.
Given a time series we need a method for
finding significant change points in the series. We
adopt the strategy proposed in
K( ) =
1 l j
l
X k=j+1 k
j 1 X j k=1 k (1)
In order to understand if a mean shift is
statistically significant at time j, a bootstrapping
The goal of the evaluation is twofold: 1) to build a standard benchmarking for meaning shift detection for the Italian language; 2) to evaluate the performance of the proposed methods and compare them with the baseline model based on the word frequency.
A list of meaning shifts for the Italian language is not available, then we build a new dataset using a pooling strategy. In particular, we retrieve the list of meaning shifts, as explained in Section 2.2, using the cumulative strategy for each of the following methods: word frequency, TRI, TRRI with one iteration and TRRI with two iterations.
Taking into account the first 50 words for each system, we manually check for each word if a meaning shift occurs by exploiting some dictionaries. We use two dictionaries: the “Sabatino Coletti” available on-line2 and the “Dizionario Eti2http://dizionari.corriere.it/ dizionario_italiano/ mologico Zanichelli” available on CD-ROM. Finally, we obtain a gold standard that consists of 40 words and their corresponding change points.
All the methods, with exception of word frequency, are built using co-occurrences information extracted from 5-grams in the Google Ngram dataset for the Italian. The vector dimension is set to 1,000 for all the approaches based on Random Indexing using two non-zero elements in the random vector.
We adopt accuracy as evaluation metric. Given a list of n change points returned by the system, we compute the ratio between the number of change points correctly identified in the gold standard3 and n. In order to identify the correct change points, we consider not only the word4, but also the year of the change point. In particular, the year predicted by the system must be equal or greater than one of the years reported in the gold standard. We compute the accuracy using different values of n (10, 100, ALL). Results of the evaluation are reported in Table 1. In particular, we evaluate 7 systems: logf req is the baseline based on word frequency; T RI is the Temporal Random Indexing method, T RRI1 is the Temporal Reflective Random Indexing with one iteration, while T RRI2 adopts two iterations. For the methods based on Random Indexing, we investigate both the pointwise and the cumulative strategy to compute the change points.
The analysis of the results shows that T RI generally provides better results than T RRI. Moreover, the point-wise strategy always outperforms the cumulative one. With respect to the baseline, it has the same accuracy of T RI for both acc@10 3The gold standard adopted in this evaluation is available here: https://dl.dropboxusercontent.com/u/ 16026979/data/TRI_CLIC_2016_change_word.
4The word matching is performed taking into account also the inflected forms. and acc@100, while it performs worse than T RI and T RRI1 when the accuracy is computed over the whole list of terms (ALL). These results suggest that, while there are not too many differences between the two methods considering smaller lists of results, T RI is actually able to detect more meaning shifts on a larger set of terms. T RRI2 always provides the worst results; we speculate that two iterations introduce too much noise in the model. A closer scrutiny to the list of words provided by T RRI2 highlights the presence of many foreign words: a simplistic conclusion may suggest that this approach is able to identify foreign terms that are introduced in the Italian language. However, we think that the output of this method deserves more investigations carried out by designing an ad-hoc evaluation.
The evaluation is based on the predicted year, which has to be equal or greater than one of the years reported in the gold standard, we conduct a further analysis to measure how far the prediction is from the exact value. In particular, we compute the mean and the standard deviation taking into account the differences between the predicted and the exact year. The results of this analysis are reported in Table 2. We observe the both T RRI1cum and T RRI2cum produce the best results despite their low accuracy, while T RIcum reports the best trade-off between accuracy and precision in detecting the correct year. It is important to underline that the size of the time interval influences this kind of analysis since if the algorithm predicts 1900, the change point could happen in the interval 1900-19095. We plan to design a more accurate analysis by exploring a time interval set to one year as future work.
In this work we proposed several methods based on Random Indexing for the diachronic analysis of the Italian language. We built a dataset for the evaluation of meaning shift by exploiting etymological information taken from two Italian dictionaries. We compared our approaches with respect a baseline based on word frequency obtaining promising results. In particular, the TRI method showed its better capability in retrieving more meaning shifts on a longer list of terms. As future work, we plan to extend the dataset with further words and to investigate other methods based on word-embeddings.
This work is partially supported by the project “Multilingual Entity Liking” funded by the Apulia Region under the program FutureInResearch.