The meaning of a concept or a word changes over time. Such concept drift re ects the change of the social consensus as well. Studying concept drift over time is valuable for researchers who are interested in language or culture evolution. Recent word embedding technologies inspire us to automatically detect concept drift in large-scale corpora. However, comparing embeddings generated from di erent corpora is a complex task. In this paper, we propose to use a simple approach for detecting concept drift based on the change in word contexts from di erent time periods and apply it to subsequent time periods so that the detailed drift could be detected and visualised. We dive into certain words to track how the meaning of a word changes gradually over a long time span with relevant historical events which demonstrates the e ect of our method.
Concept drift or diachronic semantic shift studies how the meaning of a concept
or a word changes over time [
Recent word embedding technologies inspire us to automatically detect
concept drift in large-scale corpora [
In this paper, we propose to use a simple approach to quantify the concept drift based on their contexts generated from two time periods and apply it to subsequent time periods to study concept drift over a long period of time. We study more than 50 thousand English books published between 1800 and 1950. We rst divide the whole period into subsequent 20-year time spans. Based on word embeddings corresponding to each individual time span, we calculate the context of each word at that particular period. Secondly, we measure the concept drift by comparing the contexts of the same word from di erent periods. Looking how the context changes from the starting period to the ending period, we can easily identify the most dynamic words over the 150 years. For these words which potentially underwent drastic change in their meaning, we further measure how their contexts change in subsequent time periods and visualise the change over time. Some interesting associations to historical events are also discovered. 2
Diachronic semantic shifts has gained much attention because of the availability
of large corpora and recent success in computing distributional semantics [
To study semantic shift, it is therefore natural to rst construct word
embeddings in separated time periods before comparing these embeddings across
time. However, the similarity between the word embeddings generated from
separate time-speci c corpora cannot be computed directly because the stochastic
embedding algorithms could only roughly guarantee the stability of the pairwise
similarities between words but the numerical embeddings are often rotated after
each training, i.e. invariant under rotation. An earlier proposal was to
incrementally update diachronic embedding models [
Another approach is to study the neighbours or the context of a word at
two di erent time periods to measure semantic shift. Azarbonyad et. al. in [
More recently, researchers also proposed dynamic word embeddings models to
jointly learn word embeddings across di erent times periods [
Here, we rst describe the method which measures the changes in word meaning based on its contexts at two di erent time periods. Secondly, we divide the whole corpus into subsets corresponding to consequent time periods and study how the meaning of a word change over time. 3.1
Inspired by the method proposed in [
J (A; B) = j A \ B j = j A [ B j
j A \ B j j A j + j B j j A \ B j (1) A high Jaccard similarity coe cient suggests that the context of the word has not changed much, while a low value suggests that the meaning of the word might have changed over time. 3.2
Given a corpus which spans a long period of time, it is then possible to study how individual words change over time. We divide the whole corpus into multiple subsets corresponding to subsequent time periods. We measure the drift of words from the beginning period to the last. This way, we could detect the most dynamic and static words over the long period of time. It is also possible to look more carefully when a word has undergone a critical moment when its meaning changed dramatically. 4
We download the full text content of 51,625 English books from Project Gutenberg.1 Unfortunately, the exact years of publication for these books are missing. However, the birth and death years of the author are available in the data. We therefore took the average of the birth and death year of the author as the approximate year of publication for the book.
After grouping books by their year of publication, we nd that, although the earliest books were written before 500 BC, the number of books published
earlier than the 18th century is far less than that from the 18th century to the mid 20th century. Because of the copyright restriction, books in recent years are also limited. In this study we focus on the books published from 1800 to 1950. We further divide the corpus into consequent groups of 20-year time spans, as hown in Table 1.
For each time period show in Table 1, we trained a word2vec model using the
gensim library2 using the full text content of the books published within that
period of time. We chose the continuous Bag-of-Words(CBOW) model [
After the context of a word is calculated for each time period, we can now measure how much this word has drifted from one time period to another. Sensibility of parameter K The model is generally stable. We examine how k parameter, which de nes the length of similar words lists, a ect the stability. We change k as [20,30,60,100], and the curves highly coincide with each other in the whole plot for Fig.2, so in most part of these plot, we regard the same
distribution as proof of the stability of our model. In the following experiments, we take k = 20.
We calculate the Jaccard similarity between the context at 1800 and that at 1950. The distribution of the Jaccard similarity is shown in Fig. 3. As the distribution shows, very few words have stable context throughout these 150 years. Words such as sister, daughter, mother, wife and husband are rather stable word, while other words such as witch, foster, hive and potion have changed dramatically.
Many words have completely changed their contexts. However, these words are mostly infrequent words, such as chestnut, hive, vantage, and co n. Their embeddings and consequent contexts are over sensitive to the corpus. We could not make solid conclusions in terms of the drift of their meaning.
This still helps us to identify interesting cases of concept drift among the words that have a low context similarity. Once identi ed, we can dive into the more granule time periods and inspect the drift more closely. For example, Fig. 4 shows the drift of word peace over time. The Jaccard similarities between the current context and that of the previous period are plotted. The sharp decrease from 1890 to 1930 suggests that there was a drastic change of the meaning of the word peace in that period. 5.3
The change in Jaccard similarity as shown in Fig. 4 only provides a signal of drift, but not the content of drift. To dive deeper into what exactly happened for speci c words in speci c periods, we present a visualization method that helps to see how words have changed.
Our visualization is inspired by the work of Wijaya and Yeniterzi[
In Fig. 5, the word bishop at 1910 is mostly associated with religious words, such as prelate, church, cathedral, and dioce while at 1950, it is more related to the chess game, as its context includes words like checkmate, pawn, and knight. It is worth mentioning that ktxb, bxkt, and rxkt are notation words in chess but not meaningless words.
Compared to the popular t-SNE visualization [
Fig.6 shows the sequential change of the word peace from 1800 to 1950. Fig.6 (b) shows that war is in the same context with peace at 1800 and 1910. It suggests that peace and war were often mentioned together .
However, the link between war and peace disappeared in 1950. New relationships emerged, such as spirit, humanity, tyranny, and poverty. Link to the statistics of war. There were high-intensity con icts around the 1800s (Napoleonic Wars, etc), 1860s (American Civil War, etc.), and 1910s (World War I). There were few wars after World War II (1945). It makes sense that people transferred their (a) The word peace at 1800 and 1850 (b) The word peace at 1800 and 1910 (c)The word peace at 1800 and 1950
Fig. 6. Concept Drift of the word peace over time concerns about peace into other topics like spirit, humanity, tyranny, and poverty in the 1950s.
Because of the copyright restriction, Gutenberg data set only has few books after 1970. It limits us to apply our method to recently published books. A potential future work for this work is to apply this approach on possible contemporary book corpus. There would be more interesting ndings closer to our life. 6
conclusion Concept drift re ects the change of the social consensus. Detecting word usage in di erent periods is an important research method. We propose a computational approach to discover the drastic concept drifts by their context in the historical English books over centuries. It quanti es the extend of concept drift and makes the rank of drastic change possible. We also present a new way to compare the concept of a word in di erent periods. We show that our visualization is simple and intuitive. It also has the unique advantage of demonstrating the gradual change of concept overtime.