Statistical Natural Language Processing (NLP) techniques allow to quantify lexical semantic change using large text corpora. Wordlevel results of these methods can be hard to analyse in the context of sets of semantically or linguistically related words. On the other hand, structured knowledge sources represent such relationships explicitly, but ignore the problem of semantic change. We aim to address these limitations by combining the statistical and symbolic approach: we enrich WordNet, a structured lexical database, with quantitative lexical change scores provided by HistWords, a dataset produced by distributional NLP methods. We publish the result as Linked Open Data and demonstrate how queries on the combined dataset can provide new insights.
How words have been used in discourse over time, have adopted new senses or
changed their meaning is studied in the humanities and social sciences (e.g.,
[1{3]) and information sciences (e.g., [
Semantic change in words is increasingly modelled using distributional NLP methods (word embeddings). These techniques represent the meaning of a word in terms of its tendency to co-occur with other words in the lexicon, as observed in large text corpora. Since this results in vectors, cosine distances can be used to quantify the correspondence between two such representations. When vectors are assembled for the lexicon in separate time spans, the notion of distance can be applied to nd a word's nearest neighbours within a time frame, or to calculate the degree of change a word underwent from one time interval to the next.
However, word embeddings alone are not su cient to gain insight into the dynamics of the lexicon. They operate on the level of individual terms, often without metadata, making it hard to see patterns and connections. It is thinkable, though, that language change a ects not just individual terms but also clusters of (related) terms, that show interaction in their motions of semantic drift. Also, some types of words might change more than others. Structured knowledge sources can help derive such insights. For instance, lexical resources allow to group and connect ndings for individual terms by their relation.
At the same time, structured knowledge bases (KBs) bene t from enrichment
with semantic change information derived from word embeddings. The largest
diverse collection of open knowledge, the Linked Open Data cloud, contains billions
of facts about entities and their relations. However, shifts in meaning of these
entities are not explicitly encoded, and thus unavailable to these applications. For
instance in digital library applications, this could cause problems when mapping
contemporary user queries to the metadata vocabulary of archived documents.
Khan et al.[
This paper is a step towards the goal of a structured, interconnected knowledge source of diachronic lexical semantics. It presents an interlinking e ort between HistWords, a unique corpus of (open) lexical change data, and WordNet, a lexical database that is part of the Linked Open Data cloud. This combination results in a knowledge graph were concepts, linguistic data elements such as lexemes, and semantic change scores can be queried together. By publishing the data in RDF, we aim to contribute to the (re-)usability of these open corpora.
In the remainder of this paper, we discuss how we linked the HistWords data to lexical entries in WordNet and how the result was represented in an RDF data model. With example queries on this aggregated dataset we demonstrate the use as well as the limitations of the approach. 2
HistWords. HistWords is a research project of `Word embeddings for Historical
Text' at Stanford University that has produced sets of word embeddings and
cross-decade lexical change scores. We used all ready-made lexical change scores
for English1, i.e., for the 10.000 most frequent, non-proper-noun words from the
English Google N-Grams dataset2. The entries in HistWords are not lemmatised,
disambiguated or part-of-speech tagged, hence each similarity score re ects all
senses and grammatical functions in which the word can occur. The linking e ort
to WordNet, which does distinguish between di erent parts of speech, does not
solve this issue, but does make it more explicit. The similarity scores are given
between discreet decades. They were calculated as the cosine similarity between
the vector for a term derived from corpus material in one decade, and the vector
for the same term derived from materials from the other decade. Training was
done by the ("word2vec") skip-gram method with negative sampling [
Figures are available for every two consecutive decades between 1810 and 1990; i.e., the degree of semantic stability of a lexical term from the 1810s to the 1820s, the 1820s to 1830s, and so on, up to 1980s-1990s. As an example, the word gay seems to have underwent semantic change between the 1980s and 1990s, where the cosine similarity between the two term representations fell to 0.91 (from 0.96 for the 1970s-1980s). In addition, there are gures for every 1 http://snap.stanford.edu/historical_embeddings/eng-all_sgns.zip, fullstats 2 http://storage.googleapis.com/books/ngrams/books/datasetsv2.html decade vs. the 1990s, i.e., for 1810s vs.1990s until 1980s vs. 1990s. These can be used to express the overall change of a lexeme in, for instance, the 20th century (1900s-1990s), or over the entire dataset (1810s-1990s). Due to corpus characteristics, some entries have (some) missing values, which were left out. wn:Synset wn:entail, wn:hyponym, ...
wn:Synset wn:synset member
wn:gloss wn:lexical domain
WordNet. WordNet [
The RDF conversion of WordNet [
The sourced similarity scores were transformed into change data and connected to WordNet through (stemming and) string matching. The result was represented in RDF and OWL and made available as a Turtle download3.
Deriving semantic change scores. The scores were converted to distance measures as we care about the degree of change rather than the degree of stability of the words' meaning. This was done with an arc-cosine transformation rather than by the formula 1 cosine similarity to stretch the scale of the change interval and trace more ne-grained di erences. The semantic change rate thus lies between 0 and =2 (in our dataset, between 0.09 and 1.48). For instance, between the 1980s and 1990s the change values ranged from 0.11 (pepper ) to 1.12 (web). The rates for a larger period are generally higher than those for consecutive decades, e.g. 0.97 for gang between the 1810s and 1990s. The change scores have no clear absolute meaning but can be used contrastively between terms or time frames.
Linking HistWords to WordNet. The words in HistWords were mapped onto lemon:LexicalEntry instances in RDF-WordNet. First, we merged on an exact match between a word in HistWords and the value of the lemon:writtenRep property of the lemon:Form corresponding to the lemon:LexicalEntry instance. Since the HistWord words are not part-of-speech speci c, they were mapped onto all lexical matches in WordNet, irrespective of grammatical type. This step resulted in 7.365 matches for the 10.000 source words.
Second, unmapped HistWords entries were Porter stemmed and re-matched based on an exact match of the stem and a WordNet entry. We included the matches as new lemon:lexicalEntry instances with their unstemmed form as the canonical form, and connected them to their WordNet lemon:lexicalEntry counterparts through the lemon:lexicalVariant property. This brought the total number of mappings to 8.878 out of 10.000 source entries, connected to 12.469 lemon:LexicalEntry instances. In future work, it is likely that more words can be matched by re ning our stem-and-match technique. Data model. The resulting data, i.e., the tuples flexical entry, decade1, decade2, change valueg, were represented in RDF. Existing vocabularies were used where possible; newly introduced classes and properties are recognisable by the cwi-sc pre x. Figure 2 illustrates how a lemon:LexicalEntry is connected to a (blank) node of type cwi-sc:SemanticChange for each data tuple with a value and an onset and o set decade. The latter two were modelled, in accordance with OWL-Time4, as intervals with a start and an end date.
Following OWL-Time ensures interoperability and supports temporal reasoning, but complicates queries for the semantic change of a word between two speci ed decades. For this reason we introduced a shortcut property for each set of decades, which directly connects a lemon:LexicalEntry instance to the semantic change value. The property URI encodes the decades it contrasts, e.g., cwi-sc:semantic change 1910s-1920s leads to the change score between the 1910s and the 1920s.
Note that instead of at the lemon:LexicalEntry level, we could have linked the HistWords entries to the lemon:Form level, representing the lexeme. We
decided against this since it would greatly complicate the queries that we anticipate at the LexicalEntry or Synset level, while yielding only 334 mappings to in ectional variants, part of which were now matched in the second mapping step.
lemon:LexicalEntry cwi-sc:semantic change cwi-sc:SemanticChange cwi-sc:semantic change [decade1]-[decade2] cwi-sc:onset reference
cwi-sc:o set referencredfs:value ot:TemporalEntity ot:TemporalEntity xsd: oat ot:hasBeginning ot:hasEnd ot:hasBeginning
ot:hasEnd
ot:Instant
ot:Instant
ot:Instant
ot:Instant
We used the semantic web server ClioPatria [
Figure 3 summarises the results and shows the spread of the change scores grouped by the parts of speech distinguished in WordNet. It shows that the change rates are evenly distributed over the grammatical categories. Looking at the distribution over parts of speech of the word entries themselves (Table 1), though, we see that our dataset contains relatively many verbs and adjectives and few nouns as compared to WordNet.
Table 2 shows examples of semantic domains and the mean change score of
their lexical entries. Words referring to processes, phenomena and events have
seen a higher degree of change than words for food, feelings, or the weather.
Example 2: the relationship between polysemy and semantic change.
The synset structure of WordNet provides a simple way to quantify the degree
of polysemy of a word. Hamilton et al.[
We plot the change score for 1810s-1990s of each word form (again, as a
proxy for the overall change, as do [
Synsets and change rates by term
Synsets and change rates by lexical entry ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1 ● ● ● ● ● ● ● ● ●
● ● ● ●
●● ● ● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●
● ● ● ● ●● ● ● ● ● ● ● ●● ●● ●● ● ● ●● ● ● ●●●●● ●●●●● ●●●●● ●●●● ●●● ●●● ● ●● ●●● ● ● ● ●● ●●●
● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●●●●● ●●●●●●●● ●●●●●●●● ●● ●●● ●●●● ●● ●●●● ●● ●●●● ●●●● ●●● ●● ● ● ● ● ● ● ● ●●●●●● ●●●●●● ●●●●●● ●●● ●●● ●● ●●● ●● ●●●● ● ● ● ● ● ● ● ● ● ● ● ●
● 1 2
3 Number of synsets, log−scaled ● ● ● ● ● ● ● ● 4 ● integrated to bene t from a single source.
In future work, we intend to enrich the dataset in several directions. Envisaged are a cross-lingual dictionary such as Babelnet, to see if other languages display parallels in their lexical patterns of change, and a frame-semantic source like Framenet as an alternative ground for grouping term-level ndings. Another addition we aspire is a score set based on part-of-speech-tagged word vectors. 6
This work was partially supported by H2020 project VRE4EIC under grant agreement No 676247.