We propose a complex network approach to the emergence of word meaning through the analysis of semantic spaces: NLP techniques able to capture an aspect of meaning based on distributional semantic theories, so that words are linked to each other if they can be substituted in the same linguistic contexts, forming clusters representing semantic elds. This approach can be used to model a mental lexicon of word similarities: a graph G = (N; L) where N are words connected by some type of semantic or associative property L. Networks extracted from a baseline neural language model are analyzed in terms of global properties: they are small world and the probability of degree distribution follows a truncated power law. Moreover, they throw in a strong degree assortativity, a peculiarity that introduces us to the problem of semantic eld identi cation. We support the idea that semantic elds can be identi ed exploiting the topological information of networks. Several community discovery methods have been tested, identifying from time to time strict semantic elds as crisp communities, linguistic contexts as overlapping communities or meaning conveyed by single words as communities produced starting from a seed-set expansion.
In this work we assume distributional semantic theories - modeled by semantic
spaces[
The aim of the work is the identi cation of semantic elds through the exploitable topological information of networks. Treating semantic elds as communities (i.e., set of nodes tightly connected to each other), we want to partition a graph using several community discovery algorithms.
Details about the state-of-art of complex semantic networks will be given in Section 2. Data preparation (i.e., how complex networks have been extracted) will be described in Section 3. Global properties of networks will be analyzed in Section 4 - in terms of degree distribution, small world properties and assortativity. Section 5 will be about the analysis of the di erent types of semantic elds that we modeled. Section 6 will introduce several futures lines of research. 2
The relationship between complex networks and language - in this case, its
semantic level - is not trivial. The rst metaphors of semantics as network of
words refer to the models of semantic memory. Nowadays, they have been using
to model a mental lexicon of word similarities, arguing how its structure may
result in a complex system[
Complex semantic networks are graphs G = (N; L) in which N are words
connected by some type of semantic property L. Clarify the type of semantic
property helps us to classify complex semantic networks from the aspect of
meaning that they are modeling. Practically, in the previous examples nodes are real
words and semantic properties represent relations like synonymy, hyponymy,
hypernymy, free associations or the arguments of verbs, while in network
extracted from semantic spaces nodes are vectors connected by an high value of
a similarity function. Literature points out how it is hard to verify if networks
extracted from semantic spaces are scale-free, although the presence of small
world phenomenon: it is argued how degree distributions could follow an
exponential distribution[
Given a training corpus C = fw1; w2; :::; wng as a sequence of tokens and a
semantic space as a vector space V in R, the basis of V is the set of the types of
the corpus, i.e., the unique occurrence of a token, resulting in a vocabulary T =
ft1; t2; :::; tng. The dimension of V is RjT j, where jT j is the length of vocabulary.
Words are rst represented as one-hot vectors v(ti), where 1 is the position of ti
in T . Then, they are embedded in a reduced dense space of dimension Rk, with
k jT j. Word2vec[
Although the complexity of the framework, it can be simply implemented
in Python with Gensim[
# kv1kkv2k
Then, vectors are represented as nodes, with Lmax as the cosine similarity
distribution, namely the number of all distances between all vectors. It is needed
a graph in which L Lmax, whereby L must contain strong semantic relations:
thus, L is chosen with -method [
Global level is analyzed in terms of degree distribution, small world properties
and assortativity by degree. Fig. 2 sums up the analyses on degree distribution
and assortativity. As regards rst one, it is applied the likelihood-ratio test
(LRtest) (also visualized in Table 1) with powerlaw library[
Power law, truncated power law, exponential and lognormal distributions have been compared. Assortativity describes if nodes, on average, connect to nodes with similar degree. Average K-Nearest-Neighbors knn(k) is used for assortativity, i.e, the average knn for each node with degree k, where knn is the average neighbors degree of each node: there is correlation between k and knn(k), thus networks are assortative. In Table 2 a general description of global network properties shows high global clustering coe cient C and short average path length hdi, thus networks are small world.
Hubs (i.e, highest degree nodes) suggest that words belong to semantic elds, e.g., in Wikipedia networks hubs are words like iperpiressia, in ammazione, ulcerativo, etc, while in Paisa' ones they are words like facciata, marmoreo, porticato, etc. If degrees represent the extent of a semantic eld, these words could belong to clusters whose lengths depend on the number of nodes tightly connected to each other:hubs suggest how there might be huge semantic elds lexically richer than others.
Starting from this simple interpretation of degrees, we can think about the lexical richness as a property of semantic elds able to be captured exploiting topological information. Together with other aspects and properties of semantic elds, this will be argument of the next section. 5
Community discovery is the task of the detection of highly connected nodes in complex network structures. In a complex semantic network meso-scale structure is formed by semantic elds.
The idea to represent a semantic elds as a community is so explained: (i) graph clusters are strict semantic elds identi ed by crisp communities in which a node belongs to one and only one community; (ii) semantic elds are represented by linguistic contexts identi ed by overlapping communities in which a node belongs to more than one community; (iii) clusters are local semantic elds conveyed by single words, identi ed by communities produced starting from a seed-set expansion.
Community discovery algorithms produce di erent types of community on
the basis of the speci c topological property they choose to detect: e.g.,
Louvain[
As regards crisp communities, Louvain and Infomap got good results. Partitions have been analyzed in terms of dimension N , number of edges L, internal edge density, average degree hki and ratio of triads: Fig. 3, on the left, shows strong correlations between N , L and hki, both in Louvain and Infomap partitions. Both algorithms divide the graphs in consistent semantic elds: Louvain captures more general semantic domains while Infomap enhances them with granular partitions, e.g., if the largest community extracted with Louvain from Wikipedia 0.65 belongs to the eld of chemistry, Infomap can break up it in the subdisciplines of chemistry (organic chemistry, biochemistry, pharmacology, etc), consistently with the dataset from which knowledge is extracted. However, limits of crisp partitions are not trivial: the approach is not word-oriented, e.g. molecola or atomo (in according to graph partitioning, they belong to the chemistry domain) can appear in one and only one community, contrary to their concrete use in more than one domain. We focused on other algorithms capable of produce overlapping communities, in particular Demon. Partitions have been analyzed in terms of overlapping distribution, showed in Fig. 3, in the center. In some cases, the idea of linguistic context as semantic eld can be captured. The term Siria is one of the nodes with the largest number of communities in a Demon partition with = 0:5. A human analysis can easily interprets the communities in which the term is present as two di erent linguistic context: in the rst one its semantics concerns with an historical geographical entity, in the second one with a modern geographical entity, e.g., a community is composed by terms like anatolia, babilonese, egitto, eufrate, mesopotamia, persia, sumero, another one by terms like arabia, armenia, egitto, iran, iraq, marocco, turchia. However, even this approach is not totally word-oriented.
In order to focusing on the semantics of single words, communities
identied by a seed-set expansion are preferred. The proposed method chains Lemon
and Label Propagation algorithms: starting from the seed-centered communities
extracted by Lemon for each node, then Label Propagation is applied to break
them into smaller and denser word sets. Advantages of this approach focus on
the possibility to capture polysemy, i.e., multiple meaning expressed by words.
We take account of two terms, stima as example of polysemy and pesca as
example of homonymy. As regards the rst term, the partition could be coherent
with its polysemic nature, ready to be interpreted both as the price or value of a
possession (words in communities are miliardo, dollaro, milione, sterlina, euro)
and as an approximate measurement (words in communities are grossomodo,
pressapoco, incirca), but it is surely mismatched a potential third community
composed by words denoting the meaning of stima as an appreciation to others.
As regards the second term, only words related to the sense of the activity of
shing are nd, without words related to the sense of the fruit. Then, limits of
this approach may be related to the corpora and to the language model
themselves, i.e. to the missing textual information and to the bias of word2vec models
to create a unique embedding for homonymic words. Moreover, it was performed
a Ground Truth Testing to compare and evaluate quality of communities
produced in this third approach. Ground Truth Partitions have been extracted from
Wikipedia itself, using the disambiguation pages: each hyperlink is a community
composed by terms present in the hyperlinked page whose frequency was higher
than 1. Filtered networks were compared with NF1 measure, the normalized
harmonic mean of Precision and Recall[
Word meaning has been modeled as a complex system self-organized in semantic elds. The notion of word meaning was strictly related to word vectors, i.e., computational representations of meaning. Global properties of networks extracted from word2vec models were consistent with the previous literature.Assortativity might be trace of a context-based representation of word meaning: hubs belongs to few giant semantic elds because they can be substituted in the same wide contexts. A question for future researches could be about interpretations of that: does the assortativity depend on a biased distribution of corpora or on the used language model? Or it reveal real hidden structures of semantic elds in texts? Semantic elds discovery was treated as a task of graph clustering instead of word vectors clustering, namely an alternative approach aiming to model several de nitions of semantic eld with several community discovery methodologies. We have obtained good results only exploiting the topological information. However, several lines of future research can be followed. Future approach could focus on community discovery methods able to integrate network topology and external information about nodes (e.g., attributes on their frequencies in texts or on the age in which they are learned or on their categories in other levels of language analysis) to better represent a community structure in a complex semantic network.
This work is partially supported by the European Community's H2020 Program under the funding scheme \INFRAIA-1-2014-2015: Research Infrastructures" grant agreement 654024, http://www.sobigdata.eu, \SoBigData".