This work introduces reference vectors (Ref-Vectors), a new kind of word vectors in which the semantics is determined by the property of words to refer to world entities (i.e. objects or events), rather than by contextual information retrieved in a corpus. Ref-Vectors are here compared with state-of-the-art word embeddings in a verb semantic similarity task. The SimVerb-3500 dataset has been used as a benchmark to verify the presence of a statistical correlation between the semantic similarity derived by human judgments and those measured with RefVectors and verb embeddings. Results show that Ref-Vector similarities are closer to human judgments, proving that, within the action domain, these vectors capture verb semantics better than word embeddings.
Natural Language Understanding is a key point in Arti cial Intelligence and Robotics, especially when it deals with human-machine interaction, such as instruction given to arti cial agents or active learning from observation of human actions and activities. In these scenarios, enabling arti cial agents to interpret the semantics and, most of all, the pragmatics behind human speech acts is of paramount importance. Moreover, the verb class is crucial for language processing and understanding, given that it is around verbs that the human language builds sentences.
As an example, consider two of the possible interpretation of a sentence such as John pushes the glass : (1) apply a continuous and controlled force to move an object from position A to position B; (2) shove an object away from its position. If we are in a kitchen, in front of a table set with cutlery, plates and glasses, no human will interpret the sentence with (2). This naive example shows how much important is contextual understanding of action verbs.
In this contribution we present a novel semantic representation of action verbs
through what we will call reference vectors (Ref-vectors ). Ref-vectors encode
the verb possibility to refer to di erent types of action. The reference values are
extracted from the IMAGACT ontology of action, a multilingual and multimodal
ontology built on human competence (sec.3.1). We compared the resulting verb
embeddings with other popular word embeddings (namely, word2vec, fastText,
GloVe) on the task of word similarity, and benchmarked the results against
simVerb-3500 human similarity judgments [
Vector Space Models, also known as Distributional Semantic Models or Word
Space Models [
Word embeddings (also known as neural embeddings ) can be considered the
evolution of classical vector models. They are still based on the distributional
hypothesis (the semantic of each word is strictly related to word occurrences in
a corpus), but are built by means of a neural network that learn to predict words
in contexts (e.g. [
In this work we also use non-contextual information, but without combining
it to traditional word embeddings. Our space is not a co-occurrence matrix but
rather a co-referentiality matrix: it encodes the ability of two or more verbs to
refer to the same action concepts, i.e. local equivalence [
Action concepts in IMAGACT are instantiated as videos depicting actions.
The combination of linguistic and visual features to perform a more accurate
classi cation of actions has been widely used in recent years [
We evaluated the performance of the di erent representation models by
means of a verb similarity task, using the SimVerb-3500 dataset [
IMAGACT1 [
The resource consists in a ne-grained categorization of action concepts, each
represented by one or more visual prototypes in the form of recorded videos and
3D animations. IMAGACT currently contains 1,010 scenes which encompass the
action concepts most commonly referred to in everyday language usage. Action
concepts have been gathered through an information bootstrapping from Italian
and English spontaneous spoken corpora, and the occurrences of verbs
referring to physical actions have been manually annotated [
The database continuously evolves and at present contains 12 fully-mapped
languages and 17 which are underway. The insertion of new languages is obtained
through competence-based extension (CBE) [
The visual representations convey action information in a cross-linguistic environment, o ering the possibility to model a conceptualization avoiding bias from monolingual-centric approaches. 3.2
The IMAGACT database contains 9189 verbs from 12 languages (Table 1), which have been manually assigned by native speakers to 1010 scenes. It is important to notice that the task has been performed on the whole set of 1,010 scenes for each language and the di erences between the number of verbs depend on linguistic factors: some examples of verb-rich languages are (a) Polish and Serbian, in which perfective and imperfective forms are lemmatized as di erent dictionary entries, (b) German, that have particle verb compositionality, (c) Spanish and Portuguese, for which verbs belong to both American and European varieties.
Judgments of applicability of a verb to a video scene rely on the semantic
competence of annotators. An evaluation of CBE assignments has been made
for Arabic and Greek [
From the IMAGACT database, we derived our dataset as a binary matrix C9189 1010 with one row per verb (in each language) and one column per video prototype. Matrix values are the assignments of verbs to videos: Ci;j = (1 if verb i refers to action j
0 else In this way, the matrix C encodes referential properties of verbs.
In order to provide an exploitable vector representation, an approximated matrix C0 has been created from C, by using Singular Value Decomposition (SVD) for dimensionality reduction.
Language Verbs Arabic (Syria) 571 Danish 646 English 662 German 990 Greek 638 Hindi 470 Italian 646 Japanese 736 Polish 1,193 Portuguese 805 Serbian 1,096 Spanish 736
TOTAL 9189 Table 1. Number of verbs per language
SVD is a widely used technique in distributional semantics to reduce the feature space. The application of SVD to our dataset allowed us to remove the matrix sparsity and to obtain a xed-size feature space, that is independent of the number of action videos.
Finally, the output C0 is a dense matrix 9189 300. 4
We compared our co-referentiality vectors to state-of-the-art word embeddings in a verb semantic similarity task. For the evaluation, we considered the full set of 220 English verbs that are shared by the IMAGACT and the SimVerb-3500 dataset. We sampled the comparison dataset (Comp-DS) by taking those pairs of verbs for which similarity scores were present in SimVerb-3500, obtaining 624 verb pairs. Data are reported in Table 3.
Verb semantic similarity has been automatically estimated for each verb pair in the Comp-DS by computing the cosine similarity between the related Refvectors. Then, the correlation between automatic and human judgments about verb pair similarity has been determined through the Spearman's rank correlation coe cient2. The result is a positive correlation of 0.212 (Table 5). This number highlights a low correlation, but it is not informative without a comparison with other semantic vectors. 2 We applied the Spearman's rank, because data are non-parametric: Shapiro-Wilk normality test reports W = 0:9578 and p = 1:984e 12.
To this aim, we considered 6 state-of-the-art word embedding, created with
3 algorithms - word2vec3 [
Algorithm Corpus Lemmas Window Dimensions Word2Vec English Wikipedia 2017 296,630 5 300 Word2Vec Gigaword 5th Ed. 261,794 5 300 FastText Skipgram English Wikipedia 2017 273,930 5 300 FastText Skipgram Gigaword 5th Ed. 262,269 5 300 Global Vectors English Wikipedia 2017 273,930 5 300 Global Vectors Gigaword 5th Ed. 262,269 5 300 Table 4. Numbers of the lemmatized word embeddings used for comparison.
The previous procedure has been repeated by using these embeddings instead of Ref-vectors: cosine similarity has been measured between each pair of the Comp-DS, by using di erent embeddings. The Spearman's rank correlation coe cient with the Comp-DS is reported in Table 5. Data show that Ref-vectors are closer to human judgments in estimating verb semantic similarity (0.212), with the exception of word2vec trained on the GigaW corpus that obtained almost the same degree of correlation (0.211). All the other verb embeddings considered report a lower correlation with Comp-DS.
The same analysis has been conducted considering semantic classes. SimVerb3500, and thus Comp-DS, contains the annotation of the semantic relation between the two verbs: the pairs can be synonyms, hyper-hyponyms, cohyponyms, antonyms or not related. We used this information to measure the correlation of vector similarity with Comp-DS in verb pairs for each semantic relation. Table 6
4 https://fasttext.cc 5 https://nlp.stanford.edu/projects/glove/ 6 https://archive.org/details/enwiki-20170920 7 https://catalog.ldc.upenn.edu/LDC2011T07 shows that Ref-vectors have a stronger correlation (0.26 and 0.35) with human judgments with two classes of related pairs, i.e. hyper-hyponyms and synonyms. Their results are rather poor instead with antonyms and for non semantically related pairs. This suggests that Ref-vectors are better at capturing semantic similarity rather than semantic relatedness. Antonyms, indeed, cannot be considered as semantically similar, since they have opposite meanings. Moreover, if we do not consider pairs that are not semantically related, the general results of Ref-vectors outperform those of the other systems (Table 7). In this paper we have introduced a novel model to represent action verbs semantics based on their referential properties, rather than standard corpus cooccurrences. We compared our model to state-of-the-art word embeddings systems in a verb semantic similarity task. We have shown that our referential vectors correlate better to human judgments from the SimVerb-3500 dataset in presence of speci c types of semantic relations. These results suggest that di erent types of embeddings capture di erent types of relations, like semantic similarity or simple relatedness. They bring new interesting directions into semantic modeling, a eld in which research has focused in the last years mainly on corpus co-occurrences data. We believe that merging referential and corpus data into semantic modeling can improve language processing and understanding, and foster Natural Language Understanding in Arti cial Intelligence towards a more contextually informed dimension.