English. Herein we present a study dealing with the linking of two multilingual and multimedia resources, BabelNet and IMAGACT, which seeks to connect videos contained in the IMAGACT Ontology of Actions with related action concepts in BabelNet. The linking is based on a machine learning algorithm that exploits the lexical information of the two resources. The algorithm has been firstly trained and tested on a manually annotated dataset and then it was run on all the data, allowing to connect 773 IMAGACT action videos with 517 BabelNet synsets. This linkage aims to enrich BabelNet verbal entries with a visual representations and to connect the IMAGACT ontology to the huge BabelNet semantic network.
Italiano. In questo articolo si presenta
uno studio sul linking tra due risorse
linguistiche multilingui e multimediali,
BabelNet e IMAGACT. L’esperimento
ha l’obiettivo di collegare i video
dell’ontologia dell’azione IMAGACT con
i concetti azionali contenuti in BabelNet.
Il collegamento e` realizzato attraverso
un algoritmo di Machine Learning che
sfrutta l’informazione lessicale delle due
risorse. L’algoritmo e` stato addestrato e
valutato su un dataset annotato
manualmente e poi eseguito sull’insieme totale
dei dati, permettendo di collegare 773
video di IMAGACT con 517 synset di
BabelNet. Questo linking ha lo scopo di
arricchire le entrate verbali di BabelNet
con una rappresentazione visuale e di
collegare IMAGACT alla rete semantica
di BabelNet.
1 Introduction1
Ontologies are widely used to represent language
resources on the web, allowing them to be
easily accessed and exploited by machines. For this
reason, data interconnection between different
semantic resources is a crucial task in order to
enhance disambiguation and information retrieval
capabilities in Artificial Intelligence, as evidenced
by the increasing research into mapping and
linking techniques among ontologies
Aside the general interest for knowledge bases interconnection in a web-based perspective, there is also a growing interest in multimodal resources, which combine textual and visual data. These resources can be exploited by intelligent algorithms integrating vision and natural language processing techinques2. This integrated approach was successfully applied for some challenging tasks involving verbs and their action reference as a video. Regneri et al. (2013) developed machine learning models for the automatic identification of similarity among actions, by using a corpus of natural language descriptions, derived from the videos of the MPII Cooking Composite Activities dataset, which represents actions involved in basic cooking tasks. Instead, the algorithm developed by Mathe 1Lorenzo Gregori developed the linking algorithm and wrote sections 3, 4, and 5; Andrea Amelio Ravelli performed the data annotation and wrote sections 1 and 2; Alessandro Panunzi supervised the research work and revised the paper.
2Several works in this field have been developed within The European Network on Integrating Vision and Language (iV&L Net), http://ivl-net.eu/ et al. (2008) extracts higher level semantic features in common among a sample set of verbs, using a fine-grained analysis of the represented action concepts, intended as a subsequent stable set of abstract features of the objects involved in the videos. Within this interdisciplinary perspective, a knowledge base which relates verbal lemmas in different languages with video prototypes can help in serveral applications, and be exploited by both humans and machines. 2
This paper presents a linking between BabelNet
BabelNet3 is a multilingual semantic network created from the mapping together of the WordNet thesaurus and the Wikipedia enciclopedia. At present, BabelNet 3.7 contains 271 languages and it is the widest multilingual resources available for semantic disambiguation. Concepts and entities are represented by BabelSynsets (BS), extensions of WordNet synsets: a BS is a unitary concept identified by several kinds of informations (semantic features, glosses, usage examples, etc.) and related to lemmas (in any language) which have a sense matching with that concept. BSs are not isolated, but connected together by semantic relations. Moreover, BabelNet received a large contributions from its mapping with other resources such as ImageNet, GeoNames, OmegaWiki (along with many others), which increased its information beyond the lexicon and produced a wide-ranging, multimedia knowledge base.
IMAGACT4 is a visual ontology of action that
provides a video-based translation and
disambiguation framework for general verbs. The
database evolves continously
Other attempts have previously been made to link IMAGACT with other resources. Two experiments by De Felice et al. (2014) and by Bartolini et al. (2014) were conducted in an intralinguistic perspective: their aim was to evaluate the results of a mapping between the action concepts defined in ItalWordNet and the ones categorized by IMAGACT (in terms of perfect matches or hypernym/hyponym relations).
On the contrary, the objective behind our work is to obtain a light link between the resources by enriching the action concepts in BabelNet with a visual representation; in this way, we overpass the problem of finding a match between the generic semantic concepts in BabelNet and the specific pragmatic concepts in IMAGACT. This methodology is also enforced by the multilingual frame in which the experiment is conducted. As a matter of fact, the relation between words and concepts can deeply differ across languages, while the prototypical scenes ensure a language-independent modality which is able to keep together the different lexicalizations of the action space.
This work is a further step from a previous
IMAGACT-BabelNet linking experiment
The aim of this experiment is to link the IMAGACT video scenes to the BabelNet interlinguistic concepts (BabelSynsets). In fact, the BabelNet objects are already enriched with visual objects, though this information contains static images which are inadequate for representing action concepts. In this way, adding video scenes to the verbs is very desirable and would suggest itself as a natural extension of BabelNet. 4.1
A manually annotated dataset of 50 scenes and 57 BabelSynsets (2,850 judgments) was created in order to test the algorithm and evaluate the results.
The sampling was carried on in two steps. First of all, a purely actional semantic area has been selected by taking BSs and scenes linked to 7 English action verbs, which are general and very frequent in the language use: put, move, take, insert, press, give and strike. The wide variation of these verbs allowed us to obtain a big set of concepts, with a high variation in terms of frequency and generality. On this set, a second sampling has been performed by preserving the variability in terms of number of connected verbs, that is a measurable parameter in both the resources.
Each hBS,Scenei pair has been evaluated to check if the scene is appropriate in representing the BS. Three annotators compiled the binary judgment table and we reported the values shared by at least 2 of 3. The measured Fleiss’ kappa inter-rater agreement for this task was 0.74 6.
Finally, the dataset has been split in a training set and a test set, with the proportions of 80% and 20% respectively (10 randomly chosen scenes for the test set and the remaining 40 scenes for the training set). 4.2
For this task, we developed a new algorithm which uses Machine Learning techniques, by exploiting the training set. As in the previous experiment, the features are extracted from the lexical items belonging to both the candidate BabelSynset and its neighbours7. Beside the algorithm, a baseline 6The manually annotated training set is published at http://bit.ly/29J0ypx
7This test is based on BabelNet 3.6; the data was extracted
using the Java API
Table 1 reports on the 17 languages common to both BabelNet and IMAGACT, detailing the relative number of verbs in each, and constitutes the quantitative data which the matching algorithms can exploit.
English (EN) Polish (PL) Chinese (ZH) Italian (IT) Spanish (ES) Russian (RU) Portuguese (PT) Arabic (AR) German (DE) Norwegian (NO) Danish (DA) Hebrew (HE) Serbian (SR) Hindi (HI) Urdu (UR) Sanskrit (SA) Oriya (OR) Total nb: the number of verbs connected to the BS; nsb: the number of verbs that are shared between the Scene and the BS;
These 3 features have been calculated for each candidate BS and for the ones which are semantically related to it. We took the 8 BabelNet semantic relations available for verbs (see table 2) and for each BS we extracted 8 groups of related synsets, each one containing the set of BS connected to the main one by the same relation. Then, ns, nb and nsb are calculated for each group by summing the values of the BSs belonging to it. The feature set is comprised of 27 features: 3 features for the main BS and 3 features for each BabelNet relation. The set of candidates consists of all the possible BSs for each verb connected to the scene. A machine learning algorithm was trained on the annotated dataset: we used Support Vector Machine (SVM) classifier with a RBF kernel.
Table 2 shows the list of relations between the verbal BSs ranked by their relevance values for this task; this value is measured with Information Gain on the annotated dataset.
Hyponym 0.057 Hypernym 0.026 Also See 0.019 Verb Group 0.019 Gloss Related 0.009 Entailment 0.003 Antonym 0.000
Cause 0.000 The algorithm was run on the training set and evaluated on the test set; the results are reported in Table 3.
The results in terms of F-measure are not so satisfying and the value obtained with the algorithm is barely better than the baseline. Despite this, it’s important to consider the difference with the baseline in terms of precision and recall, since precision is more important for this task: for this reason, the algorithm provides a much more reliable result compared to the baseline. We also have to point out that a low recall is mainly caused by multiple possiblities in the interpretation of a scene from different points of view: for example, the scene linked to the English verb to throw described by the sentence John throws the ball to Mark can represent not only a sense of throw, but also senses of other verbs, like to play or to catch, that refer to different semantic concepts; in these cases, the scene in IMAGACT is not linked to the alternative verbs, but it can be described with them (i.e. John and Mark play with the ball, Mark catches the ball). For this reason, the manual annotation provides more BS-to-scene relations than an algorithm can foresee on the basis of a lexical match, causing a low recall value.
Table 4 reports some statistics about the linking process; the entire results are browsable at the page http://bit.ly/2a4FefT.
IM Scenes linked to BS BS linked to Scenes IM English Verbs related to Scenes BabelNet English Verbs related to BS
Switching to Machine Learning had a strong
impact on this linking task. The main advantage
from the previous linking experiment
For example the BS ”bn:00090224v” (Put into a certain place or abstract location) expresses a general concept and is linked to 72 scenes, comprising the actions involving one or more objects or a body part, relating to different ways of putting (like inserting, throwing, attaching,...) or to different states of the Theme (e.g. solid or liquid). Conversely, the BS ”bn:00084326v” (Fasten with buttons) is much more specific and is linked to only one scene (c17d7346) that represents a man that fastens his jacket. 5
The experiment described in this paper shows that is possible to obtain an extensive linking between IMAGACT and BabelNet through visual entities (see Figure 1 for a visual representation of a linking example); this can be advantageous for both the resources. BabelNet can add a clear video representation of the verbal synsets that refer to actions; IMAGACT can import verb translation candidates from many languages by exploiting the BabelNet semantic network; their integration can be exploited as a unified multimedial resource to accomplish complex tasks that combine natural language processing and computer vision.
Finally, we feel important to note that this procedure is scalable and the statistical model can be retrained at resource changes. This is a fundamental feature, especially considering the continuous update of IMAGACT languages and lemmas inventory.
This research has been supported by the MODELACT Project, funded by the Futuro in Ricerca 2012 programme (Project Code RBFR12C608); http://modelact.lablita.it. may. European Language Resources Association (ELRA). Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL 2012), Jeju, Korea.