This paper presents the status of our work aimed at enriching the IMAGACT Ontology of Action by linking it to other resources. In order to achieve this goal we performed a visual mapping, exploiting the IMAGACT visual component (video scenes that represent physical actions) as the linkage point among resources. By using visual objects, which are free from linguistic constraints and can be interpreted and described from different perspectives, we connected resources responding to different scopes and theoretical frameworks, in which a concept-to-concept mapping appeared difficult to obtain. We provide a brief description of two linking obtained by using this technique: an automatic linking between IMAGACT and BabelNet, a multilingual semantic network, and a manual linking between IMAGACT and Praxicon, a conceptual knowledge base of action.
Action verbs contain the basic information that should be understood in order to make
sense of a sentence and that should be processed in instructions given to artificial systems.
The difficulty behind action verbs understanding comes out from the evidence that no one
to one correspondence can be established between action predicates and action concepts.
The same action can be predicated by multiple verbs (e.g. “John takes/brings/leads Mary
to the restaurant”) and, conversely, one verb can extend to multiple and different actions
(e.g. “John takes the cup from the table”, “John takes/brings the cup to Mary”). Most
of these verbs belong to the class of general verbs, which are characterized by a high
ambiguity and high frequency in the use [
The representation is more difficult in a multilingual perspective, given that
different languages operate different action space segmentations. It has been observed [
This paper reports two linking experiments performed on the IMAGACT Visual Ontology of Action, in order to gather information about actions from several perspective and at different levels: semantic, motoric and visual. Linkings have been led exploiting the visual information of IMAGACT: instead of a classic concept-to-concept mapping we performed a visual mapping, that is a concept-to-video linking. This strategy allowed us to connect linguistic resources having different conceptualizations of events.
This work, far from being definitive, could be useful for the future construction of integrated resources on action understanding, to be effectively exploited for both theoretical analysis and computational applications.
Verbs are the lexical class that is normally responsible for event categorization. Among events, actions (defined as goal-oriented events performed by an intentional agent) play an important role from a linguistic perspective: action verbs are very frequent in spoken language and they are also very ambiguous. Moreover the semantic classification of action verbs is more complex and not equally linear as the one of nouns, so that frequently it’s not possible to discriminate a coherent list of word senses.
IMAGACT Visual Ontology of Action1 [
Verbs in IMAGACT are divided into action types, according to their semantic variation; each type is linked to one or more video scenes (either 3D animations or filmed video clips), in which a prototypical action is performed. The verbs referring to the same concept are linked to the same scenes, creating an interlinguistic semantic network.
The ontology is in continous development and, at present, contains 9 fully-mapped languages and 13 that are underway, with an average of 730 action verbs per language.
This resource gives a broad picture of the variety of actions and activities that are prominent in everyday life and specifies the lexicon used to express each one in ordinary communication, in all the included languages.
In order to collect more information, we planned an extensive campaign of enrichment, through the comparison and mutual exchange with other resources.
For this task we applied a visual mapping, a methodology which aims at pointing concepts to a shared visual representation. In fact, a video depicting an event is not subject to any linguistic constraint, and the associated semantic information can be described in various manners. Starting from this observation we used the videos to link concepts of different resources, that express independent event conceptualization according to their own theoretical framework. It follows that the multimodal feature of IMAGACT is a key point for its enrichment and implementation.
Herein we present some current results we obtained by linking IMAGACT with BabelNet and Praxicon. An example of the obtained output can be observed in Figure 1, that shows a beating event with the parallel representation in the 3 resources.
BabelNet2 [
BabelNet concepts (the BSs) are interlinguistic: they gather all the word senses in different languages that are semantically equivalent (or almost equivalent). Conversely, IMAGACT action types encode small semantic differences, so they are more granular and language-dependent. Given these differences, an exact match between their concepts is very rare; it’s also hard to establish less strict semantic relations (e.g. narrow-to-broad), because the BSs boundaries are often fuzzy and the gloss is not always able to make a clear discrimination between them.
In this case visual mapping solved the problem: in fact even for the BSs where the description is not precise, it’s easy to say if a video is a good action prototype for it or not3.
Given the multilingual nature of the two resources, we could exploit a rich lexical
information, i.e. all the verbs in many languages related both to IMAGACT scenes and
BabelNet BSs. The connections between BSs and scenes have been automatically
established on the basis of the number of shared verbal lemmas through an ML algorithm
[
As a result from this linking, on the one hand, IMAGACT gained translation information for languages still not implemented in the Visual Ontology and, on the other, BSs referring to action verbs obtained a video representation. In Table 1, the detailed numbers of scenes and BSs connected through this linking are shown.
Praxicon4 is an ontology for the representation of action concepts, based on the
Minimalistic Grammar of Action [
Praxicon makes a distinction between Actions, Movements, and Events5. Actions are sets of structured motoric execution, intentionally performed by an agent to achieve a goal. The goal is a necessary component, so any non-voluntary motoric activation is addressed as a Movement, but not as an Action. Finally, actions that are too complex to be described as a set of motoric concepts, are considered Events and are out of the scope of the Praxicon resource.
Similarly to the linking with BabelNet, the IMAGACT scenes are used to connect the information of the two resources, given that their definitions of concepts are too different to try a proper and extensive sense matching. In fact, the IMAGACT scenes can work as a visual representation for Praxicon action concepts and, at the same time, Praxicon syntax could be used to analytically describe, from a physical-motoric point of view, all the low-level actions involved in the execution of more complex ones.
Differently from the previous linking, in this case it is a totally manual work, consisting in the analysis of each scene and the determination of the physical action performed.
3The measured inter-rater agreement for this task is a Fleiss k of 0.74 with 3 annotators. Annotated dataset is available at http://bit.ly/2jt2cD4 4https://github.com/CSRI/PraxiconDB 5These categories have their own definition in the Praxicon framework. We use capital letters when referring to this specific meaning
The scene annotation has been accomplished on 281 IMAGACT scenes ( 28% of the total) and we obtained the following results: 154 scenes ( 55%) have a one-to-one relation with Praxicon Action concepts; 64 scenes ( 23%) map on more than one Action concept; 19 scenes ( 7%) are Movement but not Actions (in the Praxicon framework); 30 ( 11%) are Events but not Actions (in the Praxicon framework); 14 scenes ( 5%) are unclear.
IMAGACT scenes are specifically created to provide a prototypical representation of a lexicalized action concept: every scene is a reference of at least an English action verb. This allowed us to derive from these numbers some considerations about the relation between motoric and lexical level.
In Praxicon Events motoric properties does not play a role in the verb meaning, which encodes an abstract result, that is independent from the physical action execution. Examples are verbs like to drive, to clean or to rob, that encode a complex set of motoric actions by predicating their final result: 11% of the actions that are commonly referred with the language (in English) belong to this class. Conversely, 55% of the scenes have a one-to-one mapping with a Praxicon concept, meaning that there is a low distance between motoric and lexical level: we can consider these cases as the ones where the physical execution of an action more deeply affect the verb semantics. Example verbs of this class are to push, to gallop or to brush. Then, 23% of retrieved action are at an intermediate level of abstraction: they can be expressed in terms of physical action concepts, but more than one Praxicon concept is involved into a single lexicalized action. Some example verbs are to break, to open or to glue. Finally, we found that 7% of events that in English are referred through action verbs are Movement that do not correspond to voluntary actions, like to fall or to drop.
This work is still in progress, but we believe that the integration between
linguistic and motoric knowledge on action is very relevant both for theoretical analysis and
robotic applications. From one side an integrated resource is desirable to carry on deep
investigations on the relation between language and action, that is a long debated subject
in linguistics and neuroscience [
In this paper we presented the very first steps in the construction of a comprehensive resource for the understanding of actions and their representation in the language systems, built on top of the ontological structure of IMAGACT.
We introduced the visual mapping methodology, that allows resource linking through visual representations. This approach is particularly useful when it’s hard to find relations between concepts, because it does not force any kind of convergence between senses. For this reason we feel confident that this methodology could be successfully applied also in other linking tasks involving multimodal resources.
Two case studies have been described: the linking of IMAGACT with BabelNet and Praxicon. In the first case we were dealing with lexical-semantic resources having huge differences in sense discrimination and for this reason it was hard to find inter-resource semantic relations. In the case of Praxicon we applied visual mapping to link IMAGACT with a resource of a different type, in which the concepts are motoric and not linguistic.
Finally, to extend the information connected to action concepts, we aim to enrich our
ontology with the annotation of noun senses and with the predicate argument structures
[