=Paper=
{{Paper
|id=Vol-3878/137_calamita_short
|storemode=property
|title=MACID - Multimodal ACtion IDentification: A CALAMITA Challenge
|pdfUrl=https://ceur-ws.org/Vol-3878/137_calamita_short.pdf
|volume=Vol-3878
|authors=Andrea Amelio Ravelli,Rossella Varvara,Lorenzo Gregori
|dblpUrl=https://dblp.org/rec/conf/clic-it/RavelliVG24
}}
==MACID - Multimodal ACtion IDentification: A CALAMITA Challenge==
https://ceur-ws.org/Vol-3878/137_calamita_short.pdf
MACID - Multimodal ACtion IDentification:
A CALAMITA Challenge
Andrea Amelio Ravelli1,*,† , Rossella Varvara2,† and Lorenzo Gregori3,†
1
ABSTRACTION Research Group - University of Bologna
2
Independent Researcher
3
University of Florence
Abstract
This paper presents the Multimodal ACtion IDentification challenge (MACID), part of the first CALAMITA competition.
The objective of this task is to evaluate the ability of Large Language Models (LLMs) to differentiate between closely related
action concepts based on textual descriptions alone. The challenge is inspired by the "find the intruder" task, where models
must identify an outlier among a set of 4 sentences that describe similar yet distinct actions. The dataset is composed of
“pushing” events, and it highlights action-predicate mismatches, where the same verb may describe different actions or
different verbs may refer to the same action. Although currently mono-modal (text-only), the task is designed for future
multimodal integration, linking visual and textual representations to enhance action recognition. By probing a model’s
capacity to resolve subtle linguistic ambiguities, the challenge underscores the need for deeper cognitive understanding in
action-language alignment, ultimately testing the boundaries of LLMs’ ability to interpret action verbs and their associated
concepts.
Keywords
human action recognition, action types, find the intruder, LLM, CALAMITA, CLiC-it
1. Introduction and Motivation starts from action capabilities that language emerged
during human evolution. In this view, understanding and
Human language and vision systems are deeply linked discriminating actions are of paramount importance for
together, and the two may have a common evolutionary the broader scope of language understanding.
basis. According to the Mirror System Hypothesis [1] Natural Language Processing is experiencing an un-
the mechanism that supports language in the human precedented revolution due to the development of mod-
brain may have evolved atop the mirror neuron system els capable of understanding and generating language;
for grasping, taking advantage of its ability to recognize these models show human-like performances in solving
a set of actions, and adapting it to deal with linguistic many tasks (and above-human performance on some).
acts (i.e. utterances) and to discriminate linguistic objects Moreover, the recent development of multimodal LLMs
(i.e., audio patterns for words). Thus, according to this allowed deep reasoning tasks involving the simultaneous
hypothesis, humans “invented” language by adapting the processing of both textual and visual data.
pattern recognition system, initially developed within the With the MACID task at CALAMITA [2], we aim to
vision system to recognize actions, to identify and imitate challenge LLMs on their ability to finely discriminate
audio patterns, and to link them to real-world entities (i.e. between linguistic expressions referring to cognitively
objects and events) and their mental representation. In distinct but linguistically similar actions, due to the use
other words, language is a form of action, and it probably of the same (or remarkably close) word labels to describe
them. While the discrimination of very distant actions is
CLiC-it 2024: Tenth Italian Conference on Computational Linguistics,
a quite simple task (e.g. to distinguish between “opening
Dec 04 — 06, 2024, Pisa, Italy
*
Corresponding author. a box” and “pressing a button”), grasping the nuances
†
These authors contributed equally. between actions that are much closer semantically is
$ andreaamelio.ravelli@unibo.it (A. A. Ravelli); not so obvious (e.g. “pressing a button” and “pressing
rossella.varvara01@gmail.com (R. Varvara); the wood”). These nuances are easy to highlight for a
lorenzo.gregori@unifi.it (L. Gregori) human, which can activate a simulated execution and
https://www.unibo.it/sitoweb/andreaamelio.ravelli
thus find differences in motor execution, but a model
(A. A. Ravelli);
https://scholar.google.com/citations?user=qAIgPcMAAAAJ without a physical dimension cannot. We aim to test to
(R. Varvara); which degree an LLM can find the relevant information
https://cercachi.unifi.it/p-doc2-2022-0-A-2c303c2b3930-0.html to recognize action concepts from their linguistic descrip-
(L. Gregori) tion. Moreover, visual information, in these scenarios,
0000-0002-0232-8881 (A. A. Ravelli); 0000-0001-9957-2807
can facilitate the task for the computational model, pro-
(R. Varvara); 0000-0001-9208-2311 (L. Gregori)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License viding more cues to disambiguate. For this reason, the
Attribution 4.0 International (CC BY 4.0).
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
Figure 1: An example of the data from the MACID Task.
proposed dataset has been conceived as a multimodal The task shares similarity with a word-sense discrimi-
resource, with links between textual descriptions of ac- nation task, since different senses of an action verb refer
tions and the short movie segments where these actions to different actions. However, the present task requires
are performed. a deeper cognitive understanding of the sentences pro-
Currently, the CALAMITA challenge does not deal vided, given that the action can be described through
with multi-modal LLMs, so for the first MACID com- different predicates and, the other way around, the same
petition, we are presenting the text-only version of the predicate can extend to a variety of actions. Indeed, the
dataset. task forces the model to question a one-to-one relation-
ship between meaning and form.
2. Challenge Description
3. Data description
We propose a task modeled over the typical “find the
intruder” game, similarly to Chang et al. [3], but extend- We derived the data for this proposal from a small por-
ing it to sentences instead of words in isolation. Among tion of the LSMDC dataset [4], which contains short
a group of 4 video-caption pairs, the model is asked to video clips extracted from movies, along with English
select the one that does not refer to the same kind of DVS (descriptive video services) transcription for visu-
action as the other three. For the task to be challenging, ally impaired people. The LSMDC dataset is the result
we focus on actions-predicate mismatches: of the merging of two previous dataset, both built upon
DVS from movies: the Max Plank Institute für Informatik
• different action concepts that may be defined by Movie Description Dataset (MPII-MD) [5], and the Mon-
the same verb (e.g. “pressing a button” and “press- treal Video Annotation Dataset (M-VAD) [6]. The subset
ing the wood”); considered for this task is a collection of video-caption
• the expression of the same action concept pairs restricted to the variation of the actions (and action
through different verbs (e.g. “pressing a button” verbs) linked to “pushing” events.
and “pushing a button”). Data have been manually filtered and annotated [7]
using the action conceptualization derived from the IMA-
The challenge is mono-modal (i.e., text-only), but is
GACT Multilingual and Multimodal Ontology of Actions
ready to be turned in a multi-modal task (i.e., visual and
[8]. IMAGACT is a multimodal and multilingual ontol-
linguistic information through video-caption pairs).
ogy of actions that provides a fine-grained categorization (1) I due ragazzi spingono il carrello verso la colonna
of action concepts, each represented by one or more vi- (The two boys push the cart toward the column)
sual prototypes in the form of recorded videos and 3D [action id: 65431186]
animations. IMAGACT currently contains 1,010 scenes
(2) La donna spinge la signora anziana sulla sedia a
that encompass the action concepts most commonly re-
rotelle (The woman pushes the elderly lady in the
ferred to in everyday language usage. Scenes belonging
wheelchair)
to the same action concept are grouped together and
[action id: 65431186]
labeled with a unique identification number. The cate-
gorization of action concepts proposed in the theoretical (3) L’uomo spinge a terra l’aggressore (The man pushes
framework behind IMAGACT has been validated in a the attacker to the ground)
series of experiments with a high inter-annotator agree- [action id: 18ad2fa9]
ment [9], confirming that the theoretical framework can (4) L’infermiere spinge la barella (The nurse pushes the
be considered well-founded and reproducible. gurney)
We wrote an Italian caption for each of the selected [action id: 65431186]
videos from LSMDC, which originally had only an En-
glish textual description. The captioning took into ac- TUPLE_2
count the necessity to produce a sounding Italian de-
scription, thus we chose the most appropriate verb (and (1) La donna si spinge fuori dalla piscina (The woman
construction) to describe the action depicted in the videos. pushes herself out of the pool)
Moreover, we choose to keep the anonymization as pro- [action id: 950a69d5]
posed in the LSMDC, but instead of using SOMEONE as (2) L’uomo si solleva leggermente dalla donna sdraiata
the only replacement of nouns, we choose to use general (The man lifts himself slightly off the lying woman)
expressions such as il ragazzo (the boy), la donna (the [action id: 950a69d5]
woman, and so on. In this way, we removed some ambi-
guities from the original dataset (e.g., SOMEONE pushes (3) Il ragazzo a terra si alza in ginocchio con fatica
SOMEONE). (The boy on the ground gets up to his knees with
The MACID Task can also be framed as a multilingual difficulty)
task, given the already available parallel English captions, [action id: 950a69d5]
and the possibility to provide more translations in other (4) L’uomo preme il fazzoletto contro la sua narice
languages. (The man presses the tissue against his nostril)
[action id: 8b2675f8]
3.1. Data format
For each group, the model must select the caption
The MACID dataset is available on HuggingFace.1 referring to the intruder action. The action ID will be
The dataset consists of groups of 4 captions (or video- masked to the system and used for evaluating the model’s
caption pairs, in the case of the multimodal version), performance, but the ID of the corresponding video will
three of which belong to the same action concept, and be added, in order to enable researchers to evaluate also
one describing another action type. multimodal models.
Data are released in CSV format (columns: id, s1, v1, s2,
v2, s3, v3, s4, v4, intruder), with the following meaning:
3.2. Example of prompts used for zero
• id: the tuple id; shot
• s1-4: the 4 sentences describing physical actions;
The task is evaluated with a zero-shot prompt only. The
• v1-4: the 4 videos depicting physical actions; prompt used is reported in the example below.
• intruder: the number (1-4) of the sentence (and
video) which is the intruder in the group. Le seguenti 4 frasi sono descrizioni di azioni fisiche.
Tre di queste azioni sono dello stesso tipo, mentre
An additional folder with the video files is included in una è di un tipo diverso. Individua la frase che de-
the dataset for future extension to the multimodal task. scribe l’azione di tipo diverso rispondendo soltanto
An example of the textual data follows. con il numero della frase (1, 2, 3 o 4).
1: I due ragazzi spingono il carrello verso la colonna
TUPLE_1 2: La donna spinge la signora anziana sulla sedia a
rotelle
3: L’uomo spinge a terra l’aggressore
1
https://huggingface.co/datasets/loregreg/MACID 4: L’infermiere spinge la barella
Tuples 100 3. two different verbs, with two sentences sharing
Textual descriptions 307
the same verb (2_2);
Videos 307
Action Types 18 4. two different verbs, with three sentences sharing
Action verbs 24 the same verb and one with a different one (3_1);
5. one verb in all the four sentences (4).
Table 1
MACID dataset statistics. Table 3 reports the distribution of the stimuli across
the 5 schemes. Across all the stimuli and the distribution
verb freq verb freq schemes, the intruder contains the same verb of at least
spingere 233 urtare 2 one other sentence in 62 out of 100 cases.
premere 83 tirare 2
spostare 18 respingere 2 Verb variation scheme Count
sollevare 11 passare 2 1_1_1_1 7
allontanare 8 chiudere 2 2_1_1 16
portare 5 attraversare 2 2_2 9
chiamare 5 suonare 1 3_1 44
abbassare 5 poggiare 1 4 24
scostare 4 gettare 1 Total 100
alzare 4 condurre 1
schiacciare 3 fare pressione 1 Table 3
pigiare 3 fare largo 1 Distribution of the verb variation scheme across the stimuli
of the MACID dataset.
Table 2
Frequency list of verbs used in the textual captions.
4. Metrics
3.3. Detailed data statistics
The evaluation metric proposed for the MACID Task is a
MACID dataset is made of 100 tuples, each one containing
simple accuracy: participating models will be evaluated
4 textual descriptions of human actions in the form of
on the basis of the percentage of correct times they select
short sentences in Italian, and 4 video segments depicting
the intruder sentence in each 4-word tuple.
those actions. See Table 1 for general details. The whole
dataset is built using 307 hand-crafted captions, with
each caption appearing at least once (either as positive 5. Limitations
sentence or as intruder), and for a maximum of 3 times
(counting both the possible roles). The main limitation of the MACID Task dataset is its size.
The dataset contains 18 action types, belonging to the We propose a set of 100 4-sentence tuples, as the MACID
semantic area of pushing events. Table 2 reports the Task is intended as a zero-shot LLMs-only challenge, thus
frequency list of verbs used to describe the actions. we did not designed it as a typical Machine Learning task
In building the 4-sentence tuples, we maximized the with train(-dev)-test splitting. The possibility to have
balancing between close and distant action concepts, by many more stimuli would open up to the possibility to
choosing the intruder captions on the basis of the dis- tackle the task with other kind of models, but also to offer
tance computed over the whole IMAGACT ontology data exemplars to be used to better inform LLMs about the
[10, 11, 12]. Thus, we compiled the stimuli by paying required behavior.
attention to the distance between the action concepts
of the three positive sentences and the intruder, trying
to balance as much as possible between intruders with Acknowledgments
action concepts of high, medium or low similarity with
This work was partially supported by the Project ERC-
respect to the action concept shared by the other three
2021-STG-101039777 (ABSTRACTION), funded by the
sentences in the stimulus. Furthermore, we also put our
European Union. Views and opinions expressed are how-
attention on creating stimuli which are varied in terms
ever those of the author(s) only and do not necessarily
of action verbs, resulting in 5 possible patterns of verbs
reflect those of the European Union or the European Re-
distribution across the 4 sentences of a stimulus:
search Council Executive Agency. Neither the European
1. four different verbs, i.e. one unique verb per sen- Union nor the granting authority can be held responsible
tence (1_1_1_1); for them.
2. three different verbs, with a couple of sentences
with the same verb (2_1_1);
References [12] L. Gregori, M. Moneglia, A. Panunzi, Towards a
crosslinguistic identification of action concepts. au-
[1] M. Arbib, G. Rizzolatti, Neural expectations: A tomatic clustering of video scenes based on the
possible evolutionary path from manual skills to imagact multilingual ontology, in: AREA II work-
language, Communication and Cognition 29 (1996) shop. Annotation, Recognition and Evaluation of
393–424. Action, On line Areaworkshop. org, 2022, pp. 1–9.
[2] G. Attanasio, P. Basile, F. Borazio, D. Croce, M. Fran-
cis, J. Gili, E. Musacchio, M. Nissim, V. Patti, M. Ri-
naldi, D. Scalena, CALAMITA: Challenge the Abili-
ties of LAnguage Models in ITAlian, in: Proceed-
ings of the 10th Italian Conference on Computa-
tional Linguistics (CLiC-it 2024), Pisa, Italy, Decem-
ber 4 - December 6, 2024, CEUR Workshop Proceed-
ings, CEUR-WS.org, 2024.
[3] J. Chang, S. Gerrish, C. Wang, J. Boyd-Graber,
D. Blei, Reading tea leaves: How humans inter-
pret topic models, Advances in neural information
processing systems 22 (2009).
[4] A. Rohrbach, A. Torabi, M. Rohrbach, N. Tandon,
C. Pal, H. Larochelle, A. Courville, B. Schiele, Movie
description, International Journal of Computer Vi-
sion 123 (2017) 94–120.
[5] A. Rohrbach, M. Rohrbach, N. Tandon, B. Schiele,
A dataset for movie description, in: Proceedings of
the IEEE conference on computer vision and pattern
recognition, 2015, pp. 3202–3212.
[6] A. Torabi, C. Pal, H. Larochelle, A. Courville, Us-
ing descriptive video services to create a large
data source for video annotation research, arXiv
preprint arXiv:1503.01070 (2015).
[7] A. A. Ravelli, Annotation of linguistically derived
action concepts in computer vision datasets, Ph.D.
thesis, University of Florence, 2020.
[8] M. Moneglia, S. W. Brown, F. Frontini, G. Gagliardi,
F. Khan, M. Monachini, A. Panunzi, et al., The
imagact visual ontology. an extendable multilin-
gual infrastructure for the representation of lexical
encoding of action, in: Proceedings of the Ninth
International Conference on Language Resources
and Evaluation–LREC’14, European Language Re-
sources Association (ELRA), 2014, pp. 3425–3432.
[9] G. Gagliardi, Rappresentazione dei concetti azion-
ali attraverso prototipi e accordo nella categoriz-
zazione dei verbi generali. una validazione statistica,
in: Proceedings of the First Italian Conference on
Computational Linguistics–CLiC-it, 2014, pp. 180–
185.
[10] L. Gregori, R. Varvara, A. A. Ravelli, Action type
induction from multilingual lexical features, Proce-
samiento del Lenguaje Natural 63 (2019) 85–92.
[11] A. A. Ravelli, L. Gregori, R. Varvara, Comparing ref-
vectors and word embeddings in a verb semantic
similarity task, in: Proceedings of the 3rd Work-
shop on Natural Language for Artificial Intelligence,
CEUR-WS. org, 2019, pp. 0–0.