MACID - Multimodal ACtion IDentification: A CALAMITA Challenge Andrea Amelio Ravelli1,*,† , Rossella Varvara2,† and Lorenzo Gregori3,† 1 ABSTRACTION Research Group - University of Bologna 2 Independent Researcher 3 University of Florence Abstract This paper presents the Multimodal ACtion IDentification challenge (MACID), part of the first CALAMITA competition. The objective of this task is to evaluate the ability of Large Language Models (LLMs) to differentiate between closely related action concepts based on textual descriptions alone. The challenge is inspired by the "find the intruder" task, where models must identify an outlier among a set of 4 sentences that describe similar yet distinct actions. The dataset is composed of “pushing” events, and it highlights action-predicate mismatches, where the same verb may describe different actions or different verbs may refer to the same action. Although currently mono-modal (text-only), the task is designed for future multimodal integration, linking visual and textual representations to enhance action recognition. By probing a model’s capacity to resolve subtle linguistic ambiguities, the challenge underscores the need for deeper cognitive understanding in action-language alignment, ultimately testing the boundaries of LLMs’ ability to interpret action verbs and their associated concepts. Keywords human action recognition, action types, find the intruder, LLM, CALAMITA, CLiC-it 1. Introduction and Motivation starts from action capabilities that language emerged during human evolution. In this view, understanding and Human language and vision systems are deeply linked discriminating actions are of paramount importance for together, and the two may have a common evolutionary the broader scope of language understanding. basis. According to the Mirror System Hypothesis [1] Natural Language Processing is experiencing an un- the mechanism that supports language in the human precedented revolution due to the development of mod- brain may have evolved atop the mirror neuron system els capable of understanding and generating language; for grasping, taking advantage of its ability to recognize these models show human-like performances in solving a set of actions, and adapting it to deal with linguistic many tasks (and above-human performance on some). acts (i.e. utterances) and to discriminate linguistic objects Moreover, the recent development of multimodal LLMs (i.e., audio patterns for words). Thus, according to this allowed deep reasoning tasks involving the simultaneous hypothesis, humans “invented” language by adapting the processing of both textual and visual data. pattern recognition system, initially developed within the With the MACID task at CALAMITA [2], we aim to vision system to recognize actions, to identify and imitate challenge LLMs on their ability to finely discriminate audio patterns, and to link them to real-world entities (i.e. between linguistic expressions referring to cognitively objects and events) and their mental representation. In distinct but linguistically similar actions, due to the use other words, language is a form of action, and it probably of the same (or remarkably close) word labels to describe them. While the discrimination of very distant actions is CLiC-it 2024: Tenth Italian Conference on Computational Linguistics, a quite simple task (e.g. to distinguish between “opening Dec 04 — 06, 2024, Pisa, Italy * Corresponding author. a box” and “pressing a button”), grasping the nuances † These authors contributed equally. between actions that are much closer semantically is $ andreaamelio.ravelli@unibo.it (A. A. Ravelli); not so obvious (e.g. “pressing a button” and “pressing rossella.varvara01@gmail.com (R. Varvara); the wood”). These nuances are easy to highlight for a lorenzo.gregori@unifi.it (L. Gregori) human, which can activate a simulated execution and € https://www.unibo.it/sitoweb/andreaamelio.ravelli thus find differences in motor execution, but a model (A. A. Ravelli); https://scholar.google.com/citations?user=qAIgPcMAAAAJ without a physical dimension cannot. We aim to test to (R. Varvara); which degree an LLM can find the relevant information https://cercachi.unifi.it/p-doc2-2022-0-A-2c303c2b3930-0.html to recognize action concepts from their linguistic descrip- (L. Gregori) tion. Moreover, visual information, in these scenarios,  0000-0002-0232-8881 (A. A. Ravelli); 0000-0001-9957-2807 can facilitate the task for the computational model, pro- (R. Varvara); 0000-0001-9208-2311 (L. Gregori) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License viding more cues to disambiguate. For this reason, the Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings Figure 1: An example of the data from the MACID Task. proposed dataset has been conceived as a multimodal The task shares similarity with a word-sense discrimi- resource, with links between textual descriptions of ac- nation task, since different senses of an action verb refer tions and the short movie segments where these actions to different actions. However, the present task requires are performed. a deeper cognitive understanding of the sentences pro- Currently, the CALAMITA challenge does not deal vided, given that the action can be described through with multi-modal LLMs, so for the first MACID com- different predicates and, the other way around, the same petition, we are presenting the text-only version of the predicate can extend to a variety of actions. Indeed, the dataset. task forces the model to question a one-to-one relation- ship between meaning and form. 2. Challenge Description 3. Data description We propose a task modeled over the typical “find the intruder” game, similarly to Chang et al. [3], but extend- We derived the data for this proposal from a small por- ing it to sentences instead of words in isolation. Among tion of the LSMDC dataset [4], which contains short a group of 4 video-caption pairs, the model is asked to video clips extracted from movies, along with English select the one that does not refer to the same kind of DVS (descriptive video services) transcription for visu- action as the other three. For the task to be challenging, ally impaired people. The LSMDC dataset is the result we focus on actions-predicate mismatches: of the merging of two previous dataset, both built upon DVS from movies: the Max Plank Institute für Informatik • different action concepts that may be defined by Movie Description Dataset (MPII-MD) [5], and the Mon- the same verb (e.g. “pressing a button” and “press- treal Video Annotation Dataset (M-VAD) [6]. The subset ing the wood”); considered for this task is a collection of video-caption • the expression of the same action concept pairs restricted to the variation of the actions (and action through different verbs (e.g. “pressing a button” verbs) linked to “pushing” events. and “pushing a button”). Data have been manually filtered and annotated [7] using the action conceptualization derived from the IMA- The challenge is mono-modal (i.e., text-only), but is GACT Multilingual and Multimodal Ontology of Actions ready to be turned in a multi-modal task (i.e., visual and [8]. IMAGACT is a multimodal and multilingual ontol- linguistic information through video-caption pairs). ogy of actions that provides a fine-grained categorization (1) I due ragazzi spingono il carrello verso la colonna of action concepts, each represented by one or more vi- (The two boys push the cart toward the column) sual prototypes in the form of recorded videos and 3D [action id: 65431186] animations. IMAGACT currently contains 1,010 scenes (2) La donna spinge la signora anziana sulla sedia a that encompass the action concepts most commonly re- rotelle (The woman pushes the elderly lady in the ferred to in everyday language usage. Scenes belonging wheelchair) to the same action concept are grouped together and [action id: 65431186] labeled with a unique identification number. The cate- gorization of action concepts proposed in the theoretical (3) L’uomo spinge a terra l’aggressore (The man pushes framework behind IMAGACT has been validated in a the attacker to the ground) series of experiments with a high inter-annotator agree- [action id: 18ad2fa9] ment [9], confirming that the theoretical framework can (4) L’infermiere spinge la barella (The nurse pushes the be considered well-founded and reproducible. gurney) We wrote an Italian caption for each of the selected [action id: 65431186] videos from LSMDC, which originally had only an En- glish textual description. The captioning took into ac- TUPLE_2 count the necessity to produce a sounding Italian de- scription, thus we chose the most appropriate verb (and (1) La donna si spinge fuori dalla piscina (The woman construction) to describe the action depicted in the videos. pushes herself out of the pool) Moreover, we choose to keep the anonymization as pro- [action id: 950a69d5] posed in the LSMDC, but instead of using SOMEONE as (2) L’uomo si solleva leggermente dalla donna sdraiata the only replacement of nouns, we choose to use general (The man lifts himself slightly off the lying woman) expressions such as il ragazzo (the boy), la donna (the [action id: 950a69d5] woman, and so on. In this way, we removed some ambi- guities from the original dataset (e.g., SOMEONE pushes (3) Il ragazzo a terra si alza in ginocchio con fatica SOMEONE). (The boy on the ground gets up to his knees with The MACID Task can also be framed as a multilingual difficulty) task, given the already available parallel English captions, [action id: 950a69d5] and the possibility to provide more translations in other (4) L’uomo preme il fazzoletto contro la sua narice languages. (The man presses the tissue against his nostril) [action id: 8b2675f8] 3.1. Data format For each group, the model must select the caption The MACID dataset is available on HuggingFace.1 referring to the intruder action. The action ID will be The dataset consists of groups of 4 captions (or video- masked to the system and used for evaluating the model’s caption pairs, in the case of the multimodal version), performance, but the ID of the corresponding video will three of which belong to the same action concept, and be added, in order to enable researchers to evaluate also one describing another action type. multimodal models. Data are released in CSV format (columns: id, s1, v1, s2, v2, s3, v3, s4, v4, intruder), with the following meaning: 3.2. Example of prompts used for zero • id: the tuple id; shot • s1-4: the 4 sentences describing physical actions; The task is evaluated with a zero-shot prompt only. The • v1-4: the 4 videos depicting physical actions; prompt used is reported in the example below. • intruder: the number (1-4) of the sentence (and video) which is the intruder in the group. Le seguenti 4 frasi sono descrizioni di azioni fisiche. Tre di queste azioni sono dello stesso tipo, mentre An additional folder with the video files is included in una è di un tipo diverso. Individua la frase che de- the dataset for future extension to the multimodal task. scribe l’azione di tipo diverso rispondendo soltanto An example of the textual data follows. con il numero della frase (1, 2, 3 o 4). 1: I due ragazzi spingono il carrello verso la colonna TUPLE_1 2: La donna spinge la signora anziana sulla sedia a rotelle 3: L’uomo spinge a terra l’aggressore 1 https://huggingface.co/datasets/loregreg/MACID 4: L’infermiere spinge la barella Tuples 100 3. two different verbs, with two sentences sharing Textual descriptions 307 the same verb (2_2); Videos 307 Action Types 18 4. two different verbs, with three sentences sharing Action verbs 24 the same verb and one with a different one (3_1); 5. one verb in all the four sentences (4). Table 1 MACID dataset statistics. Table 3 reports the distribution of the stimuli across the 5 schemes. Across all the stimuli and the distribution verb freq verb freq schemes, the intruder contains the same verb of at least spingere 233 urtare 2 one other sentence in 62 out of 100 cases. premere 83 tirare 2 spostare 18 respingere 2 Verb variation scheme Count sollevare 11 passare 2 1_1_1_1 7 allontanare 8 chiudere 2 2_1_1 16 portare 5 attraversare 2 2_2 9 chiamare 5 suonare 1 3_1 44 abbassare 5 poggiare 1 4 24 scostare 4 gettare 1 Total 100 alzare 4 condurre 1 schiacciare 3 fare pressione 1 Table 3 pigiare 3 fare largo 1 Distribution of the verb variation scheme across the stimuli of the MACID dataset. Table 2 Frequency list of verbs used in the textual captions. 4. Metrics 3.3. Detailed data statistics The evaluation metric proposed for the MACID Task is a MACID dataset is made of 100 tuples, each one containing simple accuracy: participating models will be evaluated 4 textual descriptions of human actions in the form of on the basis of the percentage of correct times they select short sentences in Italian, and 4 video segments depicting the intruder sentence in each 4-word tuple. those actions. See Table 1 for general details. The whole dataset is built using 307 hand-crafted captions, with each caption appearing at least once (either as positive 5. Limitations sentence or as intruder), and for a maximum of 3 times (counting both the possible roles). The main limitation of the MACID Task dataset is its size. The dataset contains 18 action types, belonging to the We propose a set of 100 4-sentence tuples, as the MACID semantic area of pushing events. Table 2 reports the Task is intended as a zero-shot LLMs-only challenge, thus frequency list of verbs used to describe the actions. we did not designed it as a typical Machine Learning task In building the 4-sentence tuples, we maximized the with train(-dev)-test splitting. The possibility to have balancing between close and distant action concepts, by many more stimuli would open up to the possibility to choosing the intruder captions on the basis of the dis- tackle the task with other kind of models, but also to offer tance computed over the whole IMAGACT ontology data exemplars to be used to better inform LLMs about the [10, 11, 12]. Thus, we compiled the stimuli by paying required behavior. attention to the distance between the action concepts of the three positive sentences and the intruder, trying to balance as much as possible between intruders with Acknowledgments action concepts of high, medium or low similarity with This work was partially supported by the Project ERC- respect to the action concept shared by the other three 2021-STG-101039777 (ABSTRACTION), funded by the sentences in the stimulus. Furthermore, we also put our European Union. Views and opinions expressed are how- attention on creating stimuli which are varied in terms ever those of the author(s) only and do not necessarily of action verbs, resulting in 5 possible patterns of verbs reflect those of the European Union or the European Re- distribution across the 4 sentences of a stimulus: search Council Executive Agency. Neither the European 1. four different verbs, i.e. one unique verb per sen- Union nor the granting authority can be held responsible tence (1_1_1_1); for them. 2. three different verbs, with a couple of sentences with the same verb (2_1_1); References [12] L. Gregori, M. Moneglia, A. Panunzi, Towards a crosslinguistic identification of action concepts. au- [1] M. Arbib, G. Rizzolatti, Neural expectations: A tomatic clustering of video scenes based on the possible evolutionary path from manual skills to imagact multilingual ontology, in: AREA II work- language, Communication and Cognition 29 (1996) shop. Annotation, Recognition and Evaluation of 393–424. Action, On line Areaworkshop. org, 2022, pp. 1–9. [2] G. Attanasio, P. Basile, F. Borazio, D. Croce, M. Fran- cis, J. Gili, E. Musacchio, M. Nissim, V. Patti, M. Ri- naldi, D. Scalena, CALAMITA: Challenge the Abili- ties of LAnguage Models in ITAlian, in: Proceed- ings of the 10th Italian Conference on Computa- tional Linguistics (CLiC-it 2024), Pisa, Italy, Decem- ber 4 - December 6, 2024, CEUR Workshop Proceed- ings, CEUR-WS.org, 2024. [3] J. Chang, S. Gerrish, C. Wang, J. Boyd-Graber, D. Blei, Reading tea leaves: How humans inter- pret topic models, Advances in neural information processing systems 22 (2009). [4] A. Rohrbach, A. Torabi, M. Rohrbach, N. Tandon, C. Pal, H. Larochelle, A. Courville, B. Schiele, Movie description, International Journal of Computer Vi- sion 123 (2017) 94–120. [5] A. Rohrbach, M. Rohrbach, N. Tandon, B. Schiele, A dataset for movie description, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3202–3212. [6] A. Torabi, C. Pal, H. Larochelle, A. Courville, Us- ing descriptive video services to create a large data source for video annotation research, arXiv preprint arXiv:1503.01070 (2015). [7] A. A. Ravelli, Annotation of linguistically derived action concepts in computer vision datasets, Ph.D. thesis, University of Florence, 2020. [8] M. Moneglia, S. W. Brown, F. Frontini, G. Gagliardi, F. Khan, M. Monachini, A. Panunzi, et al., The imagact visual ontology. an extendable multilin- gual infrastructure for the representation of lexical encoding of action, in: Proceedings of the Ninth International Conference on Language Resources and Evaluation–LREC’14, European Language Re- sources Association (ELRA), 2014, pp. 3425–3432. [9] G. Gagliardi, Rappresentazione dei concetti azion- ali attraverso prototipi e accordo nella categoriz- zazione dei verbi generali. una validazione statistica, in: Proceedings of the First Italian Conference on Computational Linguistics–CLiC-it, 2014, pp. 180– 185. [10] L. Gregori, R. Varvara, A. A. Ravelli, Action type induction from multilingual lexical features, Proce- samiento del Lenguaje Natural 63 (2019) 85–92. [11] A. A. Ravelli, L. Gregori, R. Varvara, Comparing ref- vectors and word embeddings in a verb semantic similarity task, in: Proceedings of the 3rd Work- shop on Natural Language for Artificial Intelligence, CEUR-WS. org, 2019, pp. 0–0.