Transformer-based Language Models (LMs) excel in many tasks, but they appear to lack robustness in capturing crucial aspects of event knowledge due to their reliance on surface-level linguistic features and the mismatch between language descriptions and real-world occurrences. In this paper, we investigate the potential of Transformer-based Vision-Language Models (VLMs) in comprehending Generalized Event Knowledge (GEK), aiming to determine whether the inclusion of a visual component afects the mastery of GEK. To do so, we compare multimodal Transformer models with unimodal ones on a task evaluating the plausibility of curated minimal sentence pairs. We show that current VLMs generally perform worse than their unimodal counterparts, suggesting that VL pre-training strategies are not yet as efective to model semantic understanding and resulting models are more akin to bag-of-words in this context.
CLiC-it 2023: 9th Italian Conference on Computational Linguistics, Nov 30 — Dec 02, 2023, Venice, Italy $ m.cassese4@studenti.unipi.it (M. Cassese); alessandro.bondielli@unipi.it (A. Bondielli); alessandro.lenci@unipi.it (A. Lenci)
0009-0007-4765-1221 (M. Cassese); 0000-0003-3426-6643 (A. Bondielli); 0000-0001-5790-4308 (A. Lenci)
© 2023 Copyright for this paper by its authors. Use permitted under Creative CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g CCoEmmUoRns LWiceonsrekAstthribouptionP4r.0oIncteerenadtiionnagl s(CC(CBYE4U.0)R.-WS.org)
The introduction of multimodal models in NLP stems from the intrinsic limitations of computational models that are trained exclusively on distributional statistics extracted from textual data [9, 3, 10]. In fact, they lack referential competence [11], which prevents them from grounding linguistic structures onto real-world experiences [12, 13].
The earliest multimodal distributional models already showed the ability to improve the semantic representation of concrete concepts and properties [14], as well as abstract verbs that lack direct perceptual information, but benefit from integrating linguistic inputs and perceptual information [15]. However, they proved to be less efective in representing verbs, adjectives, and abstract concepts [16].
The introduction of Transformer-based VLMs such as Visual-BERT [17] and FLAVA [18], and efective techniques for Vision-Language Pre-training [19] paved the way for new research in a multimodal setting. While numerous studies has shown VLMs success on diferent multimodal tasks, less efort has been put in analyzing their diferences with unimodal counterparts on natural language understanding (NLU). [20] show that both dualstream and single-stream VLMs are equally capable of preserving NLU capabilities. The analysis conducted by [6] shows that multimodal models do not significantly outperform the text-only variants in a language-only setting. This was attributed to the use of narrow domain data and direct extensions of NLP architectures. Our work support these findings by focusing on understanding the plausibility of events.
Our goal is to evaluate the ability of LMs and VLMs to predict the semantic plausibility of sentences with respect to human judgements. In the following, we describe the data used in the experiments (Sec. 3.1), the models we considered in the evaluation (Sec. 3.2), and detail the evaluation procedure itself (Sec. 3.3). 3.1. Data We sourced our data from a number of existing datasets containing pairs of sentences describing transitive event distinguished by patient plausibility within the context of the sentence. Plausibility is rated by humans and expressed on a 1-7 Likert scale. Formally, each data point consists of a plausible sentence and its corresponding implausible one obtained through a modification of .
guished by patient prototipicality. For each plausible sentence, the implausible (i.e., atypical) one is obtained by replacing the patient with an atypical filler for that role (e.g., The actor won the award vs The actor won the battle).
The cop arrested the criminal vs. The criminal arrested the cop). The dataset is divided into two sub-datasets. In the former, henceforth referred to as − , both the agent and the patient are animate. In the latter, denoted as − , the agent of the original sentence is animate, while the patient is not. Thus the implausible sentence is also semantically impossible.
ing events in the present progressive tense. Like EventsAdapt, implausible sentences are obtained by reversing the noun phrases, which in this case always depict animate entities (e.g., The cat is chasing the mouse vs. The mouse is chasing the cat). Each sentence, both plausible and implausible, is accompanied by an image depicting the interaction between the two animated participants described in the sentence. The images are simple black and white drawings.
concreteness in VLMs’ ability to recognize plausibility, we further grouped sentences of EventsAdapt (and its subgroups) and DTFit into concrete and abstract ones. We categorized the sentences based on the level of concreteness of the verb, subject, and object in each sentence. We chose to consider sentences that refer to abstract concepts with high imageability as concrete (e.g., The priest celebrated the marriage).
To the best of our knowledge, none of the data used in this study was included in the training set of the evaluated models.
coder model initialized from pre-trained
BERTbase weights and further trained on multimodal
datasets. Visual features are extracted from a
pretrained Faster R-CNN network [
model for textual feature extraction and a ViT model for visual feature extraction, respectively.
Resulting representations are then concatenated and fed into the final model.
LXMERT [
FLAVA [18]. A foundation VLM including an image We first verify the performances of VLMs in plausibility encoder, a textual encoder, and a multimodal en- recognition via accuracy. Results of all models on the coder. It is jointly pre-trained on both unimodal datasets are reported in Table 1. and multimodal data, thus learning high-quality Both LMs and VLMs show significantly higher pervisual and textual representations. It is capable formances on − , where implauof achieving both crossmodal alignment and mul- sible sentences describe impossible events, than on timodal fusion objectives. − where implausible sentences deTo adapt multimodal models to the text-only task, we pict unlikely but not impossible events. On AN-AN sensimply modified the inputs, e.g. by feeding them empty tences, BERT, RoBERTa, VisualBERT, and FLAVA perimage tensors. FLAVA does not require to be adapted to formed above chance levels, while ViLT and LXMERT text-only inputs, as it can directly be evaluated by using performed at chance. This indicates that extracting inonly the textual encoder. formation about AN-AN sentence plausibility is gener
All models and their pre-trained weights are available ally challenging, and more so for VLMs. Among VLMs,
on Huggingface Transformers [
Going further, we consider EventsAdapt and we pro3.3. Evaluation procedure vide the density plot of PLLs divided by plausibility for To evaluate the ability of a LM (or VLM) to distinguish each model and human raters in Figure 2, and plot the between plausible and implausible sentences, we first correlation between PLLs of plausible and implausible have to compute a plausibility score for each sentence. sentences in Figure 1.
Since we are dealing with bi-directional masked language Both LMs and VLMs do not clearly distinguish between
models, we can approximate this plausibility score via the two classes and exhibit very similar distributions for
pseudo-log-likelihood (PLL), defined as the sum of loga- plausible and implausible sentences. The complexity of
rithmic probabilities of each token based on the remain- the task afect the results as well: for tasks where
huing tokens in the sentence [
First, we evaluated the models using an accuracy met- sentences (Fig. 1b). This is also clearly shown by the denric. Specifically, considering all (, ) sentence pairs sity distribution plot in Figure 2. for the AN-IN case, the for a dataset, we computed accuracy as the percentage density distribution for humans show a clear separation, of cases for which () > (). while models show more modest but still evident signs of
To provide a more detailed analysis of the perfor- separation. The density distribution for AN-AN sentences mances, we further evaluate the models via distribution shows a less separated distribution for human scores and analyses. We used the Pearson correlation coeficient almost entirely overlapped distributions for models. One between each model’s score for the plausible and implau- possible reason for this is that the grammaticality of a sible sentences. More in detail, for each pair of (, ), sentence depends on syntactic rules that can be more we plot the correlation between normalized () easily detected through statistical inference. In contrast, and (). High correlation implies similar scores linguistic acceptability may depend on extralinguistic information requiring multiple inference levels.
The role of concreteness. The experiments show that is higher for VLMs than for LMs (0.06 for LMs, 0.09
VLMs do not show improved abilities to deal with GEK for VLMs); when considering the simpler sentences of
and event plausibility with respect to textual LMs. How- − , the diferences are less marked.
ever, we could expect that this might also depend on On the other hand, multimodal models demonstrate
excelthe event concreteness, as concrete concepts are more lent recognition of abstract events in the DTFit dataset.
directly grounded on visual information than abstract Note however that abstract sentences are an order of
ones. The concreteness of an event depends on the the magnitude less than concrete ones in the dataset.
predicate itself, as well on its arguments. For instance, We also show a comparison of Pearson correlation
the verb to fight has a concrete use in the sentence The scores of results between − and
wrestler fought the opponent and an abstract use in The , shown respectively in Figures 3a and 3b.
patient fought the cancer. Cognitive research has shown While VLMs exhibit high correlation values, i.e. less
that abstract concepts require more linguistic experiences prowess on the task, values for DTFit are generally
to be understood [
To evaluate how concreteness afects the models’ abil- tween sentences with the same words but diferent orities, we first compute the accuracy on concrete and ab- der. This would explain their worse performances on stract subsets of the dataset. Results are reported in Ta- − . ble 2. Multimodal models seem to perform worse on abstract sentences with a higher degree of complexity: The impact of images Finally, we analyze whether on the − dataset, the average per- including images of the (im)plausible test events in the formance gap between abstract and concrete sentences input is beneficial for VLMs. We provide accuracy scores
ysis. First, VLMs do not achieve significantly higher accuracy values than unimodal ones in a semantic plausibility recognition task. Second, we saw that performances of VLMs is worse when dealing with more challenging senfor VLMs on the EventsRev dataset in Table 3. Including tences represented by − , exhibiting event images does not lead to any improvement: perfor- lower accuracy and a high correlation between plausible mances either remain the same or slightly degrade. and implausible sentences. Third, we saw that including images of events in the input does not lead to improved
0.76 0.66 0.76 0.79 We discuss a possible interpretation of these findings + 0.61 0.66 0.71 0.79in the following. First, the generally high correlation between PLL scores for pairs of (, ) for VLMs suggest Table 3 that these models struggle to recognize semantic diferAccuracy of VLMs on EventsRev with ( + ) and without () ences, especially between sentences with diferent word images in the input. orders (e.g., with subject-patient inversion), and relationships between sentence components, like semantic roles.
This may be further indication that VLMs model
lan4.1. Discussion guage in a bag-of-words fashion [7, 8]. The pre-training
method used in masked language modelling for VLMs,
adding visual features to language models already
specialized on linguistic tasks, may also compromise learning
as suggested by [
(b) .
mantic errors. Moreover, models using pre-trained LM still behave similarly to Bag-of-Words models, regardless
weights for text processing may face limitations in the of the degree of concreteness of the events.
type of visual information they can capture during train- In the future, we plan to focus on the analysis of models
ing. Some models rely on object categories trained on with visual grounding as their training objective, such
bounding boxes. This is computationally expensive, and as PaLM-E [
These results are in line with [
A possible explanation of why VLMs do not benefit from including test images is that in this specific case Acknowledgments (minimal sentence pairs with subject-object inversion) the images for both sentences are very similar, and difer only for the relationship between the entities. The visual encoders of the models might be too weak to diferentiate substantially similar images, leading the models to rely on their LM priors and make random choices. Finally, we saw that even the foundation Large VLM we considered – FLAVA – does not show significantly improved accuracy compared to other VLMs.
University and Research (MUR) in the framework of the PON 2014-2021 “Research and Innovation" resources – Innovation Action - DM MUR 1062/2021 - Title of the Research: “Modelli semantici multimodali per l’industria 4.0 e le digital humanities.”, and by PNRR - M4C2 - Investimento 1.3, Partenariato Esteso PE00000013 - “FAIR Future Artificial Intelligence Research" - Spoke 1 “Humancentered AI", funded by the European Commission under the NextGeneration EU programme.
evaluating the ability of VLMs to model event plausibility in both language-only and vision-language tasks against LMs. We find that VL pre-training does not lead to a significant improvement compared to unimodal LMs in this task aiming at testing their GEK. Specifically, we observed that VLMs tend to perform worse when the implausible sentence has a higher semantic complexity, because it contains two animate nouns. Our analysis also brings further support to argument that VLMs models [3] P. Pedinotti, G. Rambelli, E. Chersoni, E. Santus, beddings from multi-modal data: Since you probaA. Lenci, P. Blache, Did the cat drink the cofee? bly can’t see what I mean, in: Proceedings of the challenging transformers with generalized event 2014 Conference on Empirical Methods in Natuknowledge, arXiv preprint arXiv:2107.10922 (2021). ral Language Processing (EMNLP), Association for [4] C. Kauf, A. A. Ivanova, G. Rambelli, E. Chersoni, Computational Linguistics, Doha, Qatar, 2014, pp.
J. S. She, Z. Chowdhury, E. Fedorenko, A. Lenci, 255–265. doi:10.3115/v1/D14-1032.
Event knowledge in large language models: the [16] R. Shekhar, S. Pezzelle, Y. Klimovich, A. Herbelot, gap between the impossible and the unlikely, 2022. M. Nabi, E. Sangineto, R. Bernardi, FOIL it! find doi:10.48550/ARXIV.2212.01488. one mismatch between image and language caption, [5] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, in: Proceedings of the 55th Annual Meeting of the L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, At- Association for Computational Linguistics (Volume tention is all you need, in: Proceedings of the 31st 1: Long Papers), Association for Computational International Conference on Neural Information Linguistics, Vancouver, Canada, 2017, pp. 255–265. Processing Systems, NIPS’17, Curran Associates doi:10.18653/v1/P17-1024.
Inc., Red Hook, NY, USA, 2017, p. 6000–6010. [17] L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, K.-W. Chang, [6] T. Yun, C. Sun, E. Pavlick, Does vision-and- Visualbert: A simple and performant baseline for vilanguage pretraining improve lexical grounding?, sion and language, arXiv preprint arXiv:1908.03557 in: Findings of the Association for Computational (2019).
Linguistics: EMNLP 2021, Association for Computa- [18] A. Singh, R. Hu, V. Goswami, G. Couairon, tional Linguistics, Punta Cana, Dominican Republic, W. Galuba, M. Rohrbach, D. Kiela, Flava: A founda2021, pp. 4357–4366. doi:10.18653/v1/2021.f tional language and vision alignment model, 2022 indings-emnlp.370. IEEE/CVF Conference on Computer Vision and Pat[7] S. Castro, O. Ignat, R. Mihalcea, Scalable perfor- tern Recognition (CVPR) (2021) 15617–15629. mance analysis for vision-language models, in: [19] Z. Gan, L. Li, C. Li, L. Wang, Z. Liu, J. Gao, et al., Proceedings of the The 12th Joint Conference on Vision-language pre-training: Basics, recent adLexical and Computational Semantics (*SEM 2023), vances, and future trends, Foundations and Trends® Association for Computational Linguistics, Toronto, in Computer Graphics and Vision 14 (2022) 163– Canada, 2023, pp. 284–294. 352. [8] T. Thrush, R. Jiang, M. Bartolo, A. Singh, [20] T. Iki, A. Aizawa, Efect of visual extensions on natuA. Williams, D. Kiela, C. Ross, Winoground: Prob- ral language understanding in vision-and-language ing vision and language models for visio-linguistic models, in: Proceedings of the 2021 Conference compositionality, 2022. arXiv:2204.03162. on Empirical Methods in Natural Language Pro[9] J. Gordon, B. Van Durme, Reporting bias and cessing, Association for Computational Linguistics, knowledge acquisition, in: Proceedings of the 2013 Online and Punta Cana, Dominican Republic, 2021, Workshop on Automated Knowledge Base Con- pp. 2189–2196. doi:10.18653/v1/2021.emnlp struction, AKBC ’13, Association for Computing -main.167.
Machinery, New York, NY, USA, 2013, p. 25–30. [21] P. Vassallo, E. Chersoni, E. Santus, A. Lenci, doi:10.1145/2509558.2509563. P. Blache, Event Knowledge in Sentence Process[10] A. Lenci, M. Sahlgren, Distributional Semantics, ing: A New Dataset for the Evaluation of Argument
Cambridge University Press, Cambridge, 2023. Typicality, in: LREC 2018 Workshop on Linguistic [11] D. Marconi, Lexical Competence, The MIT Press, and Neurocognitive Resources (LiNCR), Miyazaki,
Cambridge, MA, 1997. Japan, 2018. [12] S. Harnad, The symbol grounding problem, Physica [22] E. Fedorenko, I. A. Blank, M. Siegelman, Z. Minerof, D: Nonlinear Phenomena 42 (1990) 335–346. doi:ht Lack of selectivity for syntax relative to word meantps://doi.org/10.1016/0167-2789(90)9 ings throughout the language network, Cognition 0087-6. 203 (2020) 104348. doi:https://doi.org/10.1 [13] E. M. Bender, A. Koller, Climbing towards nlu: On 016/j.cognition.2020.104348.
meaning, form, and understanding in the age of [23] A. A. Ivanova, Z. Minerof, V. Zimmerer, N. Kandata, in: Proc. ACL, Seattle, WA, 2020, pp. 5185– wisher, R. Varley, E. Fedorenko, The Language 5198. Network Is Recruited but Not Required for Nonver[14] E. Bruni, G. Boleda, M. Baroni, N.-K. Tran, Dis- bal Event Semantics, Neurobiology of Language tributional semantics in technicolor, Proceedings 2 (2021) 176–201. doi:10.1162/nol_a_00030. of the 50th Annual Meeting of the Association for arXiv:https://direct.mit.edu/nol/article-pdf/2/2/176/18 Computational Linguistics (2012) 136–145. [24] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: [15] F. Hill, A. Korhonen, Learning abstract concept em- Pre-training of deep bidirectional transformers for