In recent years, the conspicuous development of vision language models (VLMs) across various tasks is evident [1,2,3]. However, VLMs have been criticized for performing akin to bag-of-words models, lacking semantic understanding, especially compositional understanding [4,3,5]. For instance, when some tokens in the caption of an image-caption pair are rearranged to result in an unaligned caption, a VLM may fail to notice the change. Consider the two image-caption pairs in Figure 1. In the left side pair, the phrases "Three Jack-O-Lanterns" and "flowers" in its caption are swapped, resulting in a semantically very different sentence. But CLIP fails to notice the difference and somehow gives the modified caption a slightly higher similarity score. A similar effect can be seen in the right side image-caption pair, when the phrases "Clock tower" and "a bronze statue" in its caption are swapped. These are not isolated examples. As Yuksekgonul et al. [5] pointed out, VLMs "behave like bags-of-words" because they have been mostly pretrained on large-scale web datasets for retrieval tasks where image and caption matching can often be done using keywords alone.
A straightforward and effective solution involves mining hard negative samples for contrastive learning. This entails including negative instances with similar semantic components but distinct relationships in the same KiL'24: Workshop on Knowledge-infused Learning co-located with 30th ACM KDD Conference, August 26, 2024, Barcelona, Spain Envelope zshou@cse.ust.hk (Z. Shou); flin@cse.ust.hk (F. Lin) batch, challenging the model to discern the correct caption amidst such variations. For example, NegCLIP [5] constructs negative image captions by swapping tokens. However, token swap methods lack semantic understanding, resulting in patterns, and lack of plausibility and fluency. Blind Models trained solely on text, without considering images, may manipulate evaluations to their advantage [6].
Meaning representations offer an alternative approach to constructing negative samples with greater diversity and fluency. Abstract Meaning Representation (AMR, [7]) stands out as a prevalent semantic representation in text tasks and is valued for its high expressiveness and human-friendly comprehension, which encodes concepts as nodes and depicts the relationships between concepts through graphical representations. We propose to utilize AMR to create negative samples that possess entirely distinct meanings but share close plain paraphrases. To achieve this, we modify the structure of meaning representation by randomly shuffling the positions of subtrees within AMR graphs and reconstructing meaning representations. Following this process, negative captions are generated from the new meaning representations using an AMR generator. We blend our generated negatives with token swap negatives to broaden the diversity of negative samples and enhance generalization. Subsequently, vision language models undergo training to distinguish between true labels and negative samples.
Our findings indicate that incorporating negative samples generated from meaning representations improves model performance across diverse compositional understanding benchmarks. Additionally, our generated nega- tives introduce various patterns, enriching the diversity of augmentations compared to token swap negatives.
AMR encodes concepts as nodes and illustrates the relationships between these concepts as edges. It has been shown to be advantageous in various natural language processing tasks, such as data augmentation. Token edit data augmentations in NLP often result in generating illformed or incoherent sentences, as they do not consider sentence structures. AMR Data Augmentation (AMR-DA) [8] suggests utilizing AMR for data augmentation. They construct positive samples by meticulously controlling minor nuances within a carefully designed framework for meaning representation. Consequently, they produce several fluent and distinct positive augmentations for the given sentences. Inspired by AMR-DA, we explore the utilization of AMR in compositional understanding tasks for vision language models. However, our approach diverges significantly; rather than focusing on careful modifications to meaning representation for positive sample generation, we propose employing AMR for negative sample generation. Our methodology involves splitting the meaning representation and shuffling its components to construct a new negative representation.
For generating negative captions for contrastive learning, a straightforward approach involves modifying linguistic elements. To improve compositional understanding, [5] leverage Spacy for syntactic analysis to identify and swap the positions of two elements within the caption. The token swap modifications aimed at creating variations in composition are relatively straightforward to implement but often struggle to maintain grammaticality. Moreover, they can be vulnerable to exploitation, as the patterns of modification may become predictable even without considering information from the image encoder. [9] initially parse the syntactic structure of the caption. They then randomly mask text and utilize a large language model to unmask and generate a new negative caption. While the resulting caption tends to exhibit improved grammatical correctness, the modification process lacks fine control, and the generated variants remain somewhat constrained in scope. To address the limitations of semantic modification, [10] proposes leveraging scene graphs to generate semantic negative captions. They implement a strategy where they interchange the positions of the subject and object within the same relation, as well as swap the attributes of different objects. However, the modification of scene graphs is limited. Compared to scene graphs, meaning representations encode a more extensive range of relations, especially higher-level abstract semantic relations absent in scene graphs [11]. This suggests that meaning representations have a higher potential to improve downstream tasks that require an understanding of higher-level semantic information in images.
The aim of contrastive learning is to bring similar representations into closer proximity while simultaneously pushing apart dissimilar samples. This principle mirrors its application within vision language model training, exemplified by Contrastive Language-Image Pre-Training (CLIP, [1]), which has emerged as a prominent paradigm in vision language learning. The training objective of CLIP is to align text-image pairs effectively. CLIP simultaneously trains an image encoder and a text encoder to extract feature representations from each modality, denoted as πΌ π for image features and π π for text features. These features are then utilized to compute scaled pairwise cosine
A small child wearing headphones plays on the computer.
A young professional is working at his laptop while his coworker is reading material.
Children's headphones are small enough to wear while using the computer to play.
When I was a young reader , my professional work was on a laptop with a co -worker .
Negative Captions for Original Images Captions for Original Images
Negative Captions for Hard Images similarities, serving as logits. Finally, a symmetric crossentropy loss is computed over these similarity scores to guide the training process effectively.
In response to the challenge of vision language models struggling to comprehend text composition, we adopt the approach proposed by Yuksekgonul et al. [5], which introduced two extensive components to standard contrastive learning, aimed at increasing the complexity of model learning. This entails (1) introducing challenging images for the image encoder to extract features from, selected based on CLIP encoding and utilizing nearest neighbors of original images, and (2) incorporating hard negative captions for the text encoder to distinguish features. The difference is that we add AMR generated negative samples into hard negative captions, with modifications aimed at preserving most plain text tokens while completely distorting the semantic meaning. Figure 2 illustrates the training pipeline. In each batch, original images πΌ π and their nearest neighbors π πΌ π are included. Corresponding captions π π and π π π are concatenated with hard negative captions π β π and π π β π , doubling the length of captions compared to the number of images. Subsequently, a symmetric cross-entropy loss is computed as in CLIP. However, only column-wise loss for positive captions is incorporated, as negative captions lack corresponding images for comparison.
Contrary to token swap negative generation, we propose to the generation of negative samples using AMR. AMR encodes the semantics into graphs and has demonstrated effectiveness as an intermediate representation in natural language augmentation tasks. We adopt a similar pipeline to AMR-DA [8]: parsing sentences into AMR, modifying the AMR, and generating samples from the modified AMR. However, our objective differs significantly from that of AMR-DA. While they meticulously modify the intermediate AMR to construct positive samples, our task requires generating entirely different semantic representations, albeit with the same semantic components as given samples.
Abstract Meaning Representation (AMR, [7]) is a rooted, directed graph that encodes sentence concepts as nodes and the relations between these concepts as directed edges. In Figure 3, the leftmost portion depicts the AMR graph corresponding to the caption "A trunk carries a large amount of items and a few people. " In this graph, the root "carry" serves as the primary predicate of the sentence, with "trunk" designated as the first argument (denoted as ARG0) of "carry, " while the subtree originating from "and" represents the second argument. AMR A truck carries a large amount of items and a few people.
The items are carried by a few large trucks and an amount of people . facilitates readability for both human and machine comprehension and can be adapted to various purposes as needed. In this study, our proposal involves splitting the AMR graph, shuffling its components, and then reconstructing a new AMR graph. This process aims to create a hard negative graph where all semantic parts are retained, but the overall meaning is distorted.
The entire pipeline is illustrated in Figure 3. We adopt AMR-DA pipeline, which involves initially parsing the caption into an AMR graph using an AMR parser. Subsequently, we modify this AMR graph and finally utilize an AMR generator to produce negative captions based on the modified AMR. We utilize SPRING parser [12] as our AMR parser. SPRING employs a depth-first search method to linearize AMRs and utilizes a special token < π π > to manage co-referring nodes. The parser is trained based on BART model [13]. After obtaining the AMR graph for the caption, we propose a split and reconstruct algorithm to construct a new AMR graph, which is described in detail in the subsequent paragraphs. Finally, we employ PLMs-Generator [14] based on T5-base as our AMR generator to convert AMR to text. The modelbased generator exhibits tolerance, allowing for the accommodation of certain unreasonable aspects within our modified graph. AMR generator can rectify to some ex-tent and produce new samples closely resembling the given graph, this flexibility provides greater latitude for modifying the AMR graph compared to rule-based methods. For instance, in Figure 3, although the modified graph contains some illogical elements such as the node "and" lacking children, the generator is still capable of generating fluent and grammatically correct text.
The key component of generating negative samples through AMR lies in our split and reconstruct algorithm.
Unlike existing methods that rely on token swapping within the sentence or node swapping in the scene graph based on predefined rules, our approach offers greater flexibility by directly modifying the entire meaning representations. Modifications to AMR afford a broader range of possibilities owing to the diverse types of edges and nodes present.
In our algorithm, we split the AMR graph and regard the root node as a separate entity, while treating other nodes along with their incoming edges as edge-node pairs. As illustrated in Figure 3, the left-hand side depicts the AMR graph corresponding to the original caption "A trunk carries a large amount of items and a few people. " Following the split process, we obtain a root node and a collection of edge-node pairs such as "carry, [(:ARG0, trunk), (:ARG1, and), ...]". Next, we proceed to reconstruct a semantic tree by randomly concatenating nodes from the split parts. We shuffle the list of edge-node pairs and sequentially select edge-node pairs one by one. The process begins at layer 1 with the root node. At this stage, the first node has only one option, which is to connect to the root node and move to layer 2. Subsequently, at layer 2, the subsequent nodes have two options: either to remain at layer 2 by connecting to the root node, or to move to a deeper layer by connecting to the previous node at layer 2. If a node moves to a deeper layer, for instance, layer 3, the subsequent node has three options: to remain at the current layer, to move deeper, or to move back to the previous layer. This iterative process continues until all nodes are connected within the semantic tree. In Figure 3, when considering the pair (:mod, large), there are indeed three options available. The node "large" can either remain at the current layer by connecting to the node "trunk", proceed to a deeper layer by connecting to the node "few", or revert back to connect with the root node. The shuffled AMR entails reordering all nodes along with their edges except the root node, resulting in a new representation of meaning. Negative captions are then generated based on this shuffled AMR. The algorithm to reconstruct AMR graph is illustrated in Algorithm 1.
The distinction between negative AMR generation and AMR-DA lies in their respective objectives. AMR-DA aims to regulate modifications to avoid distorting the overall semantic meaning of the sentence by selectively adding or removing nuanced semantic components. On the other hand, negative AMR generation focuses on retaining the majority of the semantic components while generating entirely different semantic representations.
We conduct experiments on different evaluation datasets to explore the impact of AMR generated negatives on the performance of vision language models in compositional understanding tasks.
We explore whether AMR generated negatives improve the performance of model compositional understanding, so we follow the training setups in NegCLIP [5], which finetune CLIP based on the ViT-B/321 on the COCO dataset with token swap hard negatives.
For negative captions, we assign a specific probability to replace the original token swap caption with AMR generated negative augmentation. In the main results, the possibility of replacing negatives in NegCLIP is set at 30%. In other words, about 30% of the captions with our AMR generated hard negative captions, while the NegCLIP initially sets the batch size to 1024. However, due to device limitations, we are constrained to train the model on a single NVIDIA RTX 2080 Ti GPU, reducing our batch size to 32. Consequently, we adjust the warm-up steps to 1600. Contrastive learning relies on batch size, as it involves contrasting samples within each batch. Therefore, larger batch sizes are anticipated to yield greater improvements. We employ the AdamW optimizer with a cosine annealing schedule for a training epoch of 5. The learning rate is explored within the range of 1e-5, 5e-6, 1e-6, with reported results utilizing a learning rate of 5e-6.
We assess the efficacy of our approach on two widely used benchmarks for compositional understanding: ARO [5] and SugarCrepe [6]. ARO stands for Attribution, Relation, and Order, including four tasks: Visual Genome Relation (VG-Relation) and Visual Genome Attribution (VG-Attribution) tasks entail selecting the correct caption from two options, where negative captions alter either the object of the relation or the object's attribution. Flickr30k Order and COCO Order tasks demand models to accurately identify the order of captions from five options, where negative captions modify the order of tokens within the caption. SugarCrepe aims to address the issue of negative captions being implausible and non-fluent by employing large language models to generate fluent and challenging negative captions. The dataset encompasses three tasks: Replace, Swap, and Add, which entail various actions aimed at evaluating models' compositional understanding.
We incorporate AMR generated negative samples into our contrastive training data, simplifying our method to AMR-NegCLIP. In this study, we undertake a comparative analysis of the outcomes generated by our AMR-NegCLIP approach in contrast to the results produced by several baseline models, ViT-B-32, standard CLIP finetuned with COCO dataset (CLIP), and CLIP finetuned with tokenlevel hard negatives (NegCLIP).
From Table 1, we can find that our AMR-NegCLIP achieves superior performance across all subtasks. In Visual Gnome dataset, AMR-NegCLIP gets a 2.2% improvement in Relation task over NegCLIP and a 4.6% improvement in Attribution task. In Flickr30k Order dataset, there is a 2.9% improvement compared to Neg-CLIP and a substantial 34.4% improvement over CLIP. In the COCO Order dataset, there is a 5.6% improvement over NegCLIP and an impressive 45.6% improvement over CLIP. In Replace and Add tasks within SugarCrepe, AMR-NegCLIP exhibits limited improvements when contrasted with NegCLIP, with 1.0% in Replace task and 0.2% improvement in Add task. This discrepancy can be attributed to the nature of the Replace and Add tasks, which involve modifying concepts within the caption. AMR-NegCLIP generates negatives that maintain the same concepts as the positive caption, thereby not entirely aligning with the task requirements. In contrast, another notable observation is a significant improvement, 5.9% over NegCLIP, in the Swap task of SugarCrepe, a challenge that proves to be particularly daunting for pretrained CLIP models, as highlighted in the SugarCrepe paper [6]. In their study, SugarCrepe authors evaluate over ten vision language models and note that "all models struggle at identifying SWAP hard negatives, regardless of their pertaining dataset and model size.". This difficulty arises from the nature of the swap action in SugarCrepe, which involves neither adding nor excluding any concepts but rather swapping objects or attributes while maintaining fluency and grammatical correctness, a task demanding a deeper understanding of composition from vision language models. This closely aligns with our motivation to employ meaning representations in the
Example evaluation data of Visual Genome Relation, Flickr30k Order in ARO; Replace, Swap and Add in SugarCrepe. The italicized text represents a positive caption for the sample, while the other lines contain negative captions. Visual Genome includes two captions per sample, whereas Order test set includes five captions per sample.
the door is to the left of the shirt. the shirt is to the left of the door.
A group of people standing on the lawn in front of a building. Many people in blue jeans stand in front of a white church.
A large group of people stand outside of a church. Family members standing outside a home. People standing outside of a building.
A tan toilet and sink combination in a small room.
A white toilet and sink combination in a small room.
Three large horses eating hay while a small horse stands behind.
A small horse eating hay while three horses stand behind.
Two zebras are battling each other on hind legs. Two striped-and-spotted zebras are battling each other on hind legs.
Negative Sentences generated using Random Token Swap, Scene Graph Node Swap and AMR Reconstruction.
A truck carries a large amount of items and a few people.
A amount carries a large truck of items and a few people .
A truck carries a few amount of items and a large people.
The items are carried by a few large trucks and an amount of people .
A pigeon greets three bicyclists on a park path.
A park greets three bicyclists on a pigeon path .
A bicyclist greets three pigeon on a park path.
Greetings , three pigeon bicyclers on the path have been parkled .
People walking pass a horse drawn carriage sitting at the curb.
People walking pass a horse drawn curb sitting at the carriage.
People sitting at a horse drawn carriage walking pass the curb.
People walking by the curb , horse sitting , carriage pulling .
negative generation. Example evaluation data for ARO and SugarCrepe are provided in Table 2.
In Order evaluation dataset, negative samples exhibit greater diversity. The introduction of Swap in Sugar-Crepe aims to rectify instances of textual non-fluency and implausibility, thereby rendering it more resilient against potential hacking attempts from blind models.
In conclusion, the results indicate that integrating AMR generated negative captions significantly improves VLM's performance on various composition tasks, especially dealing with high-level compositional understanding captions.
Understanding the meaning of images has long been a goal. Scene graphs have emerged as a popular method for encoding objects, their attributes, and relationships within graphs. Abdelsalam et al.'s work [11] discusses the difference between AMR and Scene Graphs through detailed statistical analysis on entity and relation categorization. Their conclusion highlights that AMR encodes a broader range of relationships, particularly abstract semantic relationships absent in scene graphs.
Some studies have also explored leveraging scene graphs to construct negative samples, particularly focusing on token swapping, such as swapping asymmetric relations [15,10,5]. These methods have produced limited variants. However, our approach addresses the entire semantic representation rather than specific token swaps. To analyze the difference between outputs, we present the generated negative samples from Random Token Swap, Scene Graph Node Swap, and AMR Reconstruction in Table 3.
In contrast to Random Token Swap approach, leveraging scene graphs yields a richer array of syntactic and semantic cues. However, the generated negatives adhere to rule-based criteria, such as swapping exclusively between adjective words or words sharing a common relational structure. It is evident that AMR Reconstruction introduces a wider spectrum of variations to the original captions, all while upholding the core semantic components. Our methodology thus offers enhanced flexibility in generating negative training data. Furthermore, we compare AMR-NegCLIP with other negative augmentation-based methods, Semantic Negative [10], which constructs negative samples using scene graph node swaps, and CLIP-SGVL [15], which utilizes scene graphs in multiple ways, including positive and negative caption generation, as well as scene graph prediction tasks, in Table 4. However, the training and validation data sets of Semantic Negative are different from ours, but it can also be seen that it is challenging to improve the accuracy of both relationships and attributes by changing the negative samples. The findings indicate that AMR-NegCLIP achieves superior average performance in comparison to the Semantic Negative method. This observation underscores the efficacy of employing AMR generated negatives, which manifest more pronounced enhancements when compared to the strategy of swapping scene graph nodes. Negative sample generation rules in CLIP-SGVL are similar to those of Semantic Negative. Our AMR-NegCLIP demonstrated superior performance in Order tasks with more variants.
We present several case studies illustrating the results of CLIP and AMR-NegCLIP across four subtasks in Sugar-Crepe, as depicted in Figure 4. SugarCrepe utilizes large language models to generate captions with a high degree of fluency and commonsense understanding, thereby posing a challenge for VLMs to discern negative captions effectively. For instance, in Swap Object task, VLMs must comprehend the semantics of relationships such as "in" and "background", as well as discern the object and subject of these relationships. Our test results demonstrate that while CLIP exhibits closely aligned similarity scores between captions and negative captions, AMR-NegCLIP demonstrates superior discriminatory capability. Furthermore, in Swap Attribution task, models are required to accurately identify quantities and the position of corresponding objects to succeed. CLIP returns nearly identi-
AMR generated negative samples tend to distort entire semantic representations of given captions, while NegCLIP swaps the positions of tokens. Their generated negative samples address varying levels, from individual objects to complete semantics. To ensure augmented data spans different levels in the training dataset, we retain parts of negative samples from NegCLIP while replacing a ratio of NegCLIP samples with AMR generated negative samples.
To assess the impact of AMR generated negative samples on model performance, we replace NegCLIP negatives at ratios ranging from 10% to 100%, and present the results in Table 5. When replacing only 10% of NegCLIP negatives with AMR generated negative samples, the model performance exhibits noticeable improvements, particularly 6.1% in COCO Order subtasks. The best performance is achieved when 30% of the token swap negatives are replaced by AMR-generated negatives. Across replacement ratios ranging from 10% to 60%, the integration of AMR generated negatives yields improvements for NegCLIP across all subtasks. These enhancements are consistently observed, with average performance gains ranging from 1.6% to 3.8%. Beyond a 70% replacement ratio, larger ratios result in decreased model performance. Specifically, when 90% and 100% of negative samples are AMR generated, the performance is inferior to that of token swap negatives but still superior to CLIP. The reason for this phenomenon could be attributed to the greater diversity of AMR generated negatives compared to token swap negatives. Unlike token swap negatives, which follow a unified pattern, AMR generated negatives lack such consistency, making it challenging for models to effectively learn from them, particularly when the replacement ratio is high. Therefore, we propose that our AMR generated negative captions can effectively complement token swap generations.
To overcome the limitations of vision language models in comprehending composition and semantics, we suggest constructing hard negative samples through splitting and reconstructing AMR graphs. Compared to token and scene graph negative generation, AMR generated negatives have greater diversity and keep the fluency at the most possible. Compared to token and scene graph negative generation, AMR generated negatives exhibit greater diversity while maintaining optimal fluency. Our experimental results illustrate that incorporating our generated negatives in contrastive learning significantly boosts model performance, particularly in tasks that demand high-level comprehension. Furthermore, beyond simple shuffling, AMR offers the potential for more controlled modifications based on human instructions. For instance, users could add semantic components that are absent in the picture to deliberately confuse VLMs. We view this as a promising avenue for future research.
Limitaions Conducting AMR parsing and generation typically requires GPU acceleration, which incurs higher costs compared to direct token shuffling methods. However, when compared to tasks such as scene graph parsing or querying large language models, it remains an efficient approach. It's worth noting that splitting and shuffling AMR components introduce significant randomness in negative generation, and occasionally, this may lead to suboptimal results.