The exponential growth of digital imagery has profoundly influenced various fields, ranging from social media and entertainment to scientific research and healthcare. The importance and proliferation of visual media will only accelerate as a form of efficient communication, hence the phrase "a picture is worth a thousand words". Touché offers a competition on selecting the most relevant images from a crawled corpus for a set of arguments. Therefore we attempted to enter in this touche task to improve on solutions for retrieving images related to arguments. We wanted our solutions to only focus on images and descriptions of images, to try and not use any webpage text. Our solution approaches focused around combining image descriptions and the image itself as a comprehensive unit and then for our other submission implementing one more step on top of that to add a comparison to generated images that used the arguments themselves as prompts for the generated images.
The defining paper for retrieving images for arguments is by Kiesel [1]. In it they use natural language processing techniques found in the web text of surrounding these images to create expanded keyword searches in the web text to attempt to track the stance of an argument. Last year at Touché 2023 team Picard [2] constructed a similar solution -one of their submissions involved image generation using stable diffusion.The authors prompt the image generation with the arguments from the competition to create a benchmark image that is used to compare the competition images searching for the most similar using CLIP.
CLIP [3] stands for Contrastive Language Image Pretraining. It is a model developed by OpenAI that embeds images or texts into the same vector space by training on images with their corresponding captions. It is helpful to reduce the dimensionality of text or images, but still be semantically similar to retrieve relevant results from one modality to the other.
Stable Diffusion [4] is a neural network model that is capable of producing images given a prompt. By decomposing the image formation process into a sequential application of denoising autoencoders stable diffusion can achieve state-of-the-art synthesis results on image data.
Both submissions made use of CLIP from OpenAI. The competition supplied the images and their corresponding image descriptions obtained from LLava. These modalities were embedded using CLIP into a 70-30 ratio of Image to text. These embeddings were stored in a chromaDB vector database for later retrieval.
Provided arguments (only the arguments no premises or claims used) were used as queries to be brushed against the vector database. The arguments were embedded using CLIP to keep the same dimensionality as the combined image-text embeddings of the images. These arguments were compared pairwise across each image in the database via cosine similarity keeping the top 10 for our initial submission.
For the image generation submission for each topic images were generated off a set of tinyLLama generated supporting/detracting arguments depending on the stance. For example if the stance was pro the prompt would instruct to provide supporting claims if it was anti then it would prompt with claims to detract. The number of arguments generated would vary from 3-7. The prompt format for a supporting generation is found in figure 1.
{ " r o l e " : " s y s t e m " , " c o n t e n t " : " You a r e a s t u d e n t t r a i n e d t o t h i n k c r i t i c a l l y f o r each c l a i m b r e a k i t down i n t o s e v e r a l s u b c l a i m s " , } , { " r o l e " : " u s e r " , " c o n t e n t " : f " C r e a t e some numbered prompts t o g i v e t o a machine t o c r e a t e i m a g e s t h a t s u p p o r t t h e c l a i m : ' { prompt } ' " } These tinyLLama-generated supporting/detracting arguments were then fed into the stable-diffusion-2-1-base for image generation. These generated images are again embedded with CLIP and compared to the top 40 retrieved images using the method described in the prior section for a given argument. Because there was a varying number of images generated for each argument when comparing the crawled images to the generated images we take the highest average score across all generated images as our metric for most relevant images.
Our approaches didn't beat baseline of BM25 and SBERT. We do see that the added filter of comparing the top results to images generated from the arguments does increase the accuracy of the model. It appears that Images alone
The image generated approach worked best, however both submissions didn't beat baseline. Future directions of work include re-ranking LLava Visual Question Answering generations -i.e. ask LLava to describe the picture in relevance to argument in question. Utilizing BM25 and webpage text to decipher keywords that might indicate the relevance of the image.
The Data Science at Georgia Tech Club.