Models as Knowledge Bases for Visual Word Sense Disambiguation

Abstract

Visual Word Sense Disambiguation (VWSD) is a novel challenging task that lies between linguistic sense disambiguation and fine-grained multimodal retrieval. The recent advancements in the development of visiolinguistic (VL) transformers suggest some of-the-self implementations with encouraging results, which however we argue that can be further improved. To this end, we propose some knowledgeenhancement techniques towards improving the retrieval performance of VL transformers via the usage of Large Language Models (LLMs) as Knowledge Bases. More specifically, knowledge stored in LLMs is retrieved with the help of appropriate prompts in a zero-shot manner, achieving performance advancements. Moreover, we convert VWSD to a purely textual question-answering (QA) problem by considering generated image captions as multiple-choice candidate answers. Zero-shot and few-shot prompting strategies are leveraged to explore the potential of such a transformation, while Chain-ofThought (CoT) prompting in the zero-shot setting is able to reveal the internal reasoning steps an LLM follows to select the appropriate candidate. In total, our presented approach is the first one to analyze the merits of exploiting knowledge stored in LLMs in diferent ways to solve WVSD.

-

CEUR ceur-ws.org

1. Introduction

Visual Word Sense Disambiguation (VWSD) is a newly introduced task, where the most appropriate candidate image has to be retrieved given an ambiguous target word within a specific context [ 1 ]. For example, given the ambiguous target word ”andromeda” within the context ”tree”, the phrase ”andromeda tree” is formulated. The word ”andromeda” itself may have diferent meanings, referring to a constellation, flower, reptile or fish, and this ambiguity is mitigated via the given context ”tree”. However, the presence of other tree-related candidate images complicates the choice of the golden ”andromeda tree” image.

VWSD can be viewed as a multimodal extension of textual word sense disambiguation, which has received some considerable attention in recent years [ 2 ]. Resolving polysemy has been approached in several ways, with one of the most prominent being knowledge-based (G. Stamou) disambiguation. According to it, knowledge graphs representing linguistic relationships such as WordNet [ 3 ] for English and BabelNet [ 4 ] for multilingual disambiguation are exploited to frame the meaning of an ambiguous word based on its context, alleviating the need for training during the disambiguation process. More specifically, both WordNet and BabelNet knowledge graphs utilize synsets, i.e. sets of synonyms which express diferent meanings within diferent contexts; hence, this structure is inherently related to the problem of sense disambiguation, since several possible meanings are already explicitly encoded within each knowledge graph.

Nevertheless, given the ever-increasing amounts of linguistic information that need to be stored in a knowledge base, structures such as WordNet and BabelNet naturally face scalability limitations: novel words, contexts or languages demand high-quality manual engineering of the related knowledge graphs to accommodate tasks such as word sense disambiguation in a viable way. For example, given the example phrase ”andromeda tree”, a knowledge base lacking the target word ”andromeda” or the context ”tree” (with respect to ”andromeda”) would inevitably lead to disambiguation failure. To this end, the introduction of the Language Models as Knowledge Bases (LM-as-KB) paradigm [ 5 ] opened a whole new world of possibilities for knowledge representation. The expressive power of language models allows the extraction of rich relational knowledge, similar to knowledge obtained when querying a structured knowledge graph. Despite recent research advancements, the real-world utilization of the LM-as-KB paradigm is still in its infancy, with several challenges and limitations arising [ 6 ]. These challenges interrogate the adoption of the LM-as-KB framework in the visiolinguistic (VL) setting [ 7 ], even though it is already favored in some tasks, such as Visual Question Answering [ 8, 9, 10 ], Visual Commonsense Reasoning [ 11, 12 ] and Image Captioning [ 13 ].

In this work, we aim to explore the potential of following the LM-as-KB paradigm towards resolving VWSD challenges. Specifically, we focus our contributions on two main directions: • Ambiguous phrases are enriched with knowledge stored in Large Language Models (LLMs) in a zero-shot fashion to provide more context for multimodal retrieval. • We explore the conversion of VWSD to textual question-answering (QA) upon which we apply zero-shot and few-shot prompting strategies. We further investigate the usage of Chain-of-Thought (CoT) prompting to obtain explanations for LLM-based decisions.

2. Related work

Language models as Knowledge Bases is a novel paradigm that harnesses the implicit knowledge stored in neural weights of Language Models (LMs) similar to how explicit knowledge of Knowledge Graphs (KGs) has served related applications [ 5, 6 ]. Recent works probe characteristics of diferent LLM-stored knowledge senses, such as commonsense [ 14 ], factual [ 15 ] and temporal [ 16 ]. Model scale has revealed unprecedented LM capabilities related to knowledge, such as diferent types of reasoning; there are open research questions related to arithmetic [ 17 ], symbolic [ 18 ], commonsense [ 19 ] and causal reasoning [ 20 ] in large LMs, and whether such models merely overfit large amounts of data or are genuinely capable of reasoning similar to humans [ 21 ]. More recently, the LM-as-KB paradigm has been favored by the VL community to enhance popular VL tasks [ 7, 8, 9, 10, 11, 12 ], as in the case of the current paper. Prompting Language Models Traditionally, LMs are pre-trained on large amounts of text corpora and then fine-tuned on smaller labelled ones to address longstanding challenges in NLP. Scaling-up to several billion parameters (Large Language Models - LLMs) enables emergent model capabilities, which can be communicated via prompting [ 22 ]. Diferent prompting strategies have been proposed, such as zero-shot prompting, where the task is explicitly described in natural language, and few-shot prompting, where few exemplars are provided to the LM to guide answer generation [ 23 ]. More specifically, in-context learning (ICL) is a novel fewshot paradigm where few sample demonstrations from the dataset are used to retrieve related knowledge stored in the LLM without updating any parameters [ 24 ]. This technique is inspired from the way humans learn from analogy [ 25 ] and has been successfully applied ever since to serve several NLP tasks [ 26, 27, 28, 29, 30 ]. Another human-inspired prompting technique is Chain-of-Thought (CoT), where the LM is encouraged by the prompt phrasing to generate rationales together with the predicted answer [31, 32, 33]. Apart from the aforementioned discrete prompt strategies, soft prompting sacrifices human-interpretable templates to achieve more advanced knowledge retrieval [34]. In total, prompting can be viewed to be analogous to querying knowledge graphs in the traditional knowledge retrieval scenario.

3. Method

The retrieval backbones utilized in all cases are pre-trained VL transformers, which place both images and textual phrases in a joint embedding space. The retrieval of the most appropriate image with respect to a given text phrase is performed based on a similarity score score(, ) = max(sim(, )) , which can be obtained using diferent similarity measures, such as cosine similarity or euclidean/manhattan distance1. In the baseline case, no knowledge enhancement is performed, thus image candidates and original phrases are passed in the VL transformer of choice. The VL transformers used are CLIP-L (CLIP [35] with ViT [36] large encoder), CLIP (LAION/CLIP ViT-H/14)[37] trained on LAION-2B English subset of LAION-5B, ALIGN [38], and BLIP [39] variants; specifically for BLIP, BLIP and BLIP-L refer to variants pre-trained on COCO [ 40 ] and using ViT base/ViT large as backbone encoders respectively, while BLIP and BLIP-L variants are pre-trained on Flickr30k [ 41 ].

Moreover, based on the observations of Dadas [ 42 ], there are image candidates that present high similarity scores with multiple text phrases , thus imposing a majority bias. To this end, we adopt the penalty factor () , as introduced in [ 42 ], to diminish the similarity scores for such frequently similar candidates , without afecting the similarity scores for the rest of the candidates. In cases () is incorporated in VL retrieval, the similarity score is altered as: (, ) = sim(, ) − () (1)

3.1. LLM-based phrase enhancement

Following the LM-as-KB paradigm, the original text phrases are transformed to knowledgeenhanced phrases by harnessing the rich factual knowledge stored in state-of-the-art LLMs. 1We will be using the term similarity to refer to both cosine similarity and euclidean/manhattan distance. Zero-shot enhancement is achieved via the usage of discrete prompts that follow specific handcrafted templates, as presented in Tab. 1. Those templates are created based on human conversational intuition, encouraging the retrieval of related facts to produce .

Emerging capabilities of LLMs, such as multiple types of reasoning, are analogous to model size [ 33, 43 ], demonstrating that models beyond a specific scale may contain more advanced knowledge. We examine whether this statement also stands for knowledge enrichment by experimenting with models up to 13B parameters, which correspond to the upper limit our hardware can accommodate, as well as with orders of magnitude larger models (175B parameters) accessible via public APIs (susceptible to pre-defined pricing schemes). More specifically, we use GPT2-XL (1.5B parameters) [ 44 ], BLOOMZ-1.7B & 3B [ 45 ], OPT-2.7B & 6.7B [ 46 ], Galactica 6.7B [ 47 ], LLAMA-7B [ 48 ] and Vicuna 7B & 13B [ 49 ] in the lower-billion scale, as well as the 175B parameter models of GPT-3 [ 50 ] and GPT-3.5-turbo2.

As in the baseline case, we attempt to incorporate the penalty () in the VL retrieval stage, forming the following knowledge-enhanced similarity score for images and phrases : score( , ) = sim( , ) − () (2)

3.2. Prompting LLMs with question-answering prompts

We transform VWSD to a question-answering (QA) task by converting the textual phrases to questions that follow handcrafted prompt templates, as presented in Tab. 2. Our experimentation includes both zero-shot and few-shot prompting. In both scenarios, the LLMs prompted are the Vicuna-13B [ 49 ] and the 175B GPT-3.5-turbo. Since the LLMs to be prompted can currently handle textual but not VL inputs, we need to transit to exclusively textual representations for both images and phrases . Therefore, image captioning techniques are leveraged to achieve this transformation, providing captions for each image candidate . We experiment with greedy search, where only one caption per is returned, as well as with beam-search multinomial sampling using 5 beams, which returns =10 captions per image . The models selected for captioning are GiT-L [ 51 ] and BLIP-L [39], which are both based on ViT-large [36] encoder, and ViT-GPT2 [ 52 ], which uses ViT-base as the encoder and GPT-2 [ 44 ] as the decoder. Throughout this VWSD as QA conversion we aim to leverage diferent aspects of knowledge stored in the selected LLMs by employing the expressiveness of the designed QA prompts, which allow for advanced flexibility compared to the knowledge enhancement setting described in Sec. 3.1.

Reasoning capabilities of LLMs can be unlocked via the so-called Chain-of-Thought (CoT) prompting [33, 32, 31], where the LLM is asked to output a series of intermediate reasoning 2https://platform.openai.com/docs/models/gpt-3-5 think “Q: What is the most appropriate caption for the <context>? Answer choices: (A) <caption for image 1> (greedy) (B) <caption for image 2> ... A: Let’s think step by step. ” think “Q: What is the most appropriate group of captions for the <context>? Answer choices: (A) <captions (beam) for image 1 (separated with comma)> (B) <captions for image 2> ... A: Let’s think step by step. ” CoT “<think_prompt> <response of LLM with think prompt> Therefore, among A through J, the answer is” no_CoT “Q: What is the most appropriate caption for the <context>? Answer choices: (A) <caption for image 1> (greedy) (B) <caption for image 2> ... A: ” no_CoT “Q: What is the most appropriate group of captions for the <context>? Answer choices: (A) <captions (beam) for image 1> (B) <captions for image 2> ... A: ” choose You have ten images, (A) to (J), which are given to you in the form of captions.(A) <caption for image no_CoT 1>…(J) <caption for image 10> You should choose the image, and therefore the caption that could better (greedy) represent the <phrase>. What image do you choose? choose You have ten images, (A) to (J), which are given to you in the form of captions.(A) <captions for image 1 no_CoT (separated with comma)>…(J) <captions for image 10 (separated with comma)> You should choose the (beam) image, and therefore the set of captions that could better represent the <phrase>. What image do you choose? choose You have ten images, (A) to (J), which are given to you in the form of captions. (A) <caption for image 1> CoT … (J) <caption for image 10> You should choose the image, and therefore the caption that could better (greedy) represent the <phrase>.

Use the following format: Question: What image do you choose? Thought: you should always think about what you choose. Result: the result of your thought. Final Answer: the image that you choose.

Begin! Question: What image do you choose? choose You have ten images, (A) to (J), which are given to you in the form of a set of captions. (A) captions CoT for image 1 (separated with comma) … (J) captions for image 10 (separated with comma) You should (beam) choose the image, and therefore the set of captions that could better represent the <phrase>.

Use the following format: Question: What image do you choose? Thought: you should always think about what you choose. Result: the result of your thought. Final Answer: the image that you choose Begin! Question: What image do you choose? steps that logically lead to its answer. Even though CoT prompting has mostly been exploited for multi-step reasoning tasks, it is also able to provide human-understandable explanations regarding the choice of the most appropriate candidate for each phrase . To this end, the ifrst 5 templates and the prompting pipelines of Tab. 2 are adopted from Kojima et al. [31], where a ”reasoning” prompt (”Let’s think step by step”/think prompts of Tab. 2) retrieves the reasoning path stored in the LLM, followed by an ”answer” prompt (”Therefore, [...] the answer is”/CoT prompt of Tab. 2) that returns the final answer in an appropriate format. The rest of the templates (”choose” prompt names) are inspired from LangChain prompts [ 53 ], following an instructive and descriptive template to encourage the selection of the correct image candidate.

3.2.1. Zero-shot prompting

In the zero-shot setting, we input a selected prompt from Tab. 2 to the LLM (Vicuna-13B/GPT3.5-turbo), which generates the answer A. The generated A can be one of the caption options A-J or a statement that an answer cannot be defined; in any case, A is compared with the ground truth caption to determine the success or failure of the zero-shot prompting strategy under investigation. CoT prompts are accompanied with producing an explanation for choosing A.

3.2.2. Few-shot prompting

Additionally, we experiment with few-shot prompting in place of the previously described zero-shot prompting. In this case, we select k no_CoT/choose no_CoT (greedy) prompts (Tab. 2), accompanied by their ground truth A, thus forming QA in-context samples. The number of in-context samples k is defined by the user. We design three diferent ways of selecting the k in-context samples. In the baseline case (random), the k samples are randomly selected from the dataset. Nevertheless, since the relevance of selected samples with respect to a chosen sample is significant [ 54 ], as well as the sample ordering [ 55 ], we design two similaritybased sample selection methods, namely top and inverse-top. Both selection methods exploit embedding representations of full phrases t, which are obtained using ALIGN [38]. The k nearest embeddings to a given phrase embedding are eficiently retrieved with the help of cosine similarity. Then, the top ordering strategy places the top-1 most similar QA sample first (which corresponds to the most similar full phrase embedding), followed by the 2nd most similar QA sample, until the k-th most similar in the k-th position. On the other hand, the inverse-top strategy reverses this order by placing the top-1 most similar QA example in the k-th position.

4. Experiments

4.1. LLM-based phrase enhancement results

In Tab. 3 we present results regarding LLM-based phrase enhancement using diferent LLMs-asKB and the prompts of Tab. 1. More results are provided in Appendix A.

We can easily observe that LLM-based enhancement helps trespassing baseline accuracy and MRR scores when appropriate prompting is used, irrespectively of the usage of VL penalty () ; ”exact” prompt seems to be rather weak towards triggering the necessary knowledge to further drive VL retrieval, as in several cases metrics corresponding to phrases enhanced using the ”exact” prompt fall below the baseline performance. On the other hand, ”meaning_of” is the more powerful prompt attempted, resulting in non-negligible performance advancements compared to the baselines in most cases. Overall, the combination of GPT-3 phrase enrichment together with CLIP (with penalty () ) as the VL retrieval module lead to optimal results.

An interesting observation is that the Vicuna 7/13B models perform comparably to GPT3/3.5 models despite being orders of magnitude smaller. This is an encouraging result suggesting that LLM-based phrase enrichment may be successfully performed with more lightweight LLMs that do not adhere to a limiting pricing plan, which would impede large-scale experimentation. However, in most cases that LLMs in the lower-billion scale are employed for knowledge enhancement (Tab. 8), retrieval results struggle to compete with the knowledge-free baselines, revealing a non-negligible association between scale and knowledge-enhancement capabilities.

4.2. Question-answering prompting results

In Tab. 4 we present accuracy scores occurring from transforming VWSD to QA using zero-shot (with and without CoT) and few-shot (without CoT) prompting, denoting best results per prompt with color. In general, there is an obvious discrepancy between the performance of GPT-3.5turbo and Vicuna-13B, denoting that model scale does matter in the VWSD as QA scenario, contrary to the phrase enhancement case (Sec. 4.1): smaller models do not possess the necessary knowledge or reasoning abilities to infer the correct answer from captions in the QA setting, no matter the prompt template deployed or the choice between zero-shot/few-shot strategy. This ifnding agrees with the observation of Kojima et al. [31] that LLM reasoning capabilities emerge at scale. Moreover, there is no clear pattern whether beam or greedy captioning is more efective towards triggering the necessary knowledge: in the case of GiT-L and BLIP-L there is a clear preference towards greedy decoding in conjunction with CoT & no_CoT prompting. Especially for Vicuna-13B this preference is very distinctive, demonstrating significant performance drops when beam decoding is employed in place of greedy decoding. However, the opposite holds for ViT-GPT2, even though the overall performance of ViT-GPT2 is lower in comparison to the other captioners. The performance becomes even worse when beam decoding is employed with few-shot prompting. At the same time, ”choose” prompts do not demonstrate a distinct pattern with respect to the decoding strategy (greedy/beam) as well. Overall, GiT-L (greedy) exhibits the most promising captioning capabilities in the majority of Tab. 4 results, while BLIP-L (greedy) appears more capable in the few-shot prompting setting of Vicuna-13B (colored cells).

4.2.1. Zero-shot prompting and CoT reasoning

According to Tab. 4, accuracy results with zero-shot QA prompting are not always encouraging, especially when compared to LLM-based enrichment results of Tab. 3, which approach state-ofthe-art performance. Apart from the fact that proper LLM reasoning is still an open problem [ 56, 57 ], one possible fundamental reason for these lower accuracy scores can be the conversion from images to text via captioning: this intra-modality conversion may induce errors and information loss that impacts the final performance. Of course, lower performance will also afect the quality of the produced explanations, when CoT is employed. We delve into this scenario by presenting an example where the usage of CoT leads to an erroneous answer, while the no_CoT case succeeds. In Fig. 1 the candidates for the phrase ”tender embrace” are showcased. By employing GiT-L (greedy) as the image captioner, we form the question Q as: Q: What is the most appropriate caption for the tender embrace? Answer Choices: (A) a small boat sitting on top of a dock. (B) a group of people walking on a green hill. (C) a student gets a hug from a student. (D) a large fly laying on a rock in the water. (E) the bus stop at the station (F) a train is parked at a station. (G) a crowd of people watching a concert. (H) a train station with a sign on the side of it. (I) a black and red train on a track. (J) a man laying in the sand on top of a surfboard.

A F

B G

C (Gold image)

H

D I

E J

In Tab. 5 the results of GPT-3.5-turbo with and without zero-shot CoT are demonstrated. Q: What is the most appropriate [...] A: Let’s think step by step.

First, we need to understand what the tender embrace is referring to. Once we know that, we can look at the answer choices and find the one that best matches the subject of the photo. Therefore, without further information about the photo, it is not possible to choose the most appropriate caption. Could you please provide more context or information about the photo? Therefore, among A through J, the answer is not applicable without more information about the photo” X

After observing the captions produced by GiT-L, we confirm that they accurately describe the context, thus being appropriate answer options. Nevertheless, when prompted with CoT in a zero-shot manner, GPT-3.5-turbo is unable to define where ”tender embrace” refers to without receiving more information. On the contrary, it successfully returns the right answer when no CoT is used. This discrepancy reveals that even though the knowledge regarding the phrase ”tender embrace” exists within the LLM, zero-shot CoT prompting is unable to trigger it. This can be viewed as an inherent problem of CoT prompting, at least in the zero-shot setting, since the only factor that difers between successful and unsuccessful reasoning is the prompt itself.

On the other hand, zero-shot CoT prompting can provide valuable insights in several cases, such as the one presented in Fig. 2 with candidates corresponding to the phrase ”metal steel”. By again using GiT-L as the captioner, the question Q is formed as:

Q: What is the most appropriate caption for the metal steel? Answer Choices: (A) a chocolate bar with three sides (B) [unused0] and [unused0] at the concert in 2007 (C) a guitar and a guitar are displayed in front of a speaker. (D) frosty patterns on a window (E) gold in the rocks - - (F) a black piece of metal with a large black square in the middle. (G) a jar of honey on a wooden table. (H) a close up of a metal plate with a pattern of lines. (I) a large white quartz rock with a clear base. (J) gold jewelry from the late 19th century.

A F

B G

C H (Gold image)

D I

E J

Then, GPT-3.5-turbo outputs the results presented in Tab. 6, successfully guided by CoT. Q: What is the most appropriate [...] A: The most appropriate caption for the metal steel would be (F) a black piece of metal with a large black square in the middle. X Q: What is the most appropriate [...] A: Let’s think step by step.

First, we need to understand what metal steel is and what its characteristics are. Steel is a hard and strong metal alloy made mainly of iron and carbon. It is often used in construction, machinery, and transportation. Based on this information, the most appropriate caption for metal steel would be (H) a close up of a metal plate with a pattern of lines. This caption describes the texture and appearance of steel, which is often characterized by its distinctive pattern of lines. The other options do not accurately describe steel or its unique qualities. Therefore, among A through J, the answer is (H) a close up of a metal plate with a pattern of lines.” 4

This analysis highlights that the current state of CoT reasoning cannot be entirely trusted, as the deployment of CoT prompts themselves may drive the LLM towards incorrect reasoning paths. Thus, CoT reasoning for VWSD is not yet mature enough to provide valuable explanations.

Furthermore, regarding the comparison between standalone CoT/no_CoT prompts and ”choose” CoT/no_CoT prompts, there is a clear performance advancement when ”choose” prompts are leveraged, reaching more than 10% performance gain in the case of BLIP-L (beam) as captioner and GPT-3.5-turbo as the LLM to be prompted. We regard this as an evidence that more descriptive and motivating prompts can better evoke the correct reasoning process of an LLM. Finally, there is no clear indication of whether CoT facilitates performance over no_CoT when combined with the ”choose” prompting template, since best results per captioner and LLM of Tab. 4 alternate between ”choose” CoT/no_CoT strategies, and are often comparable.

4.2.2. Few-shot prompting

In the baseline few-shot setting (few-shot (random)), we randomly select k=5 instances to serve as in-context examples. These in-context samples are comprised of k questions Q followed by their ground truth answer choice, as described in Sec. 3.2.2. Results of Tab. 4 denote that few-shot performance using no_CoT prompts is significantly better compared to their zero-shot counterparts, while generally being on par with ”choose” prompt results, even though the derived accuracy is close to random choice or even worse in most cases. Therefore, despite the advanced engineering few-shot prompting requires compared to the zero-shot setting, it can be regarded as a more viable choice for retrieving internal reasoning capabilities of LLMs. As for the choice of in-context sample selection and order strategy, the results are not conclusive; similarity-based sample selection (top & inv. top columns of Tab. 4) may result in better (GIT-L greedy), worse (ViT-GPT2 beam) or comparable results to the random baseline accuracy. Moreover, sample ordering (top versus inv. top) presents some variability in accuracy, with each strategy performing better in diferent cases. Overall, we can state that few-shot prompting calls for more extensive experimentation beyond the purpose of the current work until a standard pattern emerges, while there is a possibility that no pattern can be inferred at all.

We are going to present some qualitative results regarding few-shot prompting. Fig. 3 contains candidates corresponding to the phrase ”football goal”, while GiT-L (greedy) serves as the captioner. The k=5 in-context samples are demonstrated in Tab. 7 followed by the answer A generated by GPT-3.5-turbo (in color) corresponding to the given Q. In the presented case, in-context prompting achieves in guiding GPT-3.5-turbo to select the correct candidate I.

A F

B G

C H

D I (Gold image)

E J

5. Conclusion

In this work, we explore the conjunction of the Visual Word Sense Disambiguation task and Large Language Models. More specifically, our current paper is the first one to harness the rich knowledge stored in LLMs via diferent prompting strategies. Following the LLM-as-KB paradigm we managed to boost the performance of baseline visiolinguistic pipelines. Moreover, we examined the potential of unimodal approaches by converting VWSD to a textual question-answer problem, where generated image captions are leveraged as multiple-choice Q: What is the most appropriate caption for the vanillin compound? Answer Choices: (A) a fossil fossil with a spiral pattern (B) a blue plate topped with pastries and a piece of bread. (C) the bones of the foot are very thin. (D) a spoon full of sugar on a table. (E) the tower of the building is made of brick. (F) a pair of sunglasses with a white background. (G) a close up of a plant with a yellow flower. (H) turmeric powder in a bowl (I) a man with glasses and a tie sitting at a desk. (J) a close up of a bike tire with a yellow tag on it. A: (D) a spoon full of sugar on a table. Q: What is the most appropriate caption for the nankeen fabric? Answer Choices: (A) a close up of a white quilt with a tree design on it. (B) a man in a blue jacket and tan pants holding a sword. (C) a collection of brass trombones (D) a close up of a green plant (E) a pine tree in the forest (F) a row of mannequins in black dresses. (G) two boys are standing in front of a red wall. (H) two apples sitting next to each other on a white surface. (I) a cartoon of two women in fancy hats and a mirror. (J) a close up of a woman’s face with a round ball on her forehead. A: (B) a man in a blue jacket and tan pants holding a sword.

Q: What is the most appropriate caption for the olmec indian? Answer Choices: (A) a large stone head in a garden (B) a woman lighting a candle at a table (C) [ unused0 ] is the new champion of the world (D) a table full of food (E) a group of women holding banners in a protest. (F) a woman in a traditional dress stands in front of a tent. (G) a man in a suit sitting in a chair. (H) a clay pot with a small hole on top of it. (I) a close up of three green leaves of a plant (J) a fern in the forest near the lake A: (A) a large stone head in a garden Q: What is the most appropriate caption for the charales order? Answer Choices: (A) a man walking past a store with a sign on it. (B) the road to the island (C) the art of the flower (D) a black snake with white spots on it’s body. (E) a bird perched on a branch in a tree. (F) a lion roaring in the wild. (G) a man falling from a skateboard (H) a large, thin, green plant with long thin leaves. (I) a close up of a bush with blue berries (J) a mouse in a hollow log. A: (H) a large, thin, green plant with long thin leaves.

Q: What is the most appropriate caption for the skink lizard? Answer Choices: (A) two lizards on a log with a log in the background. (B) a man walking past a store with a sign on it. (C) the art of the flower (D) a portrait of [ unused0 ], seated at a desk in front of a telescope. (E) the car is a small car that can be found in the museum. (F) a black snake with white spots on it’s body. (G) a bird perched on a branch in a tree. (H) a close up of a bush with blue berries (I) a close up of a metal container with a white label on it. (J) a white door with a glass window A: (A) two lizards on a log with a log in the background.

Q: What is the most appropriate caption for the football goal? Answer Choices: (A) a large stadium with a large field and a large structure. (B) a woman wearing a blue ski mask (C) [ unused0 ] of spain during the fifa world cup brazil group a match between spain and argentina at the estadio santiago bernabeu on june 25, 2010 in santiago, chile. (D) a woman lifting a barbell in a competition. (E) a man in a suit and tie looking at the camera. (F) person is a player for football team (G) [ unused0 ], the brazilian national team, in the 1960s. (H) the empty stand at the old traford stadium (I) a soccer field with a goal post in the middle (J) the emblem of the football club of the italian football club of the italian football club of the italian football club of the italian football club of the italian football club of the italian football club of the A: (I) a soccer field with a goal post in the middle 4 candidates. Finally, Chain-of-Thought prompting highlighted human-interpretable explainability aspects tied to the LLM-based knowledge extraction process. Overall, our analysis reveals the importance of model scale towards performing knowledge-related tasks with LLMs.

Acknowledgments

The research work was supported by the Hellenic Foundation for Research and Innovation (HFRI) under the 3rd Call for HFRI PhD Fellowships (Fellowship Number 5537). The work presented in this paper is co-funded by the European Union under the project AI4Culture, DIGITAL-2022-CULTURAL-02, Grant Agreement 101100683. T. Wang, T. Nkinyili, T. Schick, T. Kornev, T. Tunduny, T. Gerstenberg, T. Chang, T. Neeraj, T. Khot, T. Shultz, U. Shaham, V. Misra, V. Demberg, V. Nyamai, V. Raunak, V. Ramasesh, V. U. Prabhu, V. Padmakumar, V. Srikumar, W. Fedus, W. Saunders, W. Zhang, W. Vossen, X. Ren, X. Tong, X. Zhao, X. Wu, X. Shen, Y. Yaghoobzadeh, Y. Lakretz, Y. Song, Y. Bahri, Y. Choi, Y. Yang, Y. Hao, Y. Chen, Y. Belinkov, Y. Hou, Y. Hou, Y. Bai, Z. Seid, Z. Zhao, Z. Wang, Z. J. Wang, Z. Wang, Z. Wu, Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2022. arXiv:2206.04615. [28] S. Sia, K. Duh, In-context learning as maintaining coherency: A study of on-the-fly machine translation using large language models, 2023. arXiv:2305.03573. [29] M. Pourreza, D. Rafiei, Din-sql: Decomposed in-context learning of text-to-sql with selfcorrection, 2023. arXiv:2304.11015. [30] D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, C. Cui, O. Bousquet, Q. Le, E. Chi, Least-to-most prompting enables complex reasoning in large language models, 2023. arXiv:2205.10625. [31] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, Y. Iwasawa, Large language models are zero-shot reasoners, 2023. arXiv:2205.11916. [32] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, D. Zhou, Self-consistency improves chain of thought reasoning in language models, 2023. arXiv:2203.11171. [33] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, Y. Iwasawa, Large language models are zero-shot reasoners, 2023. arXiv:2205.11916. [34] G. Qin, J. Eisner, Learning how to ask: Querying LMs with mixtures of soft prompts, in: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, Online, 2021, pp. 5203–5212. URL: https://aclanthology.org/2021.naacl-main.410. doi:10.18653/v1/2021.naacl-main.410. [35] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, I. Sutskever, Learning transferable visual models from natural language supervision, 2021. arXiv:2103.00020. [36] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale, in: International Conference on Learning Representations, 2021. URL: https://openreview.net/forum?id=YicbFdNTTy. [37] C. Schuhmann, R. Beaumont, R. Vencu, C. W. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. R. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarczyk, J. Jitsev, LAION-5b: An open large-scale dataset for training next generation image-text models, in: Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL: https://openreview.net/forum?id= M3Y74vmsMcY. [38] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. V. Le, Y. Sung, Z. Li, T. Duerig, Scaling up visual and vision-language representation learning with noisy text supervision (2021). URL: https://arxiv.org/abs/2102.05918. doi:10.48550/ARXIV.2102.05918. [39] J. Li, D. Li, C. Xiong, S. Hoi, Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation, 2022. arXiv:2201.12086.

A. Scale significance for knowledge enhancement

As an extension of Tab. 3, we presents results regarding LLM-enhancements using other LLMs (1.7B to 7B parameters) in Tab. 8.

Compared to Tab. 3, smaller models present more mediocre knowledge enhancement capabilities, frequently scoring below the knowledge-free VL retrieval baseline. Such a result indicates that knowledge enhancement is not a ”free-lunch” solution to boost results, but requires tuning several aspects of the process in order to benefit the downstream task. Therefore, despite the encouraging performance of Vicuna7B/13B models, other LLMs in the lower-billion range do not possess the necessary knowledge capabilities to efectively perform knowledge enhancement for VWSD. Therefore, we can conclude that similarly to the QA prompting that demonstrated the need for scale, also prompting for knowledge enhancement is strongly associated with model size. To this end, we verify that smaller models with competitive capabilities, similar to the Vicuna series, need to be develop in order to facilitate applications related -but not limitedto LLM-based knowledge-enhancement.

B. Reproducibility

Our code can be found in GitHub: https://github.com/anastasiakrith/llm-for-vwsd/.

References

[1] A. Raganato , I. Calixto , A. Ushio , J. Camacho-Collados , M. T. Pilehvar, SemEval -2023 Task 1: Visual Word Sense Disambiguation , in: Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023) , Association for Computational Linguistics , Toronto, Canada, 2023 .

[2] M. Bevilacqua , T. Pasini , A. Raganato , R. Navigli , Recent trends in word sense disambiguation: A survey , in: Z. -H. Zhou (Ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, International Joint Conferences on Artificial Intelligence Organization , 2021 , pp. 4330 - 4338 . URL: https://doi.org/10.24963/ijcai. 2021 /593. doi: 10 .24963/ijcai. 2021 /593, survey Track.

[3] C. Fellbaum , Wordnet: An electronic lexical database ( 1998 ).

[4] R. Navigli , S. P. Ponzetto , Babelnet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network , Artificial Intelligence 193 ( 2012 ) 217 - 250 . URL: https://www.sciencedirect.com/science/article/pii/ S0004370212000793. doi:https://doi.org/10.1016/j.artint. 2012 . 07 .001.

[5] F. Petroni , T. Rocktäschel , P. Lewis , A. Bakhtin , Y. Wu , A. H. Miller , S. Riedel , Language models as knowledge bases? , 2019 . arXiv: 1909 .01066.

[6] B. AlKhamissi , M. Li , A. Celikyilmaz , M. Diab , M. Ghazvininejad , A review on language models as knowledge bases , 2022 . arXiv: 2204 . 06031 .

[7] M. Lymperaiou , G. Stamou, The contribution of knowledge in visiolinguistic learning: A survey on tasks and challenges , 2023 . arXiv: 2303 . 02411 .

[8] A. M. H. Tiong , J. Li , B. Li , S. Savarese , S. C. H. Hoi , Plug- and -play vqa: Zero-shot vqa by conjoining large pretrained models with zero training , in: Conference on Empirical Methods in Natural Language Processing , 2022 .

[9] J. Guo , J. Li , D. Li , A. M. H. Tiong , B. Li , D. Tao , S. Hoi , From images to textual prompts: Zero-shot vqa with frozen large language models , ArXiv abs/2212 .10846 ( 2022 ).

[10] J. Guo , J. Li , D. Li , A. M. H. Tiong , B. Li , D. Tao , S. Hoi , From images to textual prompts: Zero-shot vqa with frozen large language models , ArXiv abs/2212 .10846 ( 2022 ).

[11] P. Lu , S. Mishra , T. Xia , L. Qiu , K.-W. Chang , S.-C. Zhu , O. Tafjord , P. Clark , A. Kalyan , Learn to explain: Multimodal reasoning via thought chains for science question answering , 2022 . URL: https://arxiv.org/abs/2209.09513. doi: 10 .48550/ARXIV.2209.09513.

[12] Z. Zhang , A. Zhang , M. Li , H. Zhao , G. Karypis , A. Smola , Multimodal chain-of-thought reasoning in language models , 2023 . URL: https://arxiv.org/abs/2302.00923. doi: 10 .48550/ ARXIV.2302.00923.

[13] R. Mokady , A. Hertz , A. H. Bermano , Clipcap: Clip prefix for image captioning , ArXiv abs/2111 .09734 ( 2021 ).

[14] X. L. Li , A. Kuncoro , J. Hofmann , C. de Masson d'Autume, P. Blunsom , A. Nematzadeh , A systematic investigation of commonsense knowledge in large language models , 2022 . arXiv: 2111 . 00607 .

[15] P. Pezeshkpour , Measuring and modifying factual knowledge in large language models , 2023 . arXiv: 2306 . 06264 .

[16] B. Dhingra , J. R. Cole , J. M. Eisenschlos , D. Gillick , J. Eisenstein , W. W. Cohen , Timeaware language models as temporal knowledge bases , Transactions of the Association for Computational Linguistics 10 ( 2022 ) 257 - 273 . URL: https://aclanthology.org/ 2022 .tacl- 1 .15. doi: 10 .1162/tacl_a_ 00459 .

[17] A. Amini , S. Gabriel, S. Lin , R. Koncel-Kedziorski , Y. Choi , H. Hajishirzi, MathQA: Towards interpretable math word problem solving with operation-based formalisms , in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , Volume 1 (Long and Short Papers), Association for Computational Linguistics , Minneapolis, Minnesota, 2019 , pp. 2357 - 2367 . URL: https://aclanthology.org/N19-1245. doi: 10 .18653/v1/ N19 -1245.

[18] J. Wei , X. Wang , D. Schuurmans , M. Bosma , E. H. hsin Chi , Q. Le , D. Zhou , Chain of thought prompting elicits reasoning in large language models , ArXiv abs/2201 .11903 ( 2022 ).

[19] P. Bhargava , V. Ng , Commonsense knowledge reasoning and generation with pre-trained language models: A survey , 2022 . URL: https://arxiv.org/abs/2201.12438. doi: 10 .48550/ ARXIV.2201.12438.

[20] J. Li , L. Yu , A. Ettinger , Counterfactual reasoning: Do language models need world knowledge for causal understanding ?, 2022 . arXiv: 2212 . 03278 .

[21] J. Huang , K. C.-C. Chang , Towards reasoning in large language models: A survey , ArXiv abs/2212 .10403 ( 2022 ).

[22] P. Liu , W. Yuan , J. Fu , Z. Jiang , H. Hayashi , G. Neubig, Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing , 2021 . arXiv: 2107 . 13586 .

[23] T. Brown , B. Mann , N. Ryder , M. Subbiah , J. D. Kaplan , P. Dhariwal , A. Neelakantan , P. Shyam , G. Sastry , A. Askell , S. Agarwal , A. Herbert-Voss , G. Krueger, T. Henighan , R. Child , A. Ramesh , D. Ziegler , J. Wu , C. Winter , C. Hesse , M. Chen , E. Sigler, M. Litwin , S. Gray , B. Chess , J. Clark , C. Berner , S. McCandlish , A. Radford , I. Sutskever , D. Amodei , Language models are few-shot learners , in: H. Larochelle , M. Ranzato , R. Hadsell , M. Balcan , H. Lin (Eds.), Advances in Neural Information Processing Systems , volume 33 , Curran Associates , Inc., 2020 , pp. 1877 - 1901 . URL: https://proceedings.neurips.cc/paper_files/ paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.

[24] Q. Dong , L. Li , D. Dai , C. Zheng , Z. Wu , B. Chang , X. Sun , J. Xu , L. Li , Z. Sui , A survey on in-context learning , 2023 . arXiv: 2301 . 00234 .

[25] P. H. Winston , Learning and reasoning by analogy , Commun. ACM 23 ( 1980 ) 689 - 703 . URL: https://doi.org/10.1145/359038.359042. doi: 10 .1145/359038.359042.

[26] F. Shi , M. Suzgun , M. Freitag , X. Wang , S. Srivats , S. Vosoughi , H. W. Chung , Y. Tay , S. Ruder , D. Zhou , D. Das , J. Wei , Language models are multilingual chain-of-thought reasoners , 2022 . arXiv: 2210 . 03057 .

[27] A. Srivastava , A. Rastogi , A. Rao , A. A. M. Shoeb , A. Abid , A. Fisch , A. R. Brown , A. Santoro , A. Gupta , A. Garriga-Alonso , A. Kluska , A. Lewkowycz , A. Agarwal , A. Power , A. Ray , A. Warstadt , A. W. Kocurek , A. Safaya , A. Tazarv , A. Xiang , A. Parrish , A. Nie , A. Hussain , A. Askell , A. Dsouza , A. Slone , A. Rahane , A. S. Iyer , A. Andreassen , A. Madotto , A. Santilli , A. Stuhlmüller , A. Dai , A. La , A. Lampinen , A. Zou , A. Jiang , A. Chen , A. Vuong , A. Gupta , A. Gottardi , A. Norelli , A. Venkatesh , A. Gholamidavoodi , A. Tabassum , A. Menezes , A. Kirubarajan , A. Mullokandov , A. Sabharwal , A. Herrick , A. Efrat , A. Erdem , A. Karakaş , B. R. Roberts , B. S. Loe , B. Zoph , B. Bojanowski , B. Özyurt , B. Hedayatnia , B. Neyshabur ,

[40] T.-Y. Lin , M. Maire , S. Belongie , L. Bourdev , R. Girshick , J. Hays , P. Perona , D. Ramanan , C. L. Zitnick , P. Dollár , Microsoft coco: Common objects in context, 2015 . arXiv: 1405 . 0312 .

[41] P. Young , A. Lai , M. Hodosh , J. Hockenmaier , From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions, Transactions of the Association for Computational Linguistics 2 ( 2014 ) 67 - 78 . URL: https://aclanthology.org/ Q14-1006. doi: 10 .1162/tacl_a_ 00166 .

[42] S. Dadas , Opi at semeval 2023 task 1: Image-text embeddings and multimodal information retrieval for visual word sense disambiguation , 2023 . arXiv: 2304 . 07127 .

[43] J. Wei , X. Wang , D. Schuurmans , M. Bosma , B. Ichter , F. Xia , E. Chi , Q. Le , D. Zhou , Chainof-thought prompting elicits reasoning in large language models , 2023 . arXiv: 2201 . 11903 .

[44] A. Radford , J. Wu , R. Child , D. Luan , D. Amodei , I. Sutskever , et al., Language models are unsupervised multitask learners , OpenAI blog 1 ( 2019 ) 9 .

[45] N. Muennighof , T. Wang , L. Sutawika , A. Roberts , S. Biderman , T. L. Scao , M. S. Bari , S. Shen , Z.-X. Yong , H. Schoelkopf , X. Tang , D. Radev , A. F. Aji , K. Almubarak , S. Albanie , Z. Alyafeai , A. Webson , E. Raf , C. Rafel , Crosslingual generalization through multitask ifnetuning , 2023 . arXiv: 2211 . 01786 .

[46] S. Zhang , S. Roller, N. Goyal , M. Artetxe , M. Chen , S. Chen , C. Dewan , M. Diab , X. Li , X. V. Lin , T. Mihaylov , M. Ott , S. Shleifer , K. Shuster , D. Simig , P. S. Koura , A. Sridhar , T. Wang , L. Zettlemoyer , Opt: Open pre-trained transformer language models , 2022 . arXiv: 2205 . 01068 .

[47] R. Taylor , M. Kardas, G. Cucurull, T. Scialom , A. Hartshorn , E. Saravia , A. Poulton , V. Kerkez , R. Stojnic , Galactica: A large language model for science , 2022 .

[48] H. Touvron , T. Lavril , G. Izacard , X. Martinet , M. - A. Lachaux , T. Lacroix , B. Rozière , N. Goyal , E. Hambro , F. Azhar , A. Rodriguez , A. Joulin , E. Grave, G. Lample, Llama: Open and eficient foundation language models , arXiv preprint arXiv:2302.13971 ( 2023 ).

[49] L. Zheng , W.-L. Chiang, Y. Sheng , S. Zhuang , Z. Wu , Y. Zhuang , Z. Lin , Z. Li , D. Li , E. P. Xing , H. Zhang , J. E. Gonzalez , I. Stoica , Judging llm-as-a-judge with mt-bench and chatbot arena , 2023 . arXiv: 2306 . 05685 .

[50] T. B. Brown , B. Mann , N. Ryder , M. Subbiah , J. Kaplan , P. Dhariwal , A. Neelakantan , P. Shyam , G. Sastry , A. Askell , S. Agarwal , A. Herbert-Voss , G. Krueger, T. Henighan , R. Child , A. Ramesh , D. M. Ziegler , J. Wu , C. Winter , C. Hesse , M. Chen , E. Sigler, M. Litwin , S. Gray , B. Chess , J. Clark , C. Berner , S. McCandlish , A. Radford , I. Sutskever , D. Amodei , Language models are few-shot learners , 2020 . arXiv: 2005 .14165.

[51] J. Wang , Z. Yang , X. Hu , L. Li , K. Lin , Z. Gan , Z. Liu , C. Liu , L. Wang , Git: A generative image-to-text transformer for vision and language, 2022 . arXiv: 2205 . 14100 .

[52] A. Kumar , The illustrated image captioning using transformers, ankur3107 .github.io ( 2022 ). URL: https://ankur3107.github.io/blogs/ the-illustrated -image-captioning-using-transformers/.

[53] Langchain prompt templates , ???? URL: https://python.langchain.com/docs/modules/ model_io/prompts/prompt_templates/.

[54] J. Liu , D. Shen , Y. Zhang , B. Dolan , L. Carin , W. Chen, What makes good in-context examples for GPT-3?, in: Proceedings of Deep Learning Inside Out (DeeLIO 2022 ): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, Association for Computational Linguistics , Dublin, Ireland and Online, 2022 , pp. 100 - 114 . URL: https://aclanthology.org/ 2022 .deelio- 1 .10. doi: 10 .18653/v1/ 2022 .deelio- 1 . 10 .

[55] Y. Lu , M. Bartolo , A. Moore , S. Riedel , P. Stenetorp , Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity , in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1 : Long Papers) , Association for Computational Linguistics , Dublin, Ireland, 2022 , pp. 8086 - 8098 . URL: https://aclanthology.org/ 2022 . acl-long . 556 . doi: 10 .18653/v1/ 2022 . acl-long . 556 .

[56] Y. Bang , S. Cahyawijaya , N. Lee , W. Dai , D. Su , B. Wilie , H. Lovenia , Z. Ji , T. Yu , W. Chung , Q. V. Do , Y. Xu , P. Fung , A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination , and interactivity, 2023 . arXiv: 2302 . 04023 .

[57] M. Turpin , J. Michael , E. Perez , S. R. Bowman , Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting , 2023 . arXiv: 2305 . 04388 .