1. Introduction
their reconstruction. Task-specific fine-tuning steps upon this basic understanding of vision
and language, by refining the neural weights of the trained model to adapt to each specific task
at a time, upon which the final evaluation is performed.
Despite the rich VL knowledge acquired during this process, current transformer-based VL
models [
10, 11, 12, 13, 14, 15, 16, 17, 18
] lack generalization to several concepts and scenarios that
require commonsense knowledge, or knowledge of abstract entities, facts and real-world
events. Of course, this is somehow expected, since neither pre-training nor fine-tuning VL
datasets contain or demand perceiving concepts beyond visual descriptions. Figure 1 presents
some examples of this claim: questions (Q) about the image (I ) require some knowledge beyond
the visual domain, so that the correct answer (A) can be inferred.
Q: What days might
I most commonly go
to this building? A:
Sundays.
Q: In which
continent was the person
in the image born? A:
North America.
Q: Who among the
people in the image
is the eldest? A:
Person in the left.
Q: What is the name
of the object used to
eat this food? A:
Chopsticks.
The first image of Figure 1 requires knowledge about human culture and history [
19
], to
combine with visual information: the object in the image is a church, and people usually go to
the church on Sundays. The second image [
21
] requires one more reasoning step, since it is not
only required to detect that this is a postage stamp containing the photo of a person (visual
information), but also who this person is. Knowledge about named entities recognizes this
person as Alexander Hamilton. Further factual knowledge provides that Alexander Hamilton
was born in todays Saint Kitts and Nevis and Saint Kitts and Nevis is in North America. The
combination of these two facts derives the final answer Alexander Hamilton was born in North
America. The third image [
20
] requires the visual extraction of the two people present in it.
Then, named entities knowledge assigns the identities Serena Williams and Venus Williams
to these two people. Their age is provided as a combination of named entities and factual
knowledge, yielding the comparative knowledge fact that Serena Williams is older than Venus
Williams. Finally, spatial knowledge derives that Serena Williams is the person in the left. The
overall combination of named entities, comparative and spatial knowledge returns the final
answer The person in the left. It becomes obvious that answering these question requires more
knowledge from external sources, which is extracted and combined to infer an answer.
Thus, the incorporation of external knowledge in earlier or later stages of the
pre-training/finetuning process is necessary to enhance the capabilities of VL models, so that they are able to
respond to more real-world scenarios. Such knowledge is typically represented using entities,
relationships and semantic descriptions [
22
] stored in structured Knowledge Graphs (KGs)
[
23, 24, 25, 26
]. Language Models (LMs) such as BERT [
27
] have been proven capable of
storing relational knowledge learned from linguistic data during pre-training, introducing the
LM-as-KB scenario [
28
]. This knowledge can then be retrieved by constructing queries as
fillthe-blank statements, which the LM is tasked to complete. Further works validate the abilities
of LMs for world-knowledge storage and retrieval, while showcasing their scaling capacity
according to the number of parameters [
29
]. There are some prerequisites for LM to successfully
serve as knowledge bases; accessing the data similarly to KG querying, updating outdated
facts while trespassing the risk of catastrophic forgetting, unlocking their rather obscure
reasoning capabilities and measuring the degree of their interpretability and explainability are
still open challenges [
30
]. More recently, the impressive results of Large Language Models
(LLMs) [
31, 32, 33, 34, 35
] in various linguistic tasks greatly inspire their possible usage as rich
and simultaneously vast knowledge bases (KBs) to aid VL learning.
Prior surveys in VL learning [
36, 37, 38, 39, 40, 41, 42
] do not focus on the collaboration
between knowledge and deep learning VL models. An exhaustive presentation of the
knowledgeenhanced VL (KVL) topic was presented in [
43
] for the first time. In the current survey paper,
we focus on state-of-the-art endeavors involving transformer models for the VL representation,
leading to hybrid approaches when combined with external knowledge. Finally, we discuss
around potential trends regarding the external knowledge assisting VL models and how it is
expected to afect future applications in the field.
2. Knowledge and Reasoning
2.1. Types of external knowledge
External knowledge sources are divided in two main categories, explicit and implicit [
43
]. They
are both capable of providing factual, commonsense, temporal, lexical or other knowledge
senses [
44
] missing from pre-trained VL models. The type of the external knowledge source
used significantly defines the way of retrieving and harnessing knowledge for VL models.
Explicit knowledge refers to the knowledge stored in KGs in a structured format. Such
knowledge is symbolically represented in the form of triplets (h, r, t), which contain entities (h,
t) and their in-between relationships r. Extracting an answer from a KB is a fully transparent
process, and the path followed can be deterministically recovered. This is crucial especially
when evaluating multi-step and compositional reasoning, so that the factuality of the reasoning
path followed is guaranteed. Nevertheless, crafting and maintaining KGs requires manual efort
or supervision, therefore hindering the automatic extension of such KBs.
Popular open-source knowledge graphs that have contributed to VL learning are ConceptNet
[
24
], DBPedia [
26
], Wikidata [
25
], YAGO [
45
] and others. The retrieval of KG facts is primarily
based on SPARQL queries, a clear and deterministic query language designed for this purpose.
Many open-source knowledge graphs also provide APIs for frequent queries. Due to the
constrains SPARQL querying imposes, bridging the gap between natural language user queries
and SPARQL has been a useful venture [
46
].
Knowledge graph representation learning provides low-dimensional distributed vectors,
following the popular strategy of linguistic embeddings [
22
]. This approach allows the application
of Deep Learning techniques on KGs, while a better-suited communication between KGs and
VL models is established, since all three modalities can be viewed as numerical vectors.
Implicit knowledge covers unstructured knowledge stored in neural weights, depicting
facts and relationships learned during model pre-training. This process allows the integration of
multiple and large-scale data sources without the need for human supervision. Existing
knowledge can be extended and updated by re-training or fine-tuning the existing neural network
(LM/LLM). However, this process is computationally prohibitive for most research institutions,
while the factuality, fairness and trustworthiness of learned knowledge and reasoning are
questionable, since they significantly depend on the quality of the training data used.
There is an ongoing list of LMs-as-KB, since any language model pre-trained under
selfsupervised learning objectives can potentially serve as a KB. Retrieving knowledge from LMs is
not as straightforward as for KGs, due to the opaque LM structure. Therefore, there are two
main ways to access LM knowledge, indirect access (fine-tuning ) and direct access (prompting).
Fine-tuning has been the most concrete way to exploit LM knowledge, even if it does not
actually retrieve existing knowledge. Similarly to how fine-tuning works for VL models (Section
1), LM fine-tuning refers to adapting neural weights towards a downstream linguistic task by
training for a few epochs on a small labelled linguistic dataset, appropriate for the desired task.
Prompting obeys to the pre-train, prompt, predict pipeline [
47
], which allows direct access to
information stored in a pre-trained LM, should an appropriate prompt is designed. This is where
the dificulty of this approach lies: composing an optimal prompt is an open research topic,
and sub-optimal prompt templates may just denote the lower bound of knowledge contained
within LMs [
48, 49
]. Prompts in textual format, called discrete prompts are quite intuitive to
humans, therefore hints on how to craft them can be based on human conversational behavior.
Therefore, mining templates from large corpora [
49
], paraphrasing of existing prompts [
49, 50
],
ifll-blank via language generation [
51, 52
] and others have been proposed as viable directions.
On the other hand, soft prompts circumvent interpretability in the sake of eficiency, by directly
accessing the LM’s embedding space. Prefix-tuning using continuous task-specific vectors in a
frozen LM [
53
], soft prompting based on discrete prompting initialization [
54
] and others have
demonstrated promising results. The non-trivial search for prompt-based knowledge retrieval
is rewarded with few-shot or even zero-shot reasoning, able to revolutionize KVL models.
2.2. Reasoning in Knowledge Graphs and Large Language Models
Reasoning has been one of the milestones regarding the capacity of LLMs to suficiently perform
as KBs. It is regarded as the capability of drawing conclusions and making decisions based
on given information, and can be divided in formal and informal. Formal reasoning refers to
following a set of rules in a logical and deterministic manner. On the other hand, informal
reasoning is mostly based on a generic experience and intuition of the world, thus being prone
to errors, while however being more flexible [
55
].
KGs tend to approach the formal reasoning path, due to their structured nature and the
determinism accompanying decision-making. Informal reasoning is mostly interconnected
with unstructured KBs, which have acquired a more probabilistic sense of the world. LMs have
demonstrated some adequate reasoning capabilities, as long as they are large enough, thus
belonging to LLMs [
56
]. Nevertheless, the landscape of the full potential of LLM reasoning
has not yet been entirely explored. Prompting has been utilized towards unlocking reasoning
capabilities of LLMs, encouraging them to reveal their Chain-of-Thought (CoT) instead of merely
providing the final answer [
57, 58
]. CoT has been proven successful towards unveiling hidden
reasoning capabilities, either in the few-shot setting [
58
], where the LLM is prompted with
some exemplars of the desired reasoning, together with an instructive phrase, or in the
zeroshot setting [
57
], where the model receives an instructive phrase without exemplars. Another
line of work proposes the evaluation of LLMs on downstream tasks exploiting well-crafted
datasets, each of which is dedicated on diferent reasoning senses. To this end, various tests
have stressed LLM capabilities on arithmetic [
59
], symbolic [
58
], commonsense [
60
], and other
types of reasoning. Overall, the findings occurring from the aforementioned endeavors suggest
that indeed, LLMs present emergent reasoning capabilities simulating human thinking patterns,
though being incapable of tackling complex reasoning challenges. Nevertheless, such evidence
cannot conclude whether LLM present real reasoning capabilities or if they can just perfectly
overfit on the vast information they receive [
55
]. So far, knowledge-enhanced VL literature
trusts the LM-as-KB paradigm for several downstream tasks, demonstrating successful results.
3. VL tasks with knowledge
3.1. Visual Question Answering (VQA)
In Visual Question Answering (VQA), a model receives an image I and a textual question Q
referring to the image, and predicts a textual answer A. The answer A can be either selected
among pre-defined candidates, viewing VQA as a classification problem, or else be generated,
thus placing VQA (free-text VQA) in the language generation family. Knowledge guidance can
assist in addressing scenarios where standalone visual information is not adequate, as the ones
presented in Figure 1. A variety of knowledge-demanding datasets for knowledge-driven VQA
(K-VQA) have been developed [
19, 20, 61, 62, 63, 64, 65
], setting a good starting point for relevant
model implementations. Early attempts in K-VQA were heavily relying on exact matching
between visual or textual concepts and KG nodes via SPARQL queries [
66, 61, 67
], inducing errors
in cases when such explicit concept mapping does not exist. Embedding representations provide
a more flexible solution by retrieving similar KG facts to visual and textual concepts [
62, 68, 20
].
Nevertheless, the context-free nature of classic word embedding methods [
69, 70
] can only
serve a limited amount of cases, impeding generalization to scenarios when contextualization
is necessary. Transformers leveraged on the linguistic side allowed further improvements on
K-VQA models [
71, 72
] paving the way for consequent end-to-end VL approaches.
ConceptBERT [
73
] was the first breakthrough towards a unified KVL transformer-based
architecture, achieved by considering all three modalities in a joint representation, hence being
able to incorporate commonsense knowledge in the reasoning process. Factual knowledge
injection following the unified KVL strategy was also explored, only requiring fine-tuning to
leverage the contribution of external KGs [
21
]. Multiple knowledge senses can be fused in
unified KVL architectures, such as factual and visual knowledge, in order to cross-check the
validity of predicted answers [
74
]. The dynamic incorporation of external knowledge regardless
its type transforms knowledge injection to a passage retrieval problem, ofering advanced
adaptability to relevant architectures [
75
]. Passage retrieval from Wikipedia is also followed
in [
76
], where visual cues of diferent granularities (global captions, image labels and scene
text) are combined to retrieved facts; consequently, all linguistic information is provided to a T5
transformer model [
77
], which generates the final answer A. Similarly, [
78
] resorts to passage
retrieval from external sources, but suggests joint training of the retrieval module and the T5
answer generator, contrary to prior works. Generic information obtained via passage retrieval
from external knowledge sources is deemed inadequate to answer targeted visual questions. To
this end, knowledge acquisition is refined by focusing on common entities present in queries,
retrieved passages and images [
79
]. The combination of several KGs [
24, 26, 80, 81
] under a
unified larger KG can facilitate knowledge retrieval, which is performed based on the linguistic
similarity between contextualized questions (questions enhanced with visual captions and scene
text as context) and KG facts. The enriched input is provided to a T5 transformer which finally
generates the final free-text answer A [
82
]. Multimodal passage retrieval is addressed via the
proposed Multimodal Inverse Cloze Task as pre-training objective, which learns the alignment
between visual and textual information from Wikipedia entries. This technique can aid K-VQA
involving named entity recognition [
83
].
Diverging from the usage of KGs as the knowledge source at hand, first works exploiting the
LM-as-KB paradigm were introduced for K-VQA. Specifically, GPT-3 [
31
] can be used to provide
facts in a few-shot manner, receiving visual captions as prompts [
84
], similar to how traditional
KGs receive SPARQL queries. Performance gains can be achieved by utilizing multiple captions
as prompts to a variety of pre-trained LLMs, enabling zero-shot reasoning [
85
]. Using again
linguistic captions as a modality mediator, [
86
] leverages frozen LLMs to address zero-shot VQA.
Chain of Though (CoT) prompting of LLMs is another interesting direction, which enhances
explainability of the answer derivation pipeline by revealing intermediate reasoning steps
[
87
]. Instead of resorting to the linguistic modality to obtain unimodal LLM prompts, other
approaches opt to fine-tune a visual encoder jointly with the LLM, so that aligned LLM-VL
representations are achieved [
88
].
There are a few implementations combining explicit and implicit knowledge sources to
enjoy advantages of both worlds. KRISP [
89
] leverages several external KGs [
24, 26, 81
], visual
knowledge from Visual Genome [
90
], as well as implicit knowledge from BERT [
27
]. REVIVE
[
91
] deploys several visual features to retrieve knowledge from various sources, such as Wikidata
and GPT-3. Visual feature guidance was proven critical towards improving the knowledge
retrieval process. Fusing both implicit and explicit knowledge in the VL reasoning process is
also followed in KAT, using a refined framework that fetches information from Wikidata and
GPT-3 upon which joint reasoning is performed. A transformer decoder receives the output of
the reasoning module to generate the final answer [
92
].
3.2. Visual Commonsense Reasoning (VCR)
Visual Commonsense Reasoning (VCR) is a task closely related to VQA. Given a challenging
question Q regarding an image I, a VCR model is tasked to predict the answer A, accompanied
by a rationale R [
4
]. Commonsense knowledge can be provided from large-scale KGs, such as
ConceptNet [
24
] and ATOMIC [
93
], or dedicated datasets, such as SWAG [
94
], which contains
descriptions about sequences of events.
Transformer-based endeavors for knowledge-assisted VCR (K-VCR) naturally utilize BERT
[
27
] as the backbone architecture to construct end-to-end KVL models. In KVL-BERT [
95
], the
input Q together with candidate answers A guide the retrieval of relevant commonsense facts
[
24
], resulting in a knowledge-enriched linguistic input. Then, visual features among with this
enriched input are inserted in a BERT-like VL model (VL-BERT [
96
]) so that the correct A is
selected. Consequently, inferring R requires feeding VL-BERT with the predicted A, candidate
rationales R and visual features. Aligning independent modality representations within a single
multimodal embedding is proposed in [
97
]. The same work introduces extensions of VL
pretraining objectives [
43
] to incorporate commonsense knowledge from [
24
] as an extra modality,
therefore enforcing learning KVL interrelationships. Dynamic commonsense augmentation
of image-text training data is a suggested direction, accompanied by learning to reconstruct
hidden visual labels based on knowledge facts retrieved from commonsense KBs [
98
].
Implicit knowledge sources have been gaining popularity in recent K-VCR literature. GPT-2
[
99
] has assisted dynamic reasoning over images, inferring temporal hypotheses regarding
what might have happened before and what might happen after the depicted situation [
100
].
Chain of Thought (CoT) reasoning is inherently tied to VCR, as reasoning paths are highly
associated with selecting rationales R. The rise in popularity of CoT techniques for linguistic
tasks is highly interconnected with the development of LLMs, which have been proven able to
reveal intermediate reasoning steps [
57
]. There are not yet many works in the VL direction,
even though the introduction of novel appropriate datasets with grounded answer rationales
highlight the prospects of such an approach [
101
]. Specifically, [
101
] tackles VCR by captioning
the image, and then feed the caption together with the existing linguistic input to the LLM.
Another promising work in this direction introduces Multimodal-CoT without using language as
the mediating modality, proposing a two-stage process to separately infer the answer A and the
rationale R, while stating that a LM with less than 1B parameters is adequate for state-of-the-art
performance [
102
]. It is expected that the rapid rise of popularity of LLMs in complex linguistic
QA reasoning [
103
] may soon give rise to more LLM-augmented VCR approaches, addressing
more aspects of reasoning.
3.3. Image Captioning (IC)
Image Captioning (IC) is a widespread VL task, asking from a model to generate a caption c
for an image I. Several knowledge-enhanced IC (K-IC) techniques employ recurrent neural
networks (RNNs) or relevant variants such as Long Short Term Memory (LSTM) networks, while
leveraging knowledge sources for commonsense-enhanced captioning [
104, 105, 106, 107
].
The integration of external knowledge in IC using transformers, was first explored in [
108
],
where event and named-entity knowledge is fused together with textual and visual data in a
Transformer encoder [
1
] to generate entity/event-aware captions. Commonsense descriptions
derived from ConceptNet [
24
] and ATOMIC [
93
] are able to assist visual commonsense
generation (VCG), a challenging task that requires inferring intents and temporal sequence of events
[
109
]. This is achieved by incorporating commonsense descriptions to BART, a powerful
language generation model [
110
]. Geographical information guiding factual knowledge retrieval
to assist IC was first explored in [
111
], where visual features together with the extracted facts
are inserted in a Transformer encoder-decoder structure, ultimately generating the caption c.
Apart from external knowledge considerations, IC faces additional challenges as a language
generation task: VL transformers are not well-suited for generative tasks, even though they excel
in understanding tasks, where an answer has to be selected among a set of pre-defined options.
XGPT tackles this challenge by adapting generative pre-training [
99, 31
] for VL tasks [
112
],
which is achieved by introducing novel generative pre-training objectives. The collaboration of
GPT-2 [
99
] with CLIP [
15
] is viewed as highly promising, since both models have been trained
on an abundance of web-data, thus incorporating numerous knowledge senses in an implicit
manner. ClipCap [
113
] leverages this collaboration without re-training CLIP or GPT-2; instead,
a lightweight transformer-based mapping module is trained to match CLIP representations
to GPT-2, which eventually generates the caption c. The CLIP-GPT-2 combination was also
followed in VC-GPT [
114
]. Another lightweight improvement combining a pre-trained CLIP
visual encoder and a frozen GPT-2 text decoder further boosts performance of low-resource
approaches [
115
]. A cross-modal filter that selects the most relevant visual information, so
that captioning errors are reduced is proposed in [
116
], still respecting the frozen CLIP-GPT-2
framework.
A factor that overshadows the knowledge-enhanced IC capabilities is the lack of dedicated
datasets for testing. So far, IC models are evaluated on classic datasets containing images and
captions, such as COCO [
117
] and Flickr [
118
], which however are not challenging in terms of
external knowledge required. The construction of appropriate datasets that would follow the
paradigm of knowledge-demanding VQA datasets, such as OK-VQA [
19
], K-VQA [
20
], FVQA
[
62
], KB-VQA [
61
], or VCR datasets [
4
] will give prominence to the abilities of K-IC models.
3.4. Sequential Generation
There are two tasks touching sequential generation, depending on which modality is generated
at a time. Visual Storytelling (VIST) refers to generating captions 1, 2, ..., for a visual story
comprised of frames 1, 2, ..., , as an extension of IC for sequences. The reverse task of
Story Visualization (SV) involves synthesizing visual frames 1, 2, ..., from textual captions
1, 2, ..., . For both tasks, consistency throughout the story and relevance between the two
modalities are required.
Harnessing commonsense knowledge for VIST was first attempted in [
119
], followed by [
120,
121
], which however utilize RNN-bases structures for text generation. The usage of
transformerbased models is explored in [
122
], where visual concepts are enriched through ConceptNet.
All relevant enriched concepts are provided to BART, which ultimately outputs appropriate
captions. Regarding SV, there is a diferent architectural line followed [
123, 124, 125, 126
] based
on Generative Adversarial Networks (GANs) [
127
]. Commonsense and spatial knowledge
considerations were primarily addressed in [
125
], demonstrating encouraging results in favor
of the usage of external knowledge. Nevertheless, some recent approaches follow
transformerbased approached enhanced with implicit knowledge. Specifically, Story-DALL-E [
128
] is able to
even synthesize unseen stories in a zero-shot fashion. It leverages DALL-E [
129
] as a multimodal
unstructured knowledge base that provides high-quality visual synthesis from text.
Similar to IC, sequential generation tasks face the lack of appropriate datasets, upon which
the external knowledge contribution would be more evident and meaningful.
3.5. Multi-taskers
Knowledge-free VL models have already achieved incorporating a variety of VL tasks under the
same pre-training body, only requiring fine-tuning on smaller labelled text-image datasets. This
way, there is no need to design and implement a separate architecture per independent task, but
rather exploit visiolinguistic relationships present in large scale datasets used for pre-training,
such as COCO [
117
], Visual Genome [
90
], Conceptual Captions [
130
] and SBU [
131
].
Reasoning tasks, including visual question answering, visual-text entailment and visual
commonsense reasoning can be easily incorporated under the same model, due to the high
similarity of the inferences these tasks have to make. Visual cues are enhanced with higher-level
cognition provided by VisualCOMET [
100
] and among with textual inputs, they are fed in a
GPT2 model that generates free-text rationales for all three tasks, therefore providing explainability
of answers [
132
]. The same reasoning tasks were also explored in [
133
], testing results on
knowledge-demanding datasets [
19, 62
] as well. Knowledge embeddings representing facts from
external knowledge sources [
24, 25
] are aligned with textual descriptions, which are inserted
together with the knowledge features in a multimodal transformer. KB-VLP [
134
] is another
multi-task model tackling visual question answering and visual reasoning, enhanced with
commonsense and logical capabilities. Entity extraction from images and text is performed to
map VL concepts to Wikidata [
25
] entries, based on knowledge graph embeddings, thus finally
assisting the retrieval of the most relevant Wikidata facts.
Few-shot and zero-shot learning via LLMs in low-resource scenarios can be very practical
for generative VL tasks. In [
88
], an encoder-decoder Transformer model is the most
appropriate option for text generation, addressing free-form VQA and IC, while various prompts are
leveraged in inference time to enforce generating a suitable textual prediction. PromptCap
[
135
] is designed to accurately generate fine-grained and controllable captions based on GPT-3
prompting. The generated captions can be leveraged to provide context for VQA. The Socratic
model framework [
136
] is a novel and promising direction, since it demonstrates that the
complementary knowledge obtained during pre-training from VL and LM can be combined towards
several multimodal tasks with the help of multimodal prompting.
There are some significant observations arising from the construction of multi-task VL models
regarding the usage of external knowledge. For example, in [
133
], incorporation of Wikidata
failed to enhance reasoning capabilities as expected. The authors attribute this deteriorated
performance to ambiguities and noise existing in the large Wikidata KB. Furthermore, comparing
to task-specific implementations for VQA, IC etc, it is obvious that multi-task implementations
are significantly fewer. This indicates that the incorporation of external knowledge is not as
straightforward for multiple tasks as it may be for task-specific models.
3.6. The future of knowledge in VL
Recent literature around KVL research reveals several gaps that need to be covered. In our
opinion, the most prominent gap is the lack of appropriate knowledge-demanding datasets
for most tasks, which limits the extend to which KVL models are evaluated. Apart from VQA
[
19, 20, 61, 62, 63, 64, 65
] and VCR [
4
], other VL tasks are tested on classic benchmark datasets;
therefore, the contribution of knowledge in downstream performance is not as prevalent as in
situations where various senses of knowledge are explicitly required.
At the same time, we have observed that several eforts are dedicated towards constructing
linguistic benchmarks questioning reasoning capabilities in LMs, including mathematics [
59
],
symbolic reasoning [
58
], implicit reasoning based on strategies [
137
], commonsense
understanding [
138
], temporal, causal, linguistic understanding and others [
139
]. We argue that such
attempts could assist the creation of appropriate VL datasets, which would incorporate visual
and linguistic challenges, so that knowledge contribution would be more concrete.
With the surge of larger and larger models, explainability concerns are raised in the broader
AI community [
140
], especially when black-box VL models are combined with black-box
unstructured KBs. Older KVL architectures were often addressing explainability [
67
], as an
immediate result of incorporating KGs for answer prediction, since the path leading to the final
answer could be retrieved. Later works, even though widely exploiting KGs, mainly focused on
improving downstream performance rather than enhancing the interpretability of reasoning
towards these results. Currently, KVL approaches have totally deviated from the pursue of
explainability, especially since opaque LLMs started acting as KBs for VL models.
Another emerging issue is prompt design for KVL architectures that exploit LLMs as their
external knowledge source. Similar challenges to linguistic prompt search also arise for the
VL setting (especially since many models use language as a mediator between vision and
language), thus inducing some ambiguity regarding the quality of results and the reasoning
process followed to return these results. Multimodal prompt design is still an unexplored but
crucial field towards unlocking the full potential of LLM-VL models.
Ultimately, we view that the future of KVL research is highly interconnected with the current
trends in LLMs, both in terms of designing appropriate knowledge-demanding benchmarks for
KVL tasks, as well as answering the KG vs LLM ongoing dilemma, analyzed in Section 4.
4. Knowledge Graphs or Large Language Models?
Throughout our analysis we recognize some potential trends towards selecting the type of
knowledge source to assist VL models towards hybrid approaches. Even though KGs clearly
dominate previous VL implementations, we can assume some focus shifting towards LMs-as-KB,
due to the rapid development of L(L)Ms and generally their ever increasing popularity in recent
NLP literature. For example, the advent of ChatGPT1 created an unprecedented hype, opening
several discussions regarding the future of AI as a whole. The almost human-level capabilities
of ChatGPT and relevant implementations have created an abundance of opportunities in NLP
literature, while sparking a lot of unavoidable criticism. Since VL learning tends to follow the
advancements in NLP, as it happened with KG-based knowledge boosting [
141, 142, 143, 144
],
it is highly likely that more LLM-based VL implementations will soon emerge.
However, this trend comes at a cost: LLMs have already scaled to billions and even trillions
[34] of parameters, a procedure that requires massive training. Concerns regarding the cost of
pre-training language models has been raised even before this tremendous parameter scaling
[
145
]; apart from computational budget, issues such as fair access and environmental impact of
relevant implementations should question the reliance and preference of the research community
towards them.
In the meanwhile, the completely opaque learning and decision-making process may be more
harmful than beneficial; even though known biases, errors and inaccuracies 2 are reported in
LLM publications, their actual usage raises doubts regarding the quality and the trustworthiness
of the knowledge provided to the final task. Early on the introduction of LLMs, such as GPT-3
[
31
], papers exposing failure cases regarding mathematical reasoning, logic and ethical requests
[
146
] gained lots of attention. Probing how LLMs can be fooled, sheds some light to their
reasoning process and how they are being confused by misleading inputs [
147, 148
], which
can be a promising starting point towards defeating logical brittleness. Relevant endeavors
uncover LLM deductive reasoning capabilities [149], proving that exhaustive memorization
compensates for LLM inability to learn to reason. In total, reasoning capabilities of LLMs
pose several open questions [
55
], such as whether heuristics conceal reasoning incapability, or
whether reasoning steps can be trustworthy, given that inconsistencies and false rationales are
sometimes provided with certainty. More refined challenges reveal that LLMs’ world-knowledge
sufers from robustness issues as well, since they confuse likely and unlikely situations, even
though they are capable of recognizing impossible events [150].
Even if aforementioned concerns are somehow addressed in consequent versions of relevant
LLMs, the search for the optimal fact retrieval process via prompts still remains an open challenge
[
47
]. Moreover, papers promoting certain prompts to LLMs avoid describing the process behind
discovering such optimal prompts, and merely provide some experimental comparison between
similar prompt phrases [
57
]. Therefore, if the golden prompt [
57
] let’s think step by step has
been defined via extended experimentation, there is no guarantee regarding its optimality and
reliability; on the other hand, if there is a certain methodology behind, it seems that it has not
been fully unlocked (or at least released to the public). Low-performance instructive prompts
targeting CoT reasoning are semantically relevant to the golden prompt, thus raising doubts
regarding the consistency of LLM-occurring answers and their sensitivity to -slight or more
intense- input variations.
Overall, the aforementioned shortcomings are expected to be transferred to VL
implementations if LLMs eventually replace KGs to some extend, thus raising a crucial question: is it worth
it to quickly adapt to the trend or better wait?
Acknowledgments
The research work was supported by the Hellenic Foundation for Research and Innovation
(HFRI) under the 3rd Call for HFRI PhD Fellowships (Fellowship Number 5537).
R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin,
S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, D. Amodei,
Language models are few-shot learners, in: H. Larochelle, M. Ranzato, R. Hadsell, M. F.
Balcan, H. Lin (Eds.), Advances in Neural Information Processing Systems, volume 33,
Curran Associates, Inc., 2020, pp. 1877–1901. URL: https://proceedings.neurips.cc/paper/
2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
[32] J. Hofmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas,
L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. v. d. Driessche,
B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, L. Sifre,
Training compute-optimal large language models, 2022. URL: https://arxiv.org/abs/2203.
15556. doi:10.48550/ARXIV.2203.15556.
[33] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W.
Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao,
P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope,
J. Bradbury, J. Austin, M. Isard, G. Gur-Ari, P. Yin, T. Duke, A. Levskaya, S. Ghemawat,
S. Dev, H. Michalewski, X. Garcia, V. Misra, K. Robinson, L. Fedus, D. Zhou, D. Ippolito,
D. Luan, H. Lim, B. Zoph, A. Spiridonov, R. Sepassi, D. Dohan, S. Agrawal, M. Omernick,
A. M. Dai, T. S. Pillai, M. Pellat, A. Lewkowycz, E. Moreira, R. Child, O. Polozov, K. Lee,
Z. Zhou, X. Wang, B. Saeta, M. Diaz, O. Firat, M. Catasta, J. Wei, K. Meier-Hellstern,
D. Eck, J. Dean, S. Petrov, N. Fiedel, Palm: Scaling language modeling with pathways,
2022. URL: https://arxiv.org/abs/2204.02311. doi:10.48550/ARXIV.2204.02311.
[34] N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu,
O. Firat, B. Zoph, L. Fedus, M. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat,
K. Robinson, K. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen,
C. Cui, Glam: Eficient scaling of language models with mixture-of-experts, 2021. URL:
https://arxiv.org/abs/2112.06905. doi:10.48550/ARXIV.2112.06905.
[35] S. Borgeaud, A. Mensch, J. Hofmann, T. Cai, E. Rutherford, K. Millican, G. v. d. Driessche,
J.-B. Lespiau, B. Damoc, A. Clark, D. d. L. Casas, A. Guy, J. Menick, R. Ring, T. Hennigan,
S. Huang, L. Maggiore, C. Jones, A. Cassirer, A. Brock, M. Paganini, G. Irving, O. Vinyals,
S. Osindero, K. Simonyan, J. W. Rae, E. Elsen, L. Sifre, Improving language models by
retrieving from trillions of tokens, 2021. URL: https://arxiv.org/abs/2112.04426. doi:10.
48550/ARXIV.2112.04426.
[36] T. Baltrušaitis, C. Ahuja, L.-P. Morency, Multimodal machine learning: A survey and
taxonomy, 2017. arXiv:1705.09406.
[37] K. Kafle, R. Shrestha, C. Kanan, Challenges and prospects in vision and language research,
2019. arXiv:1904.09317.
[38] W. Guo, J. Wang, S. Wang, Deep multimodal representation learning: A survey, IEEE
Access 7 (2019) 63373–63394. doi:10.1109/ACCESS.2019.2916887.
[39] A. Mogadala, M. Kalimuthu, D. Klakow, Trends in integration of vision and language
research: A survey of tasks, datasets, and methods, Journal of Artificial Intelligence
Research 71 (2021) 1183–1317. URL: http://dx.doi.org/10.1613/jair.1.11688. doi:10.1613/
jair.1.11688.
[40] C. Zhang, Z. Yang, X. He, L. Deng, Multimodal intelligence: Representation learning,
information fusion, and applications, IEEE Journal of Selected Topics in Signal Processing
(2023).
[149] Z. Yuan, S. Hu, I. Vulic, A. Korhonen, Z. Meng, Can pretrained language models (yet)
reason deductively?, ArXiv abs/2210.06442 (2022).
[150] C. Kauf, A. A. Ivanova, G. Rambelli, E. Chersoni, J. S. She, Z. Chowdhury, E. Fedorenko,
A. Lenci, Event knowledge in large language models: the gap between the impossible
and the unlikely, ArXiv abs/2212.01488 (2022).