Can LLMs solve generative visual analogies?

Abstract

Recent experiments with large language models (LLMs) have provided some evidence that these models can perform abstract analogical reasoning [1], including textual puzzles similar to Raven's progressive matrices. We consider a visual analogical reasoning task that was solved using neuro-symbolic techniques in [2], and investigate how LLMs fare on this task. The task involves learning a sequence of transformations by which a sample input/output pair of images are related so as to analogously transform a test input. Note that unlike the analogical reasoning tasks in [1], this task involves generating an output as opposed to selecting from a set of choices. We evaluated various LLMs including GPT-4, GPT 3.5-turbo (ChatGPT), and GPT3 on this task for difering lengths of the sequence of transformations relating the input and output. Our results suggest that GPT-4 performs the best overall, while GPT 3.5-turbo and GPT3 perform strongly on shorter program lengths. At the same time, the performance of LLMs for this task falls far short of the neuro-symbolic approach used earlier, and we speculate as to why this may be the case, at least as of now.

Keywords

• eol>Large language models
• GPT-4
• Visual analogy
• Neural analogical reasoning

1. Introduction

2. Experiments

We applied LLMs to solve such a visual analogy task, with the image translated into symbolic form. We use the trained models provided by OpenAI API and give a few solved examples in the prompt to help the LLMs learn.

Prompting We prompt the LLMs with a set of rules for the task, which includes information on the allowed positional shifts, the set of permissible states, and the non-wrapping state of the grid. As a hint, we also specify the expected program length in the prompt. We also provide a set of solved examples to guide the LM’s learning process. Figure 1a illustrates a representative example of the prompt. (a) Prompt with rules, solved examples and test input.

(b) Chain-of-thought vs. simple

prompting , )

) Task

Solution (c) Given example input-output image pairs, generate the analogous output for the given query image.

GPT-3

GPT-3.5-turbo GPT-4

GPT3 Simple Prompting Chain-of-thought prompting

Tokens Accuracy

Tokens Accuracy 2847 48%

Chain-of-thought prompting: Previous works such as [ 3 ] have shown that language-model performance increases drastically in reasoning tasks when given chain-of-thought prompts. In our context we provide a chain-of-thought by providing, with each step of the solved example, the current state on the grid, positional shift, next position on grid, and the state of the output bufer.

Changing the program length: We experimented with diferent program lengths (3 & 5); empirically, increasing the program length makes the task more dificult.

3. Results and Conclusions

Referring to Table 1 we observe the following: (i) Chain-of-thought improves performance over simple prompting, which was expected. (ii) Further, chain-of-thought prompting is also better (albeit slightly) than providing more examples for similar token lengths, see Table 2. (ii) Sampling more examples improves performance, also expected. (iii) Analogies involving longer sequences (programs) are more dificult as expected; however we observe a drastic drop in performance for GPT3 and GPT 3.5-turbo but only a marginal drop is observed for GPT4. While the input prompts do afect the LLMs’ performance, it’s important to mention that a uniform prompt template was used for all the analyzed LLMs in this study.

Overall the performance of LLMs for our simple visual analogy task fall far short of the neuro-symbolic techniques used in [ 2 ]. We note that [ 2 ] relied search over possible sequences that could successfully transform a text input to its output. LLMs do not explicitly search over potential outputs. We speculate that incorporating elements of explicit search may enable LLMs to perform better at generative analogies.

References

[1] W. et. al, Emergent analogical reasoning in large language models , arXiv:2212.09196 ( 2022 ).

[2] S. et. al, Solving visual analogies using neural algorithmic reasoning , AAAI (Student Abstract) ( 2022 ).

[3] J. W. et. al, Chain of thought prompting elicits reasoning in large language models , NeurIPS 2022 ( 2022 ).