We introduce CLEVR-Math, a multi-modal math word problems dataset consisting of simple math word problems involving addition/subtraction, represented partly by a textual description and partly by an image illustrating the scenario. The text describes actions performed on the scene that is depicted in the image. Since the question posed may not be about the scene in the image, but about the state of the scene before or after the actions are applied, the solver envision or imagine the state changes due to these actions. Solving these word problems requires a combination of language, visual and mathematical reasoning. We apply state-of-the-art neural and neuro-symbolic models for visual question answering on CLEVR-Math and empirically evaluate their performances. Our results show how neither method generalise to chains of operations. We discuss the limitations of the two in addressing the task of multi-modal word problem solving.
problem to equation) and have employed sequence-to-sequence networks or sequence to tree
(generating the expression tree of the equation) networks ([
While text-based math word math
problems are a great setting for natural language
understanding, it would also be interesting
to consider word problems which are
accompanied by a diagram, and the information
required to derive the solution has to be
captured from both its textual and visual
representations. That is, part of the problem
scenario description is expressed as text and the
other part is represented in the form of an
image. In this paper, we introduce such a
multimodal math word problem dataset,
CLEVRMath (since it is based on CLEVR dataset [
While each instance in CLEVR dataset has mathematical equation = 9 − 2. an image and a natural language query about the scene depicted in the image, in CLEVRMath, the natural language query may not be about the scene represented in the image, but about the state of the scene after/before a sequence of actions are applied on the scene. The actions in our case are addition/removal of specific type of objects to/from the original scene. We believe this is an interesting problem setting as the ability to envision changes without them being physically manifested is an important aspect of the human mind.
Our contributions are two-fold, we • construct an open source multi-modal math word problem dataset, CLEVR-Math and • analyse the performance of state-of-the-art neural and neuro-symbolic (NeSy) solutions for solving such multi-modal problems.
Our results and analysis shows how both neural and NeSy methods are unable to compositionally generalise to chains of operations.
This section gives an overview of the existing datasets in math word problem solving,
multimodal datasets for visual question answering and existing neural/neuro-symbolic approaches
to the tasks that require a combination of visual, language and logical reasoning.
Math Word Problem Solving - Datasets Math Word Problem Solving (MAWPS) [
Experiments with Kandinsky patterns [
We construct the CLEVR-Math dataset as an extension of CLEVR by introducing three new functions and 13 templates. Using the codebase provided with CLEVR, we generate new questions based on the original scenes. We categorise the 13 templates into six types, all based on addition and subtraction. The domain is restricted to numbers between 0 − 10 to conform with CLEVR. New CLEVR functions: The three functions that we implement are - subtraction and addition to perform subtraction and addition, and choose to operate on subsets of objects. Instead of removing all blue spheres, choose allows us to remove a random number of a specific type of object, e.g. 2 blue spheres out of 4. The random number generated by choose replaces a questions “X" placeholder during generation. Figure 2a shows three examples of subtraction, and Figure 2b shows a question requiring multihop reasoning. Appendix A includes more samples from the test set.
(a) (i) Remove all gray spheres. How many spheres are there? (3), (ii) Take away 3 cubes. How many objects are there? (7), (iii) How many blocks must be removed to get 1 block? (2) (b) Take away all large green metallic spheres.
Now remove all cyan objects. How many objects are left? (4)
Question Categories: The diferent question categories are shown in Table 1. • Remove group: All objects belonging to a specific group are removed from the scene. • Insertion: A specific number of objects are added to the scene. • Count backwards: The query is about the change - that is the number of objects added/removed from the scene to get a goal state. • Remove subset: A specific number of objects are removed from the scene. • Adversarial questions: These are trick questions where the actions may be performed on one object, but the query is about an object that is not afected by the action. The adversarial actions are always on objects that are seen in the image. • Multi-hop: In contrast to the above questions, multi-hop questions perform sequences of actions (insertion, removal) on the objects. Such questions with chained functions help us test a model’s ability to generalise to infinite combinations of operations.
Remove group
"Remove all <C> <S>s. How many <S>s are there?" "Take away all <Z> <C> <M> <S>s. How many <S>s are there?" "Take away X <C> <S>s. How many objects are there?" "Take away all <C> <S>s. How many objects are there?" "Add X <Z> <C> <M> <S>s. How many <Z> <C> <M> <S>s are here?" "Add X <Z> <C> <M> <S>s. How many objects are there?" "How many <C> <S>s must be removed to get X <C> <S>s?" "Take away <C> <S>s. How many were removed if there are X <C> <S>s left?" "Take away all <Z> <C> <M> <S>s. Remove all <Z2> <C2> <M2> <S2>s. How many objects are left?" "Remove X <S>s. How many <S>s are there?" "Remove all <C1> <S1>s. Remove all <C2> <S2>s. How many <S1>s are left?" "Remove all <C1> <S1>s. How many <C2> <S2>s are left?"
Each problem in the dataset is also annotated with it’s equivalent functional program based on the CLEVR functions described in the previous section. For example, consider the question from insertion category and it’s program (the arguments of an instruction refer to another instruction - indicating it’s input is the output of the referred instruction): The program contains the choose function - choose[] operator returns ( = 3 in this case). Question generation. To support greater linguistic variation, we add synonyms for addition and subtraction to the template engine. Subtract can be replaced with remove, take away and withdraw, and addition with introduce, and insert. We use the same training and validation scenes as CLEVR, and generate 5000 new scenes as test data. Figure 3 show the distribution of attributes, words, templates and answers in CLEVR-Math, aggregated over the training, validation, and test data. The distribution is reflected in each of the splits.
2 1.5 0.5 1 0 · 105
Shape Material
Size Color (d) Template distribution over categories of templates. Each bar corresponds to a template in each respective category. We see that subset subtraction (i.e., remove 2 blue cubes) is underrepresented.
There are 50 words in the CLEVR-Math vocabulary, where the narrow language puts focus on the mathematical reasoning rather than advanced language capabilities. Figure 3c show that most questions are 8-9 words long, with a second peak at 13 for the multihop questions.
(a) Attribute distribution per category, showing even allocations. · 104 · 105 5 10 10 15 (b) Answer distribution, from 0 to 10.
· 104
49149 41600 13900 14553
Open sourcing data. We open source CLEVR-Math as a Huggingface dataset 1 with two configurations; CLEVR-Math and CLEVR-Math-multihop. The extended CLEVR source code is available on Github 2. Table 3 shows the Huggingface dataset card for CLEVR-Math. The template feature allows for filtering to perform, e.g., only singlehop training and multihop testing.
Feature template
id question image label
String String
String image path int64, 0-10
subtraction-multihop CLEVR_math_test_000010.png Remove 5 spheres. How many objects are there? CLEVR_v1.0/images/train/CLEVR_new_000010.png 5
CLIP [26] is used as a neural baseline. Questions and images are embedded using CLIP, and an additional classification layer is added to predict the correct answer. Fine tuning CLIP on CLEVR-Math as a masked language task before adding classification gave no significant improvements, while consuming significantly more computational resources. CLIP and this classification layer is trained jointly for 10 epochs with early stopping using a batch size of 64.
NS-VQA [28] is used as the neuro-symbolic baseline. Here, a mask-RCNN [29] is trained independently to convert an image to a scene graph. In our experiments, we skip this step and use the actual scene graphs associated with images. The question is parsed into a functional 1https://huggingface.co/datasets/dali-does/clevr-math 2https://github.com/dali-does/clevr-math program by a sequence to sequence (Seq2Seq) network based on Bi-LSTM. A quasi-symbolic program executor executes the program generated on the scene graph of the image to return an answer. The Seq2Seq network is pre-trained in a fully supervised fashion by providing it a few examples (around 60 examples) of (, ) pairs. The pre-trained network is then trained further using REINFORCE algorithm that returns a reward based on whether the program generated could derive the expected answer or not. Supervised pretraining and REINFORCE were run for 1000 and 5000 iterations, respectively, with a batch size of 128. Both CLIP and NS-VQA models were trained on a NVIDIA Tesla P100 GPU computing processor.
Each model is evaluated on each question category, and are trained on 2500, 5000, 10000, and 20000 samples to see the influence of the amount of data. For multihop, training and validation sets with and without multihop questions are used, with the latter named multihop (0-shot). 4.1. Results Table 4 shows the accuracy of CLIP and NS-VQA on the diferent categories as well as an aggregated accuracy over the entire dataset. Both the models were trained on 10, 000 samples. NS-VQA performs better than CLIP models for most templates apart from multihop. NS-VQA performs better on subtraction and adversarial problems (both based on ‘subtraction’ CLEVR function) than addition problems. This could be because the functional programs for addition problems always contain a choose operator. It is important to identify the argument to choose operator from the problem statement (which is mostly one of the numerical quantities in the word problem) to arrive at the correct answer. Unlike this, there are subtraction and adversarial problems (in remove group) that do not have a choose operator in the program. Neither of the methods perform well on the multi-hop questions, with a clear degradation in the performance for NS-VQA. This is because the question parser of NS-VQA relies on a Seq2Seq network that does not generalize compositionally [30]. CLEVR focus on visual attribute compositionality, and the multihop reasoning introduces higher demands on linguistic compositionality. When multihop questions are included in the training and validation data, naturally both methods improve their performance.
To gain further insight into CLIPs’ performance on CLEVR-Math, Appendix C shows a confusion matrix from training CLIP on 20 000 samples and evaluating on all question categories. These results show that most errors made by CLIP is of by ones. This reflects the generative nature of such models, in how they can get the context correct but sometimes miss out on details. We also see how CLIP focus on learning in the range 1-5, reflecting that these problems represent a majority of the problems.
Table 5 shows how diferent training sizes influence the accuracy. We can see that NS-VQA achieves high accuracy from relatively few examples and plateaus, which is consistent with the original results on CLEVR. It also seems like NS-VQA is overfitting with more data given, and one hypothesis is that more emphasise is put on the program, but that they are similar enough to confound NS-VQA. In CLEVR, the diferent questions were more distinguishable from a program perspective. CLIP scales with the number of samples, but plateaus at a much lower accuracy. We note that a larger number of samples could lead to similar performance for CLIP, but at the cost of more computational resources.
(a) Subtract all small purple matte blocks.
Subtract all blocks. How many objects are left? was answered by CLIP with 3 instead of 2.
(b) Subtract all red metallic objects. Subtract all yellow objects. How many objects are left? was answered with 9 instead of 5 by NS-VQA.
We randomly sample 20 correct and 20 incorrect answers from the multihop test data for both CLIP and NS-VQA. Appendix B contains a subset of those samples, and Figure 4 illustrates two incorrect answers. There are no clear patterns of failures, such as only performing one of the actions, but we notice multiple instances where CLIP fails to perform overlapping subtraction, or subtraction when no objects match the description. Another observation is that half of the 20 incorrect answers from CLIP, where on images with only three objects. Scenes with few objects have a much smaller possible action space associated to it, meaning that there is less room for error. In Figure 4a, there are no purple matte blocks to remove, so the corresponding equation is 3 − 0 − 1 = 2.
We introduced a new dataset, CLEVR-Math, containing word math problems about visual scenes. Our results show that the state-of-the-art NeSy model, NS-VQA, achieves higher accuracy on CLEVR-Math with less data and computational resources, than the neural model, CLIP. This is further evidence that neural methods, such as CLIP, are lacking in reasoning capabilities, even after fine tuning. Given that NS-VQA uses perfect scene graphs, the comparison is not completely fair. We still expect the results of learning end-to-end to be consistent with the current results in alignment with the original results on CLEVR for NS-VQA.
CLEVR-Math successfully introduces a focused benchmark for learning and reasoning in multimodal data. There are a few natural extensions to this work, both on further development of the dataset and on evaluation. Extending the benchmark to answers outside of the range 0-10 would provide a more challenging domain, and providing scene graphs for each step of the reasoning chain could open up for other methods. The empirical results show that neither of the models could generalize to chained actions. Hence, it is also of research interest to design neuro-symbolic models where language perception is tackled in a more generalizable manner. Focus should lie on the representations (symbols) that are learned. Other interesting directions is to introduce a representation that is manipulated internally according to the actions as they are read. Adding longer chains of operations, or chains with alternating subtraction and addition, would put even more emphasise on the reasoning capabilities. Finally, there is an opportunity to add confounding information to test the robustness, e.g. by associating each shape with a ifxed color during training and randomise it during testing.
This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. bilinear pooling for visual question answering and visual grounding, arXiv preprint arXiv:1606.01847 (2016). [23] P. Wang, Q. Wu, C. Shen, A. Dick, A. Van Den Hengel, Fvqa: Fact-based visual question answering, IEEE transactions on pattern analysis and machine intelligence 40 (2017) 2413–2427. [24] K. J. Shih, S. Singh, D. Hoiem, Where to look: Focus regions for visual question answering, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 4613–4621. [25] M. Narasimhan, S. Lazebnik, A. Schwing, Out of the box: Reasoning with graph convolution nets for factual visual question answering, Advances in neural information processing systems 31 (2018). [26] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., Learning transferable visual models from natural language supervision, in: International Conference on Machine Learning, PMLR, 2021, pp. 8748– 8763. [27] J. Mao, C. Gan, P. Kohli, J. B. Tenenbaum, J. Wu, The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision, arXiv preprint arXiv:1904.12584 (2019). [28] K. Yi, J. Wu, C. Gan, A. Torralba, P. Kohli, J. Tenenbaum, Neural-symbolic vqa: Disentangling reasoning from vision and language understanding, Advances in neural information processing systems 31 (2018). [29] K. He, G. Gkioxari, P. Dollár, R. Girshick, Mask r-cnn. corr abs/1703.06870, arXiv preprint arXiv:1703.06870 (2017). [30] B. Lake, M. Baroni, Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks, in: International conference on machine learning, PMLR, 2018, pp. 2873–2882.
B. Incorrect answers on multihop questions