Symbolic systems excel at reusing and composing modular functional units when solving problems such as simple analogical reasoning. However, they are less amenable to processing real-world data (e.g. images), and rely on additional (often hard-coded) mechanisms to convert such high-dimensional data to symbolic descriptions. In this work, we describe a modular approach 'Neural Analogical Reasoning' wherein elementary neural transformations operate and compose on distributed representations of high-dimensional inputs. We apply this approach on a class of visual analogical reasoning problems that involve discovering the sequence of transformations by which pairs of input-output images are related, so as to analogously transform future inputs. This can be viewed as a program synthesis task and solved via symbolic search if represented in symbolic form. Instead, we search for a sequence of elementary neural network transformations that manipulate distributed representations of the inputs. We present two variations of learning useful representations for this task and compare both with end-to-end meta-learning based approaches to demonstrate the importance of performing an explicit search.
Consider the class of visual reasoning problems as demonstrated in Figure 1 in which each task involves a functional analogy (i.e., ∶ ( ) ∶∶ ∶ ( ) ) [1]. Analogical reasoning in this context can be thought of as the reuse of some transformation and analogy making as the discovery of these transformations. By modelling these transformations as compositions of elementary functional units, we can cast analogy making into a program synthesis problem. Given a set of input-output pairs and a set of functional units , we must discover a composition = 1. 2. … . where ∈ such that ( ) = for all ( , ) ∈ . This program can then applied to a query image to generate an analogous output. The key questions to be answered in such a frameworks are how these elementary functional units are to be represented and how they can be ensured to be composable with each other and usable across a wide enough range
Classical reasoning systems can solve such problems via discrete search over compositions of elementary primitives [2, 3]. There have also been advances in However, they require both the inputs and the elementary primitives to be represented with predefined symbols. For the problem in Figure 1, the images would need to be explicitly annotated with discrete symbolic features such as the type of shape and its location in the image. Such use of predefined symbols allows classical reasoning systems to leverage compositionality and reuse computational units to perform robust reasoning. , , ) ) Task
Solution image pairs, generate the analogous output for the given query image.
However, it restricts their applicability in more complex domains where feature extraction may be non-trivial while also making it dificult to discover new knowledge from data.
Deep Neural Networks (DNNs), on the other hand, have shown an impressive ability to learn compressed distributed representations of high-dimensional data in complex domains such as images [4], text [5] and graphs [6]. Despite these successes, it remains unclear how to reason over such learned representations to solve analogy problems. In this paper, we take the approach of combining a discrete search procedure with neural primitives that manipulate distributed representations in a latent space. This allows us to leverage both the flexibility of DNNs and the ability of discrete search to construct arbitrarily complex compositions.
Our proposed system consists of (
that approximate the action of symbolic primitives within this representation space and that are composable with each other. Given a task as described in Figure 1, our system searches over sequences of such neural primitives to find a program that transforms the given input to the given output. By using a search procedure in place of a completely end-to-end learned system, we retain the flexibility of having independent functional units that can be reused, debugged and composed arbitrarily to build transformations in an interpret able manner. And using learned primitives means tedious hand-crafting of new symbols and procedures can be replaced with providing relevant examples of the transformation to be learned. Thus, our system aims to retain the power of compositional reasoning that symbolic primitives provide while being able to generalize beyond the practical capacity of a purely symbolic approach to richer inputs.
In such a framework, the ability of primitives to compose with each other while generalising to similar but unseen domains is essential for successful analogical reasoning. We obtain almost arbitrarily composable primitives by using neural networks that manipulate distributed representations within the same latent space ( ∶ → ). To learn these neural primitives as well as the latent space they operate on, we take inspiration from the Neural Algorithmic Reasoning framework [7], which describes how to neuralize an erstwhile symbolic algorithm (in our case, transformation) so that it can deal with rich inputs directly without requiring them to be explicitly transformed to a symbolic form. We also describe and evaluate two variations of learning distributed representations of the input - a guided approach in which representation space is derived from symbolic descriptions of the inputs and an unguided approach) in which the representation is learned directly from the high-dimensional inputs (in this case images).
Contributions Through this paper we (
Analogical reasoning has been considered a fundamental trait of human intelligence for the last half-century of AI research [8]. More recently, there has been a focus on characterising similar types of abstract reasoning as the fundamental building blocks of intelligence [9, 10, 11]. We refer to [1] for a review of various broad approaches to analogical making in AI systems, with our particular method building functional analogies. However, the ability to represent concepts in a modular way and to transfer them across domains still eludes many modern machine learning approaches [12]. Recent attempts at utilising neural networks for reasoning tasks focus on creating modular architectures of neural networks that can be learned end to end [13, 14, 10]. However, the reliance on fixed architectures limits the ability to learn and reuse functional units. Further, these approaches limit inference to applying the trained network once or at most a few times, rather than searching for the solution as is done in symbolic program synthesis [2, 3]. Our system is able to reuse neural transformation units in an unconstrained manner by exploiting the the ability of search to create arbitrarily complex compositions. This idea of inferring programs made up of neural modules has also been used for visual question and answering [15]. However this utilises a learned sequence model to generate programs and solves a classification task as opposed to a generative one.
The Neural Algorithmic Reasoning [7] framework takes a diferent approach to reasoning with neural networks. Instead of learning end-to-end on a particular task, it proposes to learn a symbolic algorithm as a neural network together with a high dimensional representation space on which it can act. Such a neural algorithm can then be utilised in various rich domains by learning an encoder and decoder for it. This approach has proven successful for learning to execute graph algorithms [16] and is even able to transfer knowledge to learn related graph algorithms [17]. We use a variant of this framework to construct neural networks that learn from input and output data to execute symbolic primitives over a learned representation space. We also investigate the properties of this representation space since learning disentangled representations has been shown to be an important for visual abstract reasoning [18, 19, 20].
Other symbolic approaches to analogical reasoning include learning to rank possible relations based on features from a relational database [21] as well as Structure Mapping [22] and more recent approaches [23] that use of graph representation learning to emulate Structure Mapping. Our method difers from such approaches by operating directly on images without requiring structured symbolic descriptions during test time. Our work also builds on a previous short abstract [24], specifically by exploring an unguided way of representation learning which enables the overall system to work without any symbolic descriptions at all.
Problem Setting We introduce a class of visual analogical reasoning problems as shown in Figure 1 to motivate our ‘neural reasoning’ system. Each problem consists of input-output image pairs which are related by an unknown transformation. Given these pairs, the task is to discover the transformation and apply it to a query image to generate the solution. The images considered consist of items from a set of 20 simple shapes and handwritten characters (from the Omniglot dataset [11]) placed on a 3 × 3 grid. The transformations between the input and output images are programs composed of either control flow operations ( r e s e t and o u t ) or spatial transformations of shapes in the image. The latter consist of positional shifts in the ordinal directions (e.g. s h i f t - r i g h t moves each shape in the image one grid-position to the right), shape conversions (e.g. t o - s q u a r e converts each shape in the image to a square), and afine transformations (e.g a f f i n e - r i g h t skews the the shape to the right).
This problem setting can be interpreted as solving
weakly supervised evaluation tasks which consists of a
input-output image pairs and a query image for which
we have to generate the output given some strongly
supervised training data consisting of similar input- (a) (b) (c) (d)
output pairs and the associated transformation
procedure. We utilise the strongly supervised data to learn: Figure 2: Example of applying various
(
In the guided approach, we aim to shape the latent space to include information about the input images that is relevant to the transformations, namely shape and position. We first train an autoencoder (Encoder: , Decoder: ) on symbolic descriptions of the input images present in the training data by minimising the negative log-likelihood loss between inputs and the reconstructed vectors. The symbolic descriptions used are concatenations of a one-hot vector denoting the shape present in the image and another denoting the position of this shape in the grid. We then train a CNN based encoder that takes images (denoted by s) as inputs to ift to the outputs of (with frozen weights) by minimising the mean-squared-error between the embeddings of images ( () ) and embeddings of the corresponding symbolic vectors ( () ): MSE ( ( () ), ( () )). To handle shapes at test time that were not seen during training, we also include an additional shape-label (u n s e e n ) and train to map shapes (whose labels weren’t included in the symbolic description) to the latent vectors generated by corresponding to the u n s e e n labels. While this approach ensures that the latent space encapsulates relevant information, it is limited by the reliance on symbolic descriptions. Addition of any new attributes of the input over which we would like to include transformations (e.g. a new shape, or continuous - such as afine - transformations of existing shapes) would require modifying the input space (size of one-hot input vector), requiring both latent space and transforms to be re-trained.
Learning the latent representation in an unguided way alleviates these limitations at the cost of not always capturing relevant information. In the unguided approach, we train a CNN based autoencoder ( , ) by minimising mean-squared-error for reconstructions: MSE ( ( ( () )), () )). We expect the autoencoder to generalize to unseen shapes and positions given enough data. One of the main issues encountered here is the accumulation of noise after applying each transform, which degrades compositional performance. Using a sigmoid activation after the final layer of the decoder drastically reduces noise as the image pixels are binary. Further details can be found in the appendix attached.
Transform Training Our system makes use of neural
networks to approximate the actions of the elementary spatial
transformations that make up the mapping between input and output
images. These neural primitives ∈ need to (
Program Execution We model mappings between input and output images in the visual analogy tasks as programs composed of primitives which are either control-flow operations ( r e s e t combine and o u t ) or spatial transformations. Figure 3 describes the execution of such programs. Since spatial transformations act independently on each shape, we first extract individual shapes by identifying connected components within the image. Each extracted image is converted into a latent vector using encoder and stored in the input and memory bufers. The program is executed sequentially with neural primitives being applied to vectors in the memory bufer and control flow operations indicating transfer of vectors between bufers. After execution is complete, the vectors present in the output bufer are converted to images using in the unguided case or by manual construction according to symbolic descriptions obtained using in the guided case. These are combined together to form the result. Further details can be found in [24] and the appendix. Searching for Solutions Given a representation space and a set of transforms we can solve the analogy tasks being considered by searching over possible programs. A naive breadth ifrst search has a branching factor 12 (4 shift operations, 4 change shape operations, 2 afine operations, r e s e t and o u t ), making it intractable for longer programs. Instead, we utilise the fact that both the input and output images are available, to extensively prune the search tree, details of which can be found in [24] and the attached appendix.
Compositional Abilities of Transformations A symbolic analogue to our approach would simply be a search over sequences of predefined hard-coded primitives. Our system uses learned primitives acting on a learned latent representation. Since these are an approximation to the underlying symbols, there may be errors in the system which lead to incorrect solutions or an inability to find a solution. To evaluate how well neural primitives in our system compose, we use of a dataset of input-output pairs, generated from 20000 random programs upto length 10 (details present in appendix), and compute the ratio of pairs for which a valid program is found.
The results are summarised in Figure 4. The guided
approach is able to find the correct solution programs 100
for 100% of the examples in the dataset. The unguided
approach shows similar performance with an overall e95
success rate of 94.7 %. The minor drop in performance trssa90
soyf munbgouliicdeadnncaontabtieoantstrdiuburitnegd ttroa(i1n)inthge,(a2b) sveenrcificeaotifotnhe cceuS85 Unguided Guided
of the generated output is performed using MSE with 80 1 2 3 4 5 6 7 8 9 10
the target, which is more susceptible to noise than com- Length of program
paring multi-hot vectors (used in unguided) and (
sented in the paper can be thought of as a few-shot learning problem, where an output has to be found for a query given input-output examples - a setting well suited for end-to-end meta learning algorithms. In order to demonstrate the significance of discrete search in performing compositional reasoning, we compare our method against three end-to-end NN based metalearning algorithms - Conditional Neural Processes (CNP) [25], Matching Networks (MAN) [26],
Model-Agnostic Meta-Learning (MAML) [27] - on a few-shot classification version of the analogical reasoning task (details present in attached appendix). From the results in Table 1, we can see that despite being presented with a more dificult task (generation vs classification), our approach is able to significantly outperform end-to-end meta learning techniques. Representation Learning The learned representation space upon which compositional reasoning is being performed is a critical part of our system. The experiments discussed above demonstrate that this space can be both disentangled enough to support robust transformations, as well as diverse enough to support interpolation and generalisation. To explore this further, we visualise the latent embeddings of images with a single shape in Figure 5. We can see that for both the guided and unguided cases, the representation space is strongly structured around the grid positions of the shapes in the images. For unguided, the clusters are more sparsely structured, indicating the presence of another component of variation that captures more shape information. Finally, Figure 5c shows that afine transformations of shapes are closer to each other than to the original shapes, indicating that a f f i n e transforms causes a shift in the distribution of latent embeddings. This explains why excluding afine transformations of seen shapes when training transforms causes reduced performance. Such transforms being applied to afine shapes would introduce significant noise since they are not aware of this distribution shift. This highlights the importance of having a well structured latent space. Generalisation To be of practical use for computation and reasoning, our system needs to be able to generalise to visual analogy tasks with similar images but with characteristics not
Regular 100% 94.7%
Transforms 95% 94.8% present in the training data. To evaluate this, we exclude images containing 5 (of the 20 total) shapes from the training data, but include these images during testing (performed similar to section 4). We evaluate generalisation for the transforms and representation space (encoder and decoder) separately. The results in Table 2 indicate that both the guided and unguided approaches are able to generalise well to shapes not present during training. This suggests that with a diverse and large enough training set, our representation space could be rich enough to support spaces with a large variety of inputs.
In the unguided approach, s h i f t transforms work robustly for images containing shapes not seen during training, indicating that the representation space captures information about position in a disentangled manner. The a f f i n e transformations, which manipulate the shape itself, also work well for shapes unseen during transform training, but only if the afine versions of seen shapes are seen when learning the representation space. This is explained by the distribution shift in latent embeddings caused by a f f i n e transformations as seen in Figure 5c and discussed in section 4. Incorporating explicit measures to disentangle the distributed representations with respect to continuous shape transformations, in addition to shape vs position, may alleviate this need, enabling robust system-level generalization. More generally, these results suggest that representations learned directly and without supervision can capture relevant information in a distributed enough manner to support reasoning within our framework; and that providing explicit symbolic guidance aids in compositionality with some loss in generalizability. Conclusion Our neural analogical reasoning approach applies compositional reasoning via search on transformations of distributed representations, both learned via neural networks, thus substituting handcrafting of symbols and procedures with provision of the relevant training data. Experiments show that our approach generalises to data drawn from related but diferent distributions (i.e., unseen shapes). Future work can tackle more complete representation learning methods to avoid extraction of connected components, enabling the system to tackle more complex problems with overlapping and arbitrarily sized shapes. We envisage a more general system for analogical reasoning (in the limited sense used here) based on our NAR approach: one that can operate on rich input distributions while continuously learning to detect when new shapes and/or transformations are encountered and retrain its elements (representation space and/or new transformation network(s)) appropriately.
This work is supported by “The DataLab” agreement between BITS Pilani, K.K. Birla Goa Campus and TCS Research. We thank Tirtharaj Dash and Ashwin Srinivasan for their input. helpful for abstract visual reasoning?, in: NeurIPS, 2019. [19] X. Steenbrugge, S. Leroux, T. Verbelen, B. Dhoedt, Improving generalization for abstract reasoning tasks using disentangled feature representations, ArXiv abs/1811.04784 (2018). [20] F. Hill, A. Santoro, D. Barrett, A. Morcos, T. Lillicrap, Learning to make analogies by contrasting abstract relational structure, in: International Conference on Learning Representations, 2019. URL: https://openreview.net/forum?id=SylLYsCcFm. [21] R. Silva, K. A. Heller, Z. Ghahramani, Analogical reasoning with relational bayesian sets, in: AISTATS, 2007. [22] D. Gentner, Structure‐mapping: A theoretical framework for analogy*, Cognitive Science 7 (1983) 155–170. [23] M. Crouse, C. Nakos, I. Abdelaziz, K. D. Forbus, Neural analogical matching, ArXiv abs/2004.03573 (2021). [24] A. Sonwane, G. Shrof, L. Vig, A. Srinivasan, T. Dash, Solving visual analogies using neural algorithmic reasoning, CoRR abs/2111.10361 (2021). URL: https://arxiv.org/abs/2111.10361. a r X i v : 2 1 1 1 . 1 0 3 6 1 . [25] M. Garnelo, D. Rosenbaum, C. Maddison, T. Ramalho, D. Saxton, M. Shanahan, Y. W.
Teh, D. Rezende, S. M. A. Eslami, Conditional neural processes, in: J. Dy, A. Krause (Eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, PMLR, 2018, pp. 1704–1713. URL: https: //proceedings.mlr.press/v80/garnelo18a.html. [26] O. Vinyals, C. Blundell, T. Lillicrap, k. kavukcuoglu, D. Wierstra, Matching networks for one shot learning, in: D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, R. Garnett (Eds.), Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016. URL: https://proceedings.neurips.cc/paper/2016/file/ 90e1357833654983612fb05e3ec9148c-Paper.pdf. [27] C. Finn, P. Abbeel, S. Levine, Model-agnostic meta-learning for fast adaptation of deep networks, in: Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML’17, JMLR.org, 2017, p. 1126–1135.
A.1. Problem Setting
Types Transformations Meaning Shifts usphi,fsth-irfitg-hdto,wnshift-left, shift- oMnoevgeraidll poofstihteiosnhsapes in the image by Shape conver- to-square, to-circle, to-delta, Convert all of the shapes in the image sions to-triangle to a particular shape Afines affine-right, affine-left tSoketwheallelft tohrersighhatpes in the image either
task consists of shapes or characters from the set described in Figure 6 placed on a 3 × 3 grid. The size of each such image is 64 × 64 pixels with the the size of shapes and characters placed inside being 20 × 20 pixels. Note that the grid isn’t explicitly drawn and all images are black and white (the background being white and the shapes being solid black). The handwritten characters used have been taken from the Omniglot dataset.
A.2. Guided Representation Space 1 2
P S
P S
Multi-Hot with = Bheight + Bwidth Position Slots,
N Shape Slots Representation Learning For the guided representation space we first train an autoencoder directly on the symbolic descriptions (as described in Figure 7) of images containing a single shape followed by training an encoder from said images to the latent embeddings of their symbolic descriptions. This procedure is described in Figure 8 The size of the latent embeddings here is 32. The encoder and decoder used for the autoencoder over symbolic descriptions are both MLPs with a single hidden layer of size 16. They are trained using batch size of 36 for 10,000 weight updates with a learning rate of 0.0003. For the encoder from images to the latent embeddings of their symbolic descriptions we use a CNN with 6 convolutional layers each using a stride of 2, padding of 1 and 3 × 3 kernels. Each layer doubles the number of channels and halves the dimensions of its input and applies a rectified linear unit activation. Since the input image has dimensions 1 × 64 × 64, the output of these six layers is 128 × 1 × 1. This is flattened and passed through a final linear transformation to output the latent embedding of size 32. This encoder is trained with a batch size of 180 (since there are 20 × 9 = 180 possible combinations of shape and grid positions) for 50,000 weight updates with a learning rate of 0.0003. Neural Transforms For the guided representation space, we use MLPs as neural transforms with a single hidden layer of size 32. These are trained with a batch size of 180 for 5000 weight updates with a learning rate of 0.0003. A overview of the training procedure is given in Figure 9.
EX
DS A.3. Unguided Representation Space Representation Learning For the unguided representation space we train an autoencoder directly on the images containing a single shape with latent embedding size of 16 × 8 × 8. The encoder used is a CNN with 3 convolutional layers each followed by a batch normalisation layer. Each layer uses a stride of 2 and padding of 1. The first two layers use 4 × 4 kernels and the last one uses 3 × 3 kernel. The first layer outputs 8 channels while the second and third both output 16 channels. The decoder is made up of 3 transpose convolution layers each followed by batch normalisation. All three layers use a stride of 2, padding of 1 and 4 × 4 kernels. The first two layers output 8 channels. The autoencoder is trained using batch size of 32 for 20000 weight updates with a learning rate of 0.001.
Neural Transforms For the unguided representation space we use MLPs as neural transforms with a single hidden layer of size 256 for shift and shape conversion transforms and of 2048 for afine transforms (although similar performance is observed for size 256). These are trained with a batch size of 18 for 4000 weight updates with a learning rate of 0.001 on dataset of images with a single shape present. Note that the latent embeddings are flattened to a 1024 long vector before being passed to the transforms.
A.4. Program Execution Algorithm Algorithm 1 Program Execution Input: Input latent vectors s, program Output: Output latent vectors 1: input = { }, memory = { }, output = ∅ 2: for = 1 ∶ length( ) do 3: if [] = o u t then 4: output ← output ∪ memory 5: else if [] = r e s e t then 6: memory ← input 7: else 8: memory ← { []( ), ∀ ∈ memory} 9: end if 10: end for 11: return output A.5. Search Algorithm Algorithm 2 Limited Depth BFS Input Encoder: , Decoder: Transforms: = { }, Example: = ( , ) Output Program satisfying all given examples 1: targets ← connected_components( ) 2: q ← LIFO(); q.push([ ]) 3: max_count ← 0 4: while True do 5: 6: 7: 8: 9: 10: 11: 12: 13: 14:
end if end for 15: end while program ← q.pop(); if length(program) = then
return None; end if for ∈ do new_program ← program + ; if s a t i s f i e s ( , new_program, , ) then
return new_program;
Alongside pruning we also reduce the constant factor involved at each step of the search by keeping track of the program state after partial execution. This creates a significant reduction in time taken since the majority of it is the neural network forward passes. This means that transforms are only applied when a new transform is added to the program. We do the same for output decoding - only applying output decoder whenever the o u t primitive is called and keeping track of output bufer in subsequent nodes. A.6. Example Run-through Consider the example task in the shown in Figure 10. An input image and target image makes up the example for an unknown transformation sequence which needs to be applied to the query image . We have an encoder and decoder trained as described in section 3 and a set of transforms trained according to section 3.
Solving this task using the guided approach, the first step is isolating connected components
in the input image. This results in two images for the input in our example
which is converted into its latent representation using the encoder and stored in the input and
(
(
Suppose that the search arrives at s h i f t - l e f t r e s e t s h i f t - d o w n o u t . We first apply the neural
network shift−left to the latent vectors present in the memory bufer ′(
′(
Examples , Primitives sshhiifftright shifttrriight shiftrgihghthtt + shiftrig reset + out
Program shiftdown shiftdown affine-right
out search execute
Query
Result
the original image but shifted to the left by one grid position.
r e s e t is a special primitive that
results in the memory bufer being reset with the contents of the input bufer:
= = {
The application of s h i f t - d o w n proceeds similarly to that of s h i f t - l e f t using the appropriate
neural network transform. Finally o u t is another special primitive which adds all the latent
(
We cast our analogical reasoning task into a few shot classification problem - each task consists of 6 examples representing a single transformation, where each example consists of an input image, 4 options for the possible outputs and an index indicating the correct output as shown in Figure 11.
The task also contains a query image and 4 option images for that query. This is further deconstructed into a binary classification problem - for each example, we compute the pixel diference between the input image and each of the 4 option images. Each such pixel diference is assigned a binary label resulting in a 6 × 4 = 24 meta-train examples and 4 meta-test queries for each task.
We used custom made Convolutional Neural Networks to get the features of the input images. CNP and MAN were run for 60 epochs whereas MAML is run for 30 epochs. MAML uses a dropout of 0.5 and CNP uses a dropout 0.8. We used ReduceLROnPlateau learning rate scheduler with a threshold of 10−3 modifying the other parameters as required. Both Dropout and Learning rate scheduler were essential to reduce the fluctuations in the training loss.
C.1. Evaluations of the representation space
100 95 80 C.2. Ability of Neural Transforms to Compose 1 2 3 4 5 6 7 8 9
10
Length of program
C.3. Evaluation of Guided Representation Space Generalisation To evaluate how well the guided representation space (encoder and decoder) generalises to unseen situations, we utilise the following scheme. Let us a tuple ( , , , ) ∈ () 4 where is the set of all shapes as described in Figure 6, denotes the shapes present in the training data for the latent space (here | | + 1 + 6 denotes the size of the multi-hot vector, where 6 is for the positional encoding and 1 is for a slot of unseen shape), denotes the shapes present in the training data for transforms, denotes the shapes present in the training data for the encoder and denotes the shapes present in the examples on which the latent space is being evaluated. We perform the following 4 experiments based on this evaluation scheme to test the generalisation capabilities of the guided representation space. The results are given in Figure 15. Note that the total number of shapes possible || = 20 .
1. For = = ∅ (all shapes considered unseen in the multi-hot) and = (all shapes included in test examples), we vary the shapes shown to the CNN encoder during training from || = 1 (just one shape being included in training data) = (all shapes included) and record the evaluation performance. 2. For = = ∅ (all shapes considered unseen in the multi-hot), we vary the shapes shown to the CNN encoder during training from || = 1 (just one shape being included in training data) = (all shapes included) and record the evaluation performance of with this encoder on = ′ (test only including shapes not present in train for encoder). 3. For = = { square, triangle, circle, delta} (four shapes + unseen in the multi-hot) and = (all shapes included in test examples), we vary the shapes shown to the CNN encoder during training from || = 1 (just one shape being included in training data) = (all shapes included) and record the evaluation performance. 4. For = = { square, triangle, circle, delta} (four shapes + unseen in the multi-hot), we vary the shapes shown to the CNN encoder during training from || = 1 (just one shape being included in training data) = (all shapes included) and record the evaluation performance of with this encoder on = ′ (test only including shapes not present in train for encoder).
C.4. Evaluation of Transform Generalisation in Guided Representation Space To evaluate how well neural transforms learned in the guided representation space generalise to unseen situations, we first learn the representation space with data containing all of the shapes or positions. We then train the neural transforms on (input, output) image pairs containing only a subset of shapes (or input positions). During testing, we include all shapes (or positions). This can be thought of as keeping = = = while is varied from to ∅. If the transforms give similar performance to those trained with all the data, we can say that they, together with the latent space, are generalising well. The results are given in Figure 16. C.5. Evaluation of Unguided Representation Space Generalisation 100 95
With affine
Without affine 1 2 3 4 5 6 7 8 9
10
Length of program
C.6. Evaluation of Search Performance