Introduction

In many situations, concepts that pertain to one set of data can also be relevant to another [1,2]. Take, for instance, the general concept of ordinality, whose semantics are defined by relations: isSuccessor, isPredecessor, isGreater, isLess and isEqual; together with their constraints. Successfully capturing this concept involves learning the corresponding relations such that they maintain data set and property independence, with no retraining. This is to say that they have been abstracted from the specific property and act instead as a generic set of characterizing relations for the semantics of ordinality. For this, we argue that the relations must be consistent with their expected constraints and coherent across ordinal properties spanning different data sets, which means their consistency is maintained regardless of data set or particular ordinal property.

As a concrete example, suppose that we have successfully learned to order images of numbers by their abstract digit identity, and are presented with a new data set containing images of individual stacks of blocks. Suppose then that we wish to obtain an ordering over them, such that we can compare arbitrary data instances using the above relations. Provided that the learned relations are consistent with their expected constraints, it should be possible to obtain an encoding that establishes each successor, via our isSuccessor relation, and immediately be able to compare data instance over the remaining relations. Following this logic, the primary purpose of this paper is to evaluate under which conditions a relation-decoder model is able to obtain the ordinality concept. We do this by taking a set of popular relation-decoder models, including a proposed Dynamic Comparator (DC) model, and assess 1. their consistencies as measured in the source data set, and 2. their ability to perform a Partial Relation Transfer (PRT) task to a novel target data set, which measures the robustness of their consistencies across domains. The evaluation takes place in two steps. In the first, we learn the above set of ordinality relations by ordering MNIST images based on their abstract digit identity and report each model's consistency profile. In the next step, we take the now pretrained isSuccessor relation-decoder and apply it to a proposed BlockStacks data set, which consists of images of multicolored block stacks. Each stack contains a single red block at various heights, which we use to test the degree to which ordering the encodings of each block stack image, subject to the pretrained isSuccessor relation, leads to transferred prediction accuracy across the remaining relations. In summary, the contributions of our work are:

• We devise an experimental setup that can expose the degree to which learning relations leads to concept abstraction, together with a new BlockStacks data set that presents a challenging ordering task based on a complex property. • We introduce a set of data set agnostic characteristic measures for relation-decoders which can help determine their ability to perform PRT. • We present a Dynamic Comparator model that achieves excellent PRT.

• Finally, we present a comprehensive analysis of model characteristics against corresponding PRT performance, for a set of popular relation-decoders.

The rest of the paper is presented as follows. Section 2 firstly positions our paper with respect to related work. Section 3 formalises the PRT task and outlines the architecture we employ to solve it, including the proposed DC relation-decoder model. We then define how we compute model consistency and gradient-conformity in Section 4. Finally, we provide results and analysis in Section 5, with concluding remarks in Section 6.

Related Work

Relational representations play a prominent role in Knowledge Graph Embedding (KGE), wherein sets of relation-decoders are jointly learned, through triplet link prediction, in order to obtain a semantic latent factor representation for entities [3,4,5,6,7,8,9,10,11]. In principle any KGE link prediction model can be employed in this work, but we focus on those that assume a Euclidean representation space and do not require any additional per-triplet engineering. Although KGE methods typically do not use a shared auto-encoder as we do in this paper, Schlichtkrull et al. [12] did adopt an auto-encoding framework, where a graph neural network is used as the encoder, however they did not work with visual data and the model was not applied to transfer. Disentanglement, which also aims to learn semantic representations for data is of relevance to this work [13,2], wherein multiple methods have been proposed, for example using Generative Adversarial Networks [14] and VAEs [15,1,16,17,18,19,20]. Of particular relevance to our work are investigations looking at the transferability of disentangled representations [21,22,23], but these did not include relation learning. A bridge between relation learning and disentanglement, wherein relation-decoders are employed as a semisupervision to VAEs, can be found in [24,25,26]. Lastly, we note that our experimental setup is most remnant of domain adaptation [27]. To the best of our knowledge, no work has compared relation-decoders in their ability to abstract concepts, as measured by their consistency and its transfer across domains.

The Partial Relation Transfer Task and Model

Partial Relation Transfer (PRT) is at its core a domain adaptation task [27], wherein we have a source and target data domain, consisting of a set of images, 𝑋 𝑠 and 𝑋 𝑡 , respectively, and a set of shared relation prediction tasks, ℛ = {𝑟 1 , … , 𝑟 𝑛 }. We approximate each relation using a relation-decoder

𝜙 𝑀 𝑟 ∶ 𝑍 × 𝑍 → [0, 1],

where 𝑍 denotes a latent space that contains all image encodings 𝑧 𝑖 ∈ 𝑍. The superscript 𝑀 denotes a specific relation-decoder model, as we test multiple variants. To obtain embeddings we use a domain-specific auto-encoder, consisting of an encoder 𝜓 𝑠/𝑡 𝑒𝑛𝑐 ∶ 𝑋 → 𝑍 and decoder 𝜓 𝑠/𝑡 𝑑𝑒𝑐 ∶ 𝑍 → 𝑋, which helps to minimise information loss through reconstruction of the input image 1 .

The evaluation takes place as a two-step procedure. In the first, all relation decoders are trained in the source domain, as a semi-supervision to the auto-encoder, using available labels, 𝑦 𝑠 ∈ ℝ |ℛ|×|𝑋 𝑠 |×|𝑋 𝑠 | , that specify whether a relation 𝑟 ∈ ℛ holds between image 𝑥 𝑖 , 𝑥 𝑗 ∈ 𝑋 𝑠 . Here, | ⋅ | denotes the cardinality of the operand set, but in practice we only use a small fraction of the available labels. In the second evaluation step, we initialise a new auto-encoder to be applied to the target dataset and use a subset of the pretrained relation-decoders, with labels 𝑦 𝑡 ∈ ℝ |ℛ|×|𝑋 𝑡 |×|𝑋 𝑡 | , to act as fixed-parameter 'guides' for the encoder.

To obtain informative data encodings, we use a Variational AutoEncoder (VAE), specifically the 𝛽-VAE, given its simplicity and demonstrated ability to separate distinct factors in the latent representation [1,15,28]. The 𝛽-VAE achieves this by optimising the ELBO objective, which for the purposes of this paper we express as a loss over both encoder and decoder:

ℒ 𝐸𝐿𝐵𝑂 𝛽-VAE = ℒ (𝜓 𝑠/𝑡 𝑒𝑛𝑐 , 𝜓 𝑠/𝑡 𝑑𝑒𝑐 ) + 𝛽ℒ (𝜓 𝑠/𝑡 𝑒𝑛𝑐 , 𝒩 (0, 𝟙)),(1)

where an additional 𝛽 scalar hyperparameter is used to influence disentanglement through stronger distribution matching pressure to an isotropic zero-mean Gaussian prior, 𝒩 (0, 𝟙).

When 𝛽 = 1 we obtain the original VAE objective [28]. We provide the full ELBO loss, with a detailed explanation, in Appendix B. Each experiment involves taking embeddings from a corresponding encoder and passing them through to sets of relation-decoders (either the full set in the case in the source domain, or only a subset in the target domain). We can treat each relation-decoder as producing a prediction ŷ 𝑟𝑖𝑗 for whether relation 𝑟 holds between data instances 𝑖 and 𝑗 [5]. Using the ground truth 𝑦 𝑟𝑖𝑗 , we can then compute the loss over all relationdecoders, ℒ 𝑅𝑒𝑙𝐷𝑒𝑐 , as the binary cross-entropy of prevision versus ground truth. This gives us the final joint objective between VAE and relation-decoders:

ℒ 𝑗𝑜𝑖𝑛𝑡 = ℒ 𝐸𝐿𝐵𝑂 𝛽-VAE − 𝜆 𝔼 𝑟,𝑦 𝑟𝑖𝑗 ,𝑧 𝑖 ,𝑧 𝑗 [𝑦 𝑟𝑖𝑗 log( ŷ 𝑟𝑖𝑗 ) + (1 − 𝑦 𝑟𝑖𝑗 ) log(1 − ŷ 𝑟𝑖𝑗 )] ⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟ ℒ 𝑅𝑒𝑙𝐷𝑒𝑐 , (2)

where 𝜆 is a scalar weighting parameter.

Dynamic Comparator

In our analysis, we include a proposed low-complexity, but nonetheless expressive, "Dynamic Comparator" (DC) model, which is designed to model systems of relations, whilst encouraging desirable properties for PRT. The overall DC model is composed of two modes, a distance-based measure, 𝜙 † 𝑟 , that can compute how close the vector difference between two inputs is to a positive or negative valued reference vector, and a step-like function, 𝜙 ‡ 𝑟 , that determines the sign of the difference between two points, optionally with an offset. The overall DC model is given by2 :

𝜙 𝐷𝐶 𝑟 (𝑧 𝑖 , 𝑧 𝑗 ) = 𝑎 0 ⋅ 𝜎 0 (𝜂 0 (‖𝑢 ⊙ (𝑧 𝑖 − 𝑧 𝑗 + 𝑏 † )‖ 2 2 )) ⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟ 𝜙 † 𝑟 +𝑎 1 ⋅ 𝜎 1 ((𝜂 1 ⋅ 𝑢 ⊤ (𝑧 𝑖 − 𝑧 𝑗 + 𝑏 ‡ ))) ⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟ 𝜙 ‡ 𝑟 . (3)

where 𝑎 = S o f t m a x (𝐴) ∈ ℝ 2 is an attention weighting between the two modes, and ensures that 𝜙 𝐷𝐶 is bound to [0,1]. 𝜎 0 , 𝜎 1 are an exponential and sigmoid function, respectively; 𝑢 = Softmax(𝑈) ∈ ℝ 𝑚 is an attention mask which is applied to 𝑚-dimensional latent embeddings; 𝑏 † , 𝑏 ‡ ∈ ℝ 𝑚 are learnable bias terms that enables an offset to each mode; and 𝜂 0 ∈ ℝ + are non-negative and 𝜂 1 ∈ ℝ any-valued scalar terms, respectively. Lastly, ⊙ denotes the Hadamard product (elementwise multiplication) and ‖ ⋅ ‖ 2 is the 𝐿2-norm. Due to a convergence issue when using a pretrained DC with fixed parameters, we needed to use a flexible fitting procedure in which we enable the DC parameters to train in the target domain, but with the additional loss term ‖𝜌 * − 𝜌‖, between pretrained 𝜌 * and untrained parameters 𝜌, respectively. In all cases we evaluated the final parameter values in the target domain and found them to be approximately equivalent to the 𝜌 * . We did not apply this method to the other models as they were all able to fit the isSuccessor relation in the target domain.

Measuring relation-decoder characteristics

In this section we describe a series of measures that we use to understand more about the intrinsic characteristics of each relation-decoder, which together help identify the behaviour of each relation-decoder model and provide insight regarding their respective PRT performance.

For any system of relations, we can write down a truth-table that defines the valid truthstates that they may collectively take, which we expect our relation-decoders to model. For example, we know that any time isGreater is true, isLess must not be. By assuming that each relation-decoder output is pairwise conditionally independent given 𝑧 𝑖 , 𝑧 𝑗 , for instance,

𝑝(isGreater, isLess|𝑧 𝑖 , 𝑧 𝑗 ) = 𝑝(isGreater|𝑧 𝑖 , 𝑧 𝑗 )𝑝(isLess|𝑧 𝑖 , 𝑧 𝑗 ),

we can produce a probability statement for whether the relations are consistent with valid entries to the truth-table . Taking 𝑟 1 = isGreater and 𝑟 2 = isLess as our entire system of relations, we can produce the following truth-table conversion, where invalid entries are omitted:

𝑟 1 (𝑥 𝑖 , 𝑥 𝑗 ) 𝑟 2 (𝑥 𝑖 , 𝑥 𝑗 ) ℱ (𝑟 1 , 𝑟 2 ) 𝑇 𝐹 𝑇 𝐹 𝑇 𝑇 𝐹 𝐹 𝑇 ⟹ ℱ (𝑟 1 , 𝑟 2 ) = ∀𝑥 𝑖 , 𝑥 𝑗 ((𝑟 1 (𝑥 𝑖 , 𝑥 𝑗 ) ∧ 𝑟 2 (𝑥 𝑖 , 𝑥 𝑗 )) ∨(¬𝑟 1 (𝑥 𝑖 , 𝑥 𝑗 ) ∧ 𝑟 2 (𝑥 𝑖 , 𝑥 𝑗 )) ∨(¬𝑟 1 (𝑥 𝑖 , 𝑥 𝑗 ) ∧ ¬𝑟 2 (𝑥 𝑖 , 𝑥 𝑗 )))(4)

which, using our relation-decoders for each relation and with 𝑧 𝑖,𝑗 = 𝜓 𝑒𝑛𝑐 (𝑥 𝑖,𝑗 ) and ¬𝜙 𝑟 (𝑧 𝑖 , 𝑧 𝑗 ) = 1 − 𝜙 𝑟 (𝑧 𝑖 , 𝑧 𝑗 ), we express the probability of ℱ being true as:

𝑝(ℱ |𝑧 𝑖 , 𝑧 𝑗 ) = ((𝜙 𝑟 1 (𝑧 𝑖 , 𝑧 𝑗 ) ⋅ 𝜙 𝑟 2 (𝑧 𝑖 , 𝑧 𝑗 )) + ((1 − 𝜙 𝑟 1 (𝑧 𝑖 , 𝑧 𝑗 )) ⋅ 𝜙 𝑟 2 (𝑧 𝑖 , 𝑧 𝑗 )) + ((1 − 𝜙 𝑟 1 (𝑧 𝑖 , 𝑧 𝑗 )) ⋅ (1 − 𝜙 𝑟 2 (𝑧 𝑖 , 𝑧 𝑗 ))).(5)

Finally, since ℱ should hold for all input combinations, we heavily penalise violations by using a binary cross-entropy loss between ℱ and the expected outcome:

𝐻 𝑇 𝑟𝑢𝑒 (𝑝(ℱ )) = − 1 𝑁 ∑ 𝑧 𝑖 ,𝑧 𝑗 ∈𝑍 1 ⋅ log 𝑝(ℱ |𝑧 𝑖 , 𝑧 𝑗 ),(6)

where 𝑍 is the latent space, as we can compute this score for any samples from this space 3 and 𝑁 is a normalising constant, equal to the number of 𝑧 𝑖 and 𝑧 𝑗 sample pairs used in the calculation. We refer to this measure as Con-A referring to the fact that we use it to measure consistency across multiple relations.

To provide a deeper understanding about how relation-decoders collectively interact with their inputs, we use a gradient evaluation to see whether models respond similarly to changes in their input. For a set of relations, we define the gradient-conformity (GC) of relation 𝑟 𝑖 against all others by the following cosine-similarity:

𝐺𝐶 = | 𝑑 𝑇 𝑖 𝑑 𝑗 ||𝑑 𝑖 || 2 ||𝑑 𝑗 || 2 | where 𝑑 𝑖 = 𝑑𝜙 𝑟 𝑖 𝑑𝑧 𝑐 | 𝑧 𝑐 =𝑧 𝑐 𝑠 and 𝑑 𝑗 = 𝑑𝜙 𝑟 𝑗 𝑑𝑧 𝑐 | 𝑧 𝑐 =𝑧 𝑐 𝑠 , ∀𝑖 ≠ 𝑗(7)

where | ⋅ | denotes the absolute of the operand and 𝑧 𝑐 is the concatenation of each relationdecoder's inputs, with gradients evaluated at reference inputs 𝑧 𝑐 𝑠 . GC will be 1 if gradients are aligned and zero if orthogonal 4 .

Results

This section presents results for the PRT task on a range of relation-decoder models. In the source domain, we learn a system of binary relations: ℛ ={isSuccessor (S), isPredecessor (P), isGreater (G), isEqual (E), isLess (L)}, on digits represented in MNIST images, alongside a 𝛽-VAE. In the target domain, we take the pretrained S relation as a fixed-parameter guide for a new 𝛽-VAE applied to BlockStacks images (see Appendix A for BlockStacks image examples), and then evaluate PRT accuracy on the held-out G, E, L and P relations. Relation-decoder models compared here are: TransR [29], HolE [30], NTN [3], our proposed DC and a basic neuralnetwork baseline, NN. NN is a simple four-layer (𝑙 in , 𝑙 1 , 𝑙 2 , 𝑙 out ) neural-network with layer sizes 𝑙 in = 2𝑑 𝑧 , 𝑙 1 = 2𝑑 𝑧 and 𝑙 2 = 𝑑 𝑧 , with ReLU activations. The final output layer 𝑙 out is a single value passed through a sigmoid function, to bound the output to [0,1]. Further model details are provided in Appendix C.

We vary 𝛽 only in the source domain, ranging across values {1, 4, 8, 12}, but fix it in the target domain. 𝜆 is fixed in both domains (see Appendix C.3 for further details on hyperparameter settings). For Con-A and GC measures, we produce encodings for three data splits: dataembeddings, where all inputs are encodings of a domain's test data; interpolation, where we obtain an empirical mean and variance for the domain's data-embeddings and sample from a corresponding Gaussian distribution; and extrapolation, where we sample from regions strictly outside the data-embeddings region.

Figure 2-top provides relation-decoder prediction accuracy in both the source MNIST (left), and target BlockStacks (right), domains. Key observations are that DC produces excellent PRT performance, whilst NN, NTN and HolE all see some degradation from their source accuracies. TransR seems to maintain a similar accuracy profile. We include 𝛽's impact on these performances in Figure 2 domain, PRT performance is significantly impacted by 𝛽 in all models, but has little effect in the source domain. Additionally, TransR has a strong positive correlation with 𝛽, whereas NN, NTN and HolE produce the best PRT performance with intermediate disentanglement pressure. To interrogate further how 𝛽 affects each model, we provide: (Figure 3-top) mean relation Con-A referenced to both source (left) and target (right) domain embeddings; and (Figure 3-bottom) source domain referenced GC measures for each model. In the left and bottom plots, blue (left group), green (middle group) and red (right group) show results for the data-embeddings, interpolation and extrapolation regions of latent space, in respective order. From the source domain Con-A results, we note that DC shows excellent consistency across relations in all regions. Most other models have worse interpolation and extrapolation consistency. Increasing 𝛽 appears to give some improvement for all but HolE, but there are indications that this trend does not persist into the largest 𝛽 = 12 value. Interestingly, Con-A values for target data-embeddings (right) are notably worse than for source data-embeddings, with values closer to those for interpolation or extrapolation in the source domain. For GC, DC performance is close to 1 for all 𝛽 with no discernible change. All other models show a weaker GC with positive correlation between GC and 𝛽. TransR and NN achieve significantly higher GC than NTN and HolE.

Key Experimental Results

Does good source task accuracy lead to successful PRT?

Since we transfer pretrained models from source to target domain and ensure that the target encoder, 𝜓 𝑡 𝑒𝑛𝑐 , fits its encodings to S, we might expect that relation-decoding performance will be the same in both domains. However, despite DC, NN, NTN and HolE all performing close to 100% accuracy, and TransR achieving above 80%, across all relations, and with all relations able to achieve similar prediction accuracy (or better in the case of DC) on the guide relation S, PRT performance varies significantly across models. It is firstly evident that DC is successful at PRT, sustaining approximately 100% accuracy across all held-out relations. NN achieves mostly good performance, with greater degradation across P and E relations. Although HolE and NTN both achieve good PRT for P, there is increasing degradation across E and G, L relations. TransR is able to achieve strong relative performance where PRT accuracy per relation is comparable to what was possible in the source domain. These results indicate that source accuracy alone is not enough to determine whether models will be successful at PRT.

How does 𝛽 affect Con-A and GC and how does this impact model coherence?

To provide an overview of how increased disentanglement pressure affects each model we can firstly compare how 𝛽 affects model performance in both source and target domain. Figure 2-bottom demonstrates that, although relation prediction accuracies for most models either do not respond, or respond negatively, to increases in 𝛽 in the source domain, their PRT behaviour differs significantly across models: DC shows no discernible change, whilst NN, NTN and HolE all show a parabolic response with a maximum PRT around 𝛽 = 8; TransR shows a general positive correlation but with diminishing returns above 𝛽 = 8. To gain further insight into the role of disentanglement pressure, it is necessary to look at how each model's intrinsic behaviour responds to 𝛽 changes.

First, we attempt to expose the relationship between 𝛽 and consistency and whether this has any effect on PRT performance. By Figure 3-top, DC clearly outperforms all other models on Con-A and this coincides with better PRT performance. The next best performing model on Con-A in the source domain is also the next best on PRT performance. In most cases Con-A degrades for all models when moving from data-embeddings to interpolation and extrapolation, but the degree of degradation changes depending on the model. Interestingly, across all models, their target Con-A is notably close to that of interpolation or extrapolation in the source domain analysis. This suggests that guiding 𝜓 𝑡 𝑒𝑛𝑐 to fit relation S produces data embeddings that lie in the interpolation or extrapolation regions with respect to MNIST embeddings. This suggests that a relation-decoder model's ability to retain consistency over regions of latent space beyond where MNIST embeddings are found leads to improved PRT. These findings provide compelling evidence in support of our claim that consistency across relations is important for PRT performance.

Secondly, we examine how gradient-conformity affects PRT performance. To achieve successful PRT, fitting the target encoder to a single pretrained relation should lead to embeddings that are structured correctly with respect to the other pretrained relations. For this to be possible there must be a degree of conformity between how each model computes its system of relations. As an extreme case, suppose we have a two-dimensional latent representation, with two relations that are each calculated using entirely different dimensions of latent space. By fitting an encoder to one of these relations, there is no guarantee that the latent dimension, that the other relation requires, receives the necessary guidance. DC shows excellent and stable GC values (near 1) across all conditions. This is by design as the use of masks per relation ensures that if masks match for any two relations, then their gradients will be either parallel or anti-parallel. Excluding HolE, all remaining models show a positive correlation between GC and 𝛽, and it appears that models with either higher GC values, or 𝛽 response, typically perform better at PRT. Together this provides tentative evidence to suggest that GC is important to model coherence, as measured by their PRT performance. It is possible that we do not see a monotonic benefit of GC against PRT, due to no further extrapolation or interpolation Con-A gains with 𝛽 > 8.

Conclusion

We provide a comprehensive analysis of relation-decoder characteristics when learning the system of relations that together define the semantics of a concept. We then compare these characteristics with a Partial Relation Transfer task setting, which determines whether, given logical constraints between relations, fitting embeddings to one relation-decoder leads to embeddings that satisfy all other relations in terms of their logical consistency and accuracy.

Our results demonstrate that model consistency, and possibly gradient-conformity, across different regions of input space together determine whether a set of relation-decoders have learned a consistent and coherent notion of a given concept, in this case ordinality. These measures make it possible to check whether a set of relation-decoders have indeed learned a transferable concept, or if they are limited to a single data domain and property.

A. BlockStacks dataset description

The BlockStacks dataset consists of 12,000 images (200×200 pixels but resized in code to 128×128) of individual block stacks, of varying height (between 1-10 blocks), block colors (uniformly sampled from options: { gray, blue, green, brown, purple, cyan, yellow} ) and position (uniformly sampled from 𝑥, 𝑦 range (-3,-3) to (3,3)), but with the requirement that each instance consists of a single red block at a random height (see Figure 4 for example images). These were rendered using the CLEVR rendering agent with the help of code from [31]. The dataset is divided into 9000:1500:1500 train, validation and test splits.

B. Explanation of the 𝛽-VAE

The VAE is derived by introducing an approximate posterior 𝑞 𝛼 (𝑍|𝑋), from which a lower bound (commonly referred to as the Evidence LOwer Bound (ELBO)) on the true marginal log 𝑝 𝜃 (𝑋) can be obtained by using Jensen's inequality [28]. The VAE maximises the log-probability by maximising this lower bound, given by:

ℒ 𝐸𝐿𝐵𝑂 𝛽-VAE = 𝔼 𝑞 𝛼 (𝑍|𝑋) [log 𝑝 𝜃 (𝑋|𝑍)] − 𝛽𝐷 𝐾 𝐿 (𝑞 𝛼 (𝑍|𝑋)‖𝑝 𝜃 (𝑍)),(8)

where 𝑞 𝛼 (𝑍|𝑋) is the approximate posterior, typically modelled as a neural network encoder with parameters 𝛼. Similarly 𝑝 𝜃 (𝑋|𝑍) is modelled as a decoder with parameters 𝜃 and is calculated as a Monte Carlo estimation. A reparameterization trick is used to enable differentiation through an otherwise undifferentiable sampling from 𝑞 𝛼 (𝑍|𝑋) (see [28]). In the 𝛽-VAE [1,15], an additional 𝛽 scalar hyperparameter was added as it was found to influence disentanglement through stronger distribution matching pressure with respect to the prior 𝑝 𝜃 (𝑍), where this prior is typically set to an isotropic zero-mean Gaussian 𝒩 (0, 𝟙)). When 𝛽 = 1 we obtain the standard VAE objective [28].

C. Model Descriptions

In this section we provide model details for each relation-decoder that we use and the VAE architecture that we employ for each data set.

C.1. Relation Decoder implementations

TransR:

𝜙 TransR 𝑟 (𝑧 𝑖 , 𝑧 𝑗 ) = ‖ℎ 𝑟 + 𝑟 − 𝑡 𝑟 ‖ 2

with, ℎ 𝑟 = 𝑀 𝑟 𝑧 𝑖 and 𝑡 𝑟 = 𝑀 𝑟 𝑧 𝑗 .

As we want to obtain a [0,1] output, we modify TransR through 𝜙 TransR + 𝑟 = 𝜎 (𝑐 − 𝜙 TransR

𝑟

), where 𝜎 is a sigmoid function and c is a scalar that ensures that at 𝜙 TransR + 𝑟 (𝑧 𝑖 , 𝑧 𝑗 ) = 0, then 𝜙 TransR + 𝑟 (𝑧 𝑖 , 𝑧 𝑗 ) ≈ 0. In all experiments we set 𝑐 = 10. NTN (modified version from [32,33]):

𝜙 𝑟 (𝑧 0 , … , 𝑧 𝑛 ) = 𝜎(𝑢 ⊤ 𝑟 [tanh(𝑧 𝑐⊤ 𝑀 𝑟 𝑧 𝑐 + 𝑉 𝑟 𝑧 𝑐 + 𝑏 𝑟 )])(9)

where 𝑢 𝑟 ∈ ℝ 𝑘 , 𝑀 𝑟 ∈ ℝ (𝑛−1)⋅𝑑 𝑧 ×(𝑛−1)⋅𝑑 𝑧 ×𝑘 , 𝑉 𝑟 ∈ ℝ 𝑘×(𝑛−1)⋅𝑑 𝑧 ) and 𝑏 𝑟 ∈ ℝ 𝑘 . The only hyperparameter to consider is 𝑘, which controls the NTN's capacity -in all experiments, we set this to 1. Here 𝑧 𝑐 is a concatenation of the inputs 𝑧 0 , … , 𝑧 𝑛 , which was introduced in [32,33]. In contrast the original NTN (see [3]) is only applicable to binary relations and does not include the outer sigmoid. HolE:

𝜙 HolE 𝑟 (𝑧 𝑖 , 𝑧 𝑗 ) = 𝜎 (𝑟 ⊤ (𝑧 𝑖 ⋆ 𝑧 𝑗 ))

where ⋆ ∶ ℝ 𝑑 × ℝ 𝑑 → ℝ 𝑑 denotes the circular correlation operator and is given by,

[𝑧 𝑖 ⋆ 𝑧 𝑗 ] 𝑘 = 𝑑−1 ∑ 𝑚=0 𝑧 𝑖,

C.2. VAE configuration

In all representation learning experiments, we use a 𝛽-VAE trained for 300,000 steps, following accepted practice from [20,22].

The encoder-decoder model parameters are given in Table 1 -we include the model configurations used for both MNIST and BlockStacks datasets.

C.3. ℒ 𝑗𝑜𝑖𝑛𝑡 configuration

In the source domain, we vary 𝛽 values between {1, 4, 8, 12} and fix 𝜆 = 10 3 . In the target domain, we fix 𝛽 to 10 −4 and 𝜆 = 10 −2 and normalise the ℒ 𝐸𝐿𝐵𝑂 𝛽-VAE reconstruction term by dividing by a factor 1 √𝐻 ⋅𝑊⋅𝐶 , for height 𝐻, width 𝑊 and color channels 𝐶, and normalize ℒ (𝜓 𝑡 𝑒𝑛𝑐 , 𝒩 (0, 𝟙)) by a factor 1 𝑑 𝑧 , for latent representation size 𝑑 𝑧 .

D. Supplementary Results

Figure 5 and Figure 6 provide additional results for Con-I (individual consistency scores for individual relation properties covering transitivity, asymmetry and reflexivity) and Con-A, configured on the same data splits as described in the main text. These results cover variants of where 𝜎 is the sigmoid function and 𝜂 is a scalar value. This modification enables a cliff-like shape for 𝜙 † 𝑟 , such that it can output close to 1 for a wider vector difference range. Note all distribution forms are unnormalized so that they cover the interval [0,1].

The NN variants vary layer depth and size, but all use a common input layer of size 𝑙 in = 2 * 𝑑 𝑧 . NN2 is a three-layer neural network with hidden layer size 𝑑 𝑧 and NN3 is a four-layer neural network which is the same as NN, but in contrast has a 𝑑 𝑧 pre-final layer size, thereby omitting which enables a pairwise comparison between each input dimension. NN1-sig is the same as NN but employs sigmoid activations, instead of ReLUs. NN-DC is the again same as the NN from the main text, but includes an additional 𝜙 † 𝑟 -type node that can compute relative differences between inputs in the same way as DC.

E. How does each model impact the retention of domain-dependent information

Figure 7 shows results for BlockStacks overall block height prediction accuracy when training on fixed encodings of each block stack, after isSuccessor relation-decoder alignment as been applied. Note 𝛽 is fixed in the target domain, so the only moving part are the pretrained models which are trained with varied source 𝛽 values. Note also that dc has an unfair advantage here, as the steered fitting approach allows more flexibility to the VAE learning phase -for this reason the result is only included in the appendix. Since we are interested in capturing general representations that encode both domain-dependent and -independent information, we use each target encoder 𝜓 𝑡 𝑒𝑛𝑐 obtained from each PRT experiment and produce encodings for the full BlockStacks test set. The resulting encodings are then divided into a new train and test subset, used to train both a Sci-Kit Learn Linear regressor and Support Vector Machine regressor with a RBF ∘ kernel [34]. We present the resulting Mean Squared Errors (MSE) in Figure 7, with Ordinary Least Squares (OLS) (a) and Support Vector Regression (SVR) (b).

There are a number of noteworthy details: firstly, DC shows no dependence on 𝛽 and leads to a lower MSE across all settings; second, excluding DC, for all models we observe an optimum MSE at 𝛽 = 8, with TransR reaching DC MSE performance for OLS and NN doing the same for SVR. These results indicate that lower MSE can be obtained by using non-linear regression, which indicates that to some degree, the block stack height factor is not encoded linearly, regardless of selected model. Next, by contrasting with Figure 3-bottom, these results suggest that models with higher GC lead to embeddings that are more amenable to domain-specific factor prediction. However, the parabolic trend, where increasing 𝛽 to 12 leads to an increase in error, is in agreement with Figure 2-bottom-right, which showed that most models do not improve at PRT for the largest 𝛽.