VSA-based Positional Encoding Can Replace Recurrent Networks in Emergent Symbol Binding

Abstract

Variable binding is an open problem in both neuroscience and machine learning relating to how neural circuits combine multiple features into a single entity. Emergent Symbols through Binding in External Memory is a recent development tackling variable binding with a compelling solution. An emergent symbolic binding network (ESBN) is able to infer abstract rules through indirection using a dual-stack setup-whereby one stack contains variables and the other contains the associated keys-by autonomously learning a relationship between the two. New keys are generated from previous ones by maintaining a strict time-ordering through the usage of recurrent networks, in particular LSTMs. It is then a natural question whether such an expensive requirement could be replaced by a more economical alternative. In this work, we explore the viability of replacing LSTMs with simpler multi-layer perceptrons (MLPs) by exploiting the properties of high-dimensional spaces through a bundling-based positional encoding. We show how a combination of vector symbolic architectures and appropriate activation functions can achieve and surpass the results reported in the ESBN work, highlighting the role that imbuing the latent space with an explicit structure can play for these unconventional symbolic models.

Keywords

• eol>Emergent symbolic binding network
• vector symbolic architectures
• symbolic reasoning
• siren
• sparse distributed memory
• variable binding
• recurrent neural network

1. Introduction

In the fields of neuroscience and philosophy, the binding problem [ 1 ] refers to the ability of the human brain to form a cohesive experience out of the myriad of inputs it receives from both the external environment as well as the continuous feedback signals which are generated internally. Visual sensation has been perhaps most studied in this regard [ 2, 3 ]. Particularly relevant nowadays is the ability of the human brain to process and decompose sentences (and more in general language [ 4 ]) into their constituent components [ 5 ].

Neural binding can be subdivided further with each sub-problem being of great interest in its own right. However, here we will focus exclusively on the variable-binding aspect. The inverse of the binding problem is the unbinding problem, or the best match problem, which deals with how to efectively separate the neural correlates into their foundational components. Both binding and unbinding play a critical role in the human brain’s ability to produce abstract concepts, elaborate them, and more in general to reason symbolically. For a neural network to exhibit the latter behaviours, it is therefore necessary for it to solve the former problems as well [ 6, 7 ].

Neuro-symbolic artificial intelligence (NSAI) [ 8, 9, 10, 11 ] with its many incarnations combines a symbolic and structured representation with the end-to-end learning capabilities of neural networks. This enables competitive, and even superior, performance with the current state of the art in visual abstract reasoning tasks [ 12 ]. By leveraging the capabilities of NSAI, emergent symbolic binding networks (ESBNs) [ 13 ] enable a form of variable-binding and indirection. Its central component is a hetero-associative memory, or external memory, which self-optimises the relationship between keys and values through pure back-propagation. This external memory bears a resemblance with Kanerva’s sparse distributed memory (SDM) [ 14 ] as a flexible model that can be both hetero-associative, where the key and value are diferent, and auto-associative, where the key and value are the same. The keys in SDM are carefully selected random vectors to maintain minimal destructive interference (thanks to the properties of high-dimensional spaces we detail in Section 2.1). Recently, SDM has been identified as a close analogue of Transformer attention [ 15 ].

It has been shown in many NSAI models that imbuing the latent space with implicit structure (e.g. creating a semantic hierarchy among concepts) significantly improves performance in reasoning tasks. In this work, we establish that this efect is also relevant in ESBNs. We exploit the properties of vector symbolic architectures (VSAs) [ 16, 17, 18, 19 ] in order to modify interactions within the key space, without altering the value space. This is made possible by VSA’s great flexibility and allows us to keep the original indirection mechanism intact and strengthen our conclusions.

We present a simpler alternative to cumbersome recurring networks in ESBNs. By combining a multi-layer perceptron (MLP) with appropriate activation functions and positional binding based on VSA, we can successfully replace an LSTM and still solve visual abstract reasoning tasks. Moreover, we characterise the regularising efect of LSTMs in ESBNs, and show that while they efectively constrains over-parametrisation, this also leads to under-expressiveness in some situations. Taken together, our findings show that MLPs are capable of the same time-ordering sensitivity as LSTMs and clarify that the resulting increase in expressiveness is beneficial.

2. Background

2.1. Vector symbolic architectures

To address the variable-binding problem we have opted here to use an approach based on VSAs [ 16, 17, 18, 19 ], which explicitly define functions to bind keys to values. This is in contrast to ESBNs [ 13 ] which learn how to do indirection on their own. VSAs are computational paradigms which exploit the properties of high dimensional spaces to represent and manipulate symbols. A detailed overview can be found in [ 20 ] and [ 21 ].

There are a variety of possible representations residing under the term VSA ([ 22 ]) and, while it is not necessary here to present all of them, it is nonetheless of interest to remark that not one solution can be found to fit all problems. It is indeed often the case that some tasks perform better with one or another. The element which binds all these representations together is their ability to imbue a high dimensional space with some structure. We focus precisely on this common aspect to better highlight the strengths of VSA.

In contrast with the well-known curse of dimensionality, high dimensional spaces also exhibit extremely beneficial properties, not the least of which is the concentration of measure [ 23 ]. This plays a role in the generation of the positional code-book, i.e. the set of vectors used to distinguish one time step from another. For example, it is possible to randomly draw an arbitrary number of vectors while still guaranteeing that all of the vectors remain quasi-orthogonal and easily distinguishable from one another. This allows for greater flexibility in the number of time steps that can be emulated, with the understanding that LSTMs and other recurrent neural networks are much more limited in how long they can be run continuously before encountering gradient issues.

In VSAs, two operations, bundling and binding, form the bedrock of any model. Binding, as the name suggests, performs key-value binding, while bundling creates sets of symbols. Within the space, bundling preserves the similarity of the inputs (i.e. the cosine similarity of the output with each of the inputs is high), while binding does not. We have given an abstract definition which outlines the necessary properties these two operators must possess; however, the specific instantiation is left to the implementer.

A definite realisation of binding and bundling, together with the accompanying space, identify a particular member of the VSA family (for more information see [ 22 ]). As binding is taken care of by the ESBN, we need only focus on bundling and we do so by choosing the simplest and most general form: component-wise addition. It must be noted that this kind of bundling is also performed (with a specific code-book) by Transformers [ 24 ] under the name of positional encoding. A detailed description of our VSA implementation is presented in Section 3.1.

Now that we have explored the bundling aspect of the architecture we can focus on binding, which follows the ESBN model. 2.2. ESBN The ESBN architecture has been introduced in [ 13 ] as a possible solution to the variable binding problem. ESBNs have been developed to solve reasoning tasks in the form of extrapolating relationships from sets of images. As we believe this is a relevant setting for assessing humanlike capabilities, we do the same. Therefore, the input to the network are black and white images, and the output a variable length binary vector encoding such relationship. For more information refer to Section 2.2.2. Here, we provide a brief overview of the fundamentals details for the function of the model, for a more in-depth treatment of the model please refer to the paper above.

2.2.1. Architecture

Symbolic binding networks are composed of two mostly independent components, which we explicitly term the value pathway and the key pathway (cfr. Figure 1, top panel) to make further analysis easier. We present now an overview of both pathways, exposing their machinery and describing their underlying components. RECURRENT [ 13 ]

POSITIONAL ENCODING + MLP (OURS) L L E C

L L E C

L L E C

MLP +

At each step, an image sample passes through the value pathway (cfr. Listing 1), where an encoder generates a feature vector and appends it to the value stack . At the same time, is compared to each element in to form a vector of weights . This mechanism closely resembles attention in that the weight of a given memory item is proportional to its similarity with . In the key pathway (cfr. Listing 2), the similarity vector is used to combine the keys in in a weighted sum (or superposition) to generate . The newly generated is then fed into an LSTM which outputs a prediction and a new key . Finally, is appended to , not .

Listing 1: One step in the value pathway.

Listing 2: One step in the key pathway. = () = ( · ) p(, ) , , = (− 1 ) = S( · ) = ∑︀()(, ) p( , ) Listings 1 and 2: is the softmax function. S is the sigmoid function. (a, b) refers to concatenation. p pushes an element onto the stack.

While the two pathways never interact explicitly, their computational graphs are connected through such that back-propagation is possible. Each image in the task, including both question and answer panels, is fed through the network. The final output ^, be it a binary or one-hot-encoded vector, is used to determine the answer in a task-dependent manner.

A key contributor to the performance of ESBNs in [ 13 ] is the introduction of a temporal context norm (TCN). This scheme acts as a regulariser by normalising over temporal windows similarly to how batch norm does over batches. It has been shown that TCN is fundamental for ESBNs in the original paper, and also improves performance for non-ESBN architecture in the tasks outlined above. To provide as representative a comparison as possible, we also include TCN in our architectures. However, we further show in Section 4.1.2 that its inclusion may not be as essential.

To highlight our intended comparison between LSTM and VSA-enhanced MLPs, we focus here only on the key pathway. We manipulate it as follows: we replace (cfr. Figure 1, bottom panel) the key encoder with a simpler feed-forward neural network instead of an LSTM, and we add a positional encoding component to its input. This allows us to emulate the time progression of an LSTM with a much more economical and well-behaved alternative. In Section 3, we analyse in more detail how these modifications are implemented and how the behaviour of the model changes. 2.2.2. Tasks The tasks (cfr. Figure 2) chosen in the original work serve to showcase the ability of the ESBN to infer abstract rules from visual cues in varying degrees of dificulty. Generally, visual reasoning datasets like CLEVR [ 25 ] and RAVEN [ 26 ] represent the litmus test for architectures — be they neurosymbolic or not — which purport to approach human-like cognition capabilities. The same tasks are, therefore, particularly suited for evaluating the performance of our architecture as well. (a) Same/diferent task (b) Relational match-to- (c) Distribution of three sample task (d) Identity rules

It has been shown [ 27 ] that conventional neural networks are unable to solve even extremely simple visual reasoning tasks like the same/diferent task, in which the model is asked to determine whether two shapes are equal. An ESBN is not only capable of successfully determining this straightforward relationship, but can also extend it to novel image pairs such as in the relational match-to-sample task. Finally, a subset of RAVEN progressive matrices [ 28 ] can be used to test cognition more at a human level with the distribution of three and identity rules tasks. These two final tasks put some strain on both the ESBN and our architecture, which further serves to highlight the diference in performance.

For each scenario a certain number of samples is held out for testing, with percentages varying from 0% to 98%. Naturally, training with fewer samples proves more dificult and also emphasises the sample-eficiency of the diferent architectures.

3. Approach

Given the minimalism of the value pathway, we focus our eforts on the key pathway. As explained in Section 2.2, keys are generated by feeding the attention weighted superposition of previous keys to the LSTM. This process intrinsically keeps tabs on the order in which it receives its inputs. We hypothesise this ordered structure to be crucial to the success of the model such that a straightforward replacement with a non-recurrent architecture would not be suitable. To preserve the properties outlined above, we introduce a new model which combines a more amenable representation of time as a discrete set of vectors (positional encoding through bundling) with an eficient feedforward network which makes use of state-of-the-art activation functions such as SIREN [ 29 ].

To better elucidate the key elements of our model we provide a ’in the footsteps’ view of a key (cfr. in Figure 1) that has been newly generated by superposition. First, is is bundled with the corresponding time-representing vector, in order to relocate it into our structured space. Then it is fed through an MLP which extracts the relevant features. Finally, it is passed to the activation function which favourably transforms these features to be added back onto the key stack.

3.1. Bundling

As discussed in Section 2.1, the bundling operation of our choice is component-wise addition. The goal of this step is to embed with information about its position in the sequence, in such a way that a non-recurrent network should accurately infer the corresponding time-step from the key alone. We hypothesise that an appropriate VSA model is able to structure the space so that this embedding is successful. On its own, bundling is not suficient to completely identify a VSA implementation. For this reason we also need to define the code-book ( in Figure 1). The code-book does not easily yield itself to a generalised representation, so we try three variants to avoid possible missteps.

The first and easiest setup amounts to choosing random vectors ( Rand strategy) in a high dimensional space, which a priori have almost no similarity to each other. This approach allows us to maximally separate the time-steps in latent space. The second retraces the positional encoding of the Transformer (TF strategy). For more information on the benefit of this choice refer to [ 24 ]. The third and final code-book is created using fractional power encoding ( FPE strategy) [ 16 ]. This method can generate any number of vectors with a controlled distance to each other. This specific property, when used to create equally-spaced codes, embeds a sequential structure in the space which more closely follows the concept of time in RNNs.

Once the key vector has been bundled (i.e. summed) with the code vector, it contains information both about the relationship with previous keys and its position in the sequence. It is then fully equipped to be correctly interpreted by a feedforward network and does necessitate any recurrent structure. The dimensionality of each code vector needs to be equal to the key, which in our case is 256. As control the same experiments are performed with no code-book at all, meaning the position in the sequence is unknown to the MLP and can at most be inferred by correlation with the other keys. 3.2. MLP As a drop-in replacement to LSTMs for feature extraction we chose an MLP, which performs well on unstructured data and is also eficient parameter-wise. Unstructured here means that, in contrast to the original image samples, keys do not necessarily possess any visual patterns which models such as CNN are better suited to pick up. The model must be, however, powerful enough to understand the bundled structure and decompose it in order to correctly process the key. This choice gives us maximum flexibility and parameter eficiency, while also afording us optimal performance during testing. The only parameters we vary in the MLP are its hidden and output activation functions, which we explore in the following section.

3.3. Activation function

Given the abundance of activation functions in the MLP space, we chose to test three variants. The two most common choices for many models are tanh and ReLU. ReLU is a biologically inspired activation function which squashes negative values while being unbounded on the positive semi-axis. By contrast, the hyperbolic tangent is a well-behaved sigmoid function which has the benefit of being bounded, and as such tends to lessen the efect of gradient issues. It was chosen as control given ReLU is used in the original ESBN.

Recently there has been renewed interested in sinusoidal activation functions, for example SIREN [ 29 ]. It has also been shown that these functions are particularly useful in reconstructing complex spaces by preserving some underlying mathematical structures such as first and second derivatives. This adaptability to time-dependent signals made SIREN an important comparator in our testing strategy.

As output of the MLP we now have a vector that is analogue to the output of the LSTM, and can therefore be used to obtain predictions by following the rest of the key pathway laid out in an ESBN. To evaluate the diferent architectural choices, and trade blows with the state-of-the art, we use the tasks outlined in Section 2.2.2.

4. Results

rmts dist3 identity_rules

Our architectures proved fully competitive with the original ESBN, with the sinusoidal activation function being clearly superior to the alternatives across the board. Initially we hypothesised this to be due to the unbounded nature of ReLU, but after observing the same behaviour on the bounded Tanh, we believed it to be the case that a periodic activation produces better latent neural representations than a monotone one. This phenomenon corroborates the ifndings of [ 29 ].

All positional encoding schemes behaved similarly, with Rand outperforming the alternatives on the maximum holdout. A more detailed analysis revealed that this observation does not hold in general, and in fact fractional power encoding has an edge when considering all holdouts (see Table 4). For this reason, we only test ReLU and Tanh activations with FPE encoding. Both SIREN and ReLU MLP architectures without positional encoding appeared to be unable to solve the identity_rules task, which validated our hypothesis that notion of position is key in replacing the LSTM

One must note that in the original ESBN, the attention vector was computed by means of the dot product. While in principle a valid choice, dot product is not robust as it can overshoot and lead to issues in both gradient computations and consistency between diferent samples. Normalisation is a common choice in these cases, and here we specifically chose to use the cosine similarity, which is bounded between − 1 and 1. To ensure that such a modification does not significantly hinder performance, we reproduced the results of [ 13 ] both with dot product and cosine similarity. We found that cosine similarity significantly improves performance on the rmts task bringing it up to 100%. All other models were tested using cosine similarity. The learning rate is kept constant at 5− 5.

Interestingly SIREN outperformed every other variant, with FPE and Transformer encoding

4.1. Ablations

4.1.1. Model size

performing especially well and often surpassing the original model. By keeping the same hyperparameters across the original ESBN and our models, we are able to report a ∼ 2.4× reduction in size1. This also improved results across the board by a straightforward implementation of VSA bundling.

We investigate the limits of model size on performance and the relevance of temporal context normalisation for our architectures.

Given we achieved a ∼ 2.4× size reduction without loss of performance while keeping the same hyperparameters, we next explored how much we could shrink both the original ESBN and our MLP model while keeping approximately the same performance. Moreover, we investigated whether instead increasing the size of the MLP to match the LSTM could yield further performance improvements at the expense of training speed. This appears unlikely as accuracy is already saturated in the current state.

Table 2 indicates that the two models do not scale the same. Our model can be reduced to a smaller size maintaining almost perfect accuracy, while the LSTM starts underperforming significantly, especially on the same_dif task. Increasing the parameter space did not yield increased performance, as predicted. rmts dist3 identity_rules LSTM MLP 300k 300k 1.9M 79.9 (33.2) 92.3 (1.2)

96.3 (7.4) 100.0 (0.0) 100.0 (0.0) 99.2 (0.6) 100.0 (0.0) 100.0 (0.0) 98.9 (1.4) 95.3 (2.5) 99.0 (1.0) 99.1 (0.4)

4.1.2. Temporal context norm

In the original work of [ 13 ], the temporal context norm plays a crucial role in the performance of ESBNs and the other architectures tested. In particular on the maximum holdouts per task it increases accuracy from near-chance to near-perfect level.

Given its importance, we tested our MLP model including its usage for a fair comparison. In addition, we also investigated whether TCN can be done away with in our architecture. To evaluate the regularising efect of TCN, we tried to both increase and decrease model size as in the previous ablation (Table 3). 1The original ESBN has ∼ 1.9 M parameters, while our SIREN model with fractional power encoding has ∼ 800 k.

rmts

A marked drop in performance was observed, especially in the same_dif task. Despite this, however, the accuracy stayed well above chance level and surpassed the original LSTM-based ESBN in both the rmts and dist3 tasks. Increasing model size did not yield increased accuracy. On the contrary, decreasing model size regularises in its own right, and hence increases performance. These results show that while TCN is an efective regulariser, it is not a necessary enabler of ESBN performance. For further analysis, refer to Section B.2.

5. Discussion

In this work, we have shown that enhancing simple MLP with VSA yields an efective and eficient alternative to LSTM when applied to ESBNs by simplifying the architecture and enhancing performance. Furthermore, we have demonstrated that a structured representation of time-ordering is entirely capable of replacing the complex machinery of an LSTM without loss of generalisation capability. We also observed a clear saturation of results, whereby the more eficient models have displayed near-perfect results on all tasks and have made further distinguishing and improvement unfeasible. It would be, therefore, of interest to test ESBNs and our SIREN alternative on more challenging datasets such as the full RAVEN, and possibly expand to other non strictly reasoning tasks.

We have identified in the LSTM a key regularising efect which reduces the degrees of freedom of the vanilla ESBN, thus mitigating its marked over-parametrisation. At the same time we have demonstrated that it is feasible to straightforwardly reduce the model size through VSA-enhanced MLP, without losing performance. Following the same principle, we have also identified in the temporal context norm another regulariser, which proves fundamental for an LSTM but not so for MLPs. This finding leads us to hypothesise that any suficiently powerful regularising technique can be employed in place of the TCN, but further investigation would be needed for confirmation.

The recently found similarities between attentional models and symbolic memories such as SDM might shed more light towards the replacement schema we have presented here, and future work should focus on more complex tasks and architecture. Specifically, it would be interesting to explore to what extent embedding VSA into other symbolic framework might enhance performance and yield results similar or better to other more conventional architectures.

Acknowledgments

This work is supported by the Swiss National Science foundation (SNF), grant no. 200800.

6. Appendices

A. Supplementary results

On the identity_rules task (cfr. Tables 11,12) all variants perform relatively similarly, with SIREN and tanh matching the vanilla model in all instances. ReLU is competitive, but on the hardest 95% holdout performance drops by about 9% w.r.t. to SIREN. All variants without positional encoding do not solve the task successfully, validating our hypothesis that a structured space significantly improves results and is key in replacing the LSTM.

On dist3 (cfr. Tables 9,10) SIREN surpasses LSTM, with ReLU and tanh trailing behind by about 4 percentage points in the maximum holdout. Interestingly here a lack of positional encoding does not seem to harm performance.

Tables 7,8 and 5,6 show that all tested architectures find the same_dif and RMTS tasks particularly easy, and all perform very well (except LSTM with cosine similarity as mentioned above).

LSTM MLP FrancescoS.Carzanigaetal.CEURWorkshopProceedings 1–23 o ss d i ) ) ) ) ) ) ) ) ) dn rea h y 0 0 0 0 0 0 0 0 0 a t n 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 t sn i a 5 (0 (0 (0 (0 (0 (0 (0 (0 (0 iraavn r10u []13w ,sah =m .000 .000 .000 .000 .000 .000 .000 .000 .000

y 0 0 0 0 0 0 0 0 0 e t 1 1 1 1 1 1 1 1 1 h s m ir c a o a a c r l ) ) ) ) ) ) ) ) ) e h f i 0 0 0 0 0 0 0 0 0 r c N im .0 .0 .0 .0 .0 .0 .0 .0 .0 Fo ea B s 0 (0 (0 (0 (0 (0 (0 (0 (0 (0

S e = 1–23 m .15 .68 .07 .31 .07 .38 .41 .53 .13

0 0 0 0 0 0 0 0 0 r e .Fo sca g n i k h d s c o a a c t e n e e d e

e fi In l † onN FPE FT on an PE on PE

a N R F N F n n o e

r n a n o p o is it e N N N N U U h ra in a n E E E E L L n p n itv † oN ISR ISR ISR ISR eR eR aT m io c o t A

a c i

P L

M e v c e n d . M am rad ity S

T r L r a o d l f n i r a m e t i P s s : 6 FrancescoS.Carzanigaetal.CEURWorkshopProceedings 1–23 r , e d

e e tegh trop reag .998 .000 .000 .000 .999 .000 .000 .000 .000 to re v 9 01 01 01 9 01 01 01 01

A

M e n d e t a d lu i rm ra av rch TM fo d a S r an tse o L e t P s b N : 7 LSTM MLP e v c e s h n d e c a d lu ra M m ra a T r v N S o d fr an tse IER L e t P s b S : 9 LSTM MLP FrancescoS.Carzanigaetal. CEURWorkshopProceedings e an tse th tc ch A c

M T S L

Activation

B. Ablation results

B.1. Model size

We report here the full results for the model size ablation. For all tasks (cfr. Tables 14,15,16) except same_dif accuracy remains high until the highest holdout. In the same_dif task (cfr. Table 14) we observe a proportional worsening of performance as holdout increases.

B.2. Temporal context norm

We report here the full results for the temporal context norm ablation.

There is a clear trade-of between expressiveness and over-fitting, whereby TCN severely and eficiently constricts parametrisation to improve performance across the board. On the same_dif task, Table 17 shows that model expressiveness greatly benefits performance, with our MLP model surpassing the LSTM-based ESBN across all holdouts except the last, only significantly underperforming when the amount of data is very limited.

On the other hand, for the identity_rules task (cfr. Table 20) we observe that overparametrisation of the MLP model is punished by a reduction in accuracy. This is especially striking for the 1.9M parameters MLP. Here performance is not strongly correlated with holdout, even increasing with increasing holdout, indicating that a smaller model generalises better. This distinctive efect is confirmed by the fact that indeed the smallest MLP performs the best out of our models.

On the dist3 and identity_rules tasks the smallest model performs the best, while it is the other way around for same_dif and rmts. This is once more an indicator that there is a strong need to strike a balance between expressiveness and the natural tendency to over-fit, role which TCN serves remarkably well.

Taken together, LSTM and TCN provide strong regularisation performance which balances even an highly overparametrised model like the original ESBN. This fact is highly beneficial, as seen in the near-perfect performance seen in [ 13 ], but is not necessary and reveals itself to be a hindrance in the more constrained cases shown in the ablations.

In fact our MLP model retains all of the expressive power and removes these limitations. It is then reasonable that any powerful enough regularisation could take TCN’s place in our model, but further investigation in this direction is needed.

Model LSTM MLP

Model 300k 800k 100.00 (0.00)

References

[1] J. Feldman , The neural binding problem(s) , Cognitive Neurodynamics 7 ( 2012 ) 1 - 11 . doi:10/gf7sk7.

[2] E. H. Land , J. J. McCann , Lightness and retinex theory , Journal of the Optical Society of America 61 ( 1971 ) 1 . doi:10/dq75zw.

[3] E. Adelson , A. Pentland, The perception of shading and reflectance , in: Perception as Bayesian Inference, Cambridge University Press, 1996 , pp. 409 - 424 . doi:10/gmx7vx.

[4] N. Chomsky , On certain formal properties of grammars , Information and Control 2 ( 1959 ) 137 - 167 . doi:10/fw3f8r.

[5] R. Jackendof , Foundations of Language , Oxford University PressOxford, 2002 . doi: 10/ fc4r6s.

[6] P. Smolensky , Tensor product variable binding and the representation of symbolic structures in connectionist systems , Artificial Intelligence 46 ( 1990 ) 159 - 216 . doi: 10/bvnjss.

[7] K. Gref , S. van Steenkiste , J. Schmidhuber , On the binding problem in artificial neural networks , CoRR abs/ 2012 .05208 ( 2020 ). arXiv: 2012 .05208.

[8] K. Yi , J. Wu , C. Gan , A. Torralba , P. Kohli , J. Tenenbaum , Neural-symbolic vqa: Disentangling reasoning from vision and language understanding , in: S. Bengio, H. Wallach , H. Larochelle , K. Grauman , N. Cesa-Bianchi , R. Garnett (Eds.), Advances in Neural Information Processing Systems , volume 31 , Curran Associates , Inc., 2018 .

[9] J. Mao , C. Gan , P. Kohli , J. B. Tenenbaum , J. Wu, The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision , in: International Conference on Learning Representations , 2019 .

[10] G. Karunaratne , M. Schmuck , M. L. Gallo , G. Cherubini , L. Benini , A. Sebastian , A. Rahimi , Robust high-dimensional memory-augmented neural networks , Nature Communications 12 ( 2021 ). doi:10/gk3vvk.

[11] M. K. Sarker , L. Zhou , A. Eberhart , P. Hitzler , Neuro-symbolic artificial intelligence , AI Communications 34 ( 2022 ) 197 - 209 . doi:10/grxtkd.

[12] M. Hersche , M. Zeqiri , L. Benini , A. Sebastian , A. Rahimi , A neuro-vector-symbolic architecture for solving raven's progressive matrices , Nature Machine Intelligence 5 ( 2023 ) 363 - 375 . doi: 10 .1038/s42256-023-00630-8.

[13] T. W. Webb , I. Sinha , J. Cohen , Emergent symbols through binding in external memory , in: International Conference on Learning Representations , 2021 .

[14] P. Kanerva , Sparse distributed memory , MIT press, 1988 .

[15] T. Bricken , C. Pehlevan , Attention approximates sparse distributed memory , in: M. Ranzato , A. Beygelzimer , Y. Dauphin , P. Liang , J. W. Vaughan (Eds.), Advances in Neural Information Processing Systems , volume 34 , Curran Associates , Inc., 2021 , pp. 15301 - 15315 .

[16] T. A. Plate, Holographic recurrent networks , in: S. Hanson, J. Cowan, C. Giles (Eds.), Advances in Neural Information Processing Systems , volume 5 , Morgan-Kaufmann, 1992 .

[17] T. A. Plate , Holographic Reduced Representation: Distributed Representation for Cognitive Structures, Center for the Study of Language and Information, 2003 .

[18] R. W. Gayler , Vector symbolic architectures answer jackendof's challenges for cognitive neuroscience ( 2004 ). doi:10/grxtkg.

[19] P. Kanerva , Hyperdimensional computing: An introduction to computing in distributed representation with high-dimensional random vectors , Cognitive Computation 1 ( 2009 ) 139 - 159 . doi:10/dfm9nw.

[20] D. Kleyko , D. A. Rachkovskij , E. Osipov , A. Rahimi , A survey on hyperdimensional computing aka vector symbolic architectures, part i: Models and data transformations , ACM Computing Surveys 55 ( 2022 ) 1 - 40 . doi:10/grktgq.

[21] D. Kleyko , D. Rachkovskij , E. Osipov , A. Rahimi , A survey on hyperdimensional computing aka vector symbolic architectures, part II: Applications, cognitive models, and challenges , ACM Computing Surveys 55 ( 2023 ) 1 - 52 . doi:10/grtc8c.

[22] M. A. Kelly , D. Blostein , D. J. K. Mewhort , Encoding structure in holographic reduced representations ., Canadian Journal of Experimental Psychology / Revue canadienne de psychologie expérimentale 67 ( 2013 ) 79 - 93 . doi:10/f42jpw.

[23] M. Talagrand , A new look at independence, The Annals of Probability 24 ( 1996 ). doi:10/ c22dpm.

[24] A. Vaswani , N. Shazeer , N. Parmar , J. Uszkoreit , L. Jones , A. N. Gomez , L. Kaiser , I. Polosukhin , Attention is all you need, 2017 . doi:10/gpnmtv.

[25] J. Johnson , B. Hariharan , L. van der Maaten, L. Fei-Fei , C. L. Zitnick , R. Girshick , Clevr: A diagnostic dataset for compositional language and elementary visual reasoning , in: CVPR, 2017 .

[26] C. Zhang , F. Gao , B. Jia , Y. Zhu , S.-C. Zhu, Raven: A dataset for relational and analogical visual reasoning , in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2019 .

[27] J. Kim , M. Ricci , T. Serre, Not-so-CLEVR: learning same-diferent relations strains feedforward neural networks , Interface Focus 8 ( 2018 ) 20180011 . doi:10/gf6mw7.

[28] J. C. Raven , Raven standard progressive matrices, 1936 . doi:10/grxtkf.

[29] V. Sitzmann , J. N. P. Martel , A. W. Bergman , D. B. Lindell , G. Wetzstein, Implicit neural representations with periodic activation functions , 2020 . doi:10/grxtkc.

) d p l h m (% o c e ) ) ) ) ) ) ) ) ) h i d 3 3 1 4 1 3 6 3 4 cy ll h y 5 .6 .1 .6 .5 .4 .9 .1 .7 .9 rccau rssao , ignw llrcea 9=m .(0294 .(0598 .(0188 .(0293 .(0862 .(0491 .(2308 .(0931 .(0985 9 9 9 9 6 9 9 6 9