Learning abstract representations from perceptual stimuli is a natural task for humans, but a real challenge for AI systems. In the vast majority of cases, in fact, systems that have to deal with symbol manipulation, like in reasoning or planning, do not need to also learn the symbols they operate on, but these are typically assumed to be given by a supervisor. Moreover, symbolic manipulation often implies compositional properties which are dificult to learn. In this paper, we consider the problem of learning symbolic representations that can be associated to abstract concepts, to be used in a variety of downstream tasks, and we analyze the many challenges related to this important problem, with a particular emphasis on paving the way towards composable symbolic representations learned by neural networks. We identify key properties for symbolic composable representations, such as non-ambiguity and purity, that suggest the need for diferent types of regularizations within the learning process, as well as new metrics for their evaluation.
or deep hashing methods [5]. Yet, we claim that those approaches do not consider crucial
properties that symbolic representations should have, such as non-ambiguity, purity, and
especially composability [
The paper is structured as follows. After discussing related works in Section 2, in Section 3 we describe the properties that symbolic representations should have, and we discuss the challenges related to the learning problem, also focusing on the metrics that should be used to evaluate the goodness of the learned representations. In Section 4 we present a preliminary experiment to show the intrinsic dificulties of the considered task. Finally, Section 5 concludes the paper.
Neural discrete representations have been successfully used for a plethora of tasks. We identified in the literature a few categories of approaches: embedding tables, probabilistic latent spaces encoding discrete probability distributions, deep hashing approaches.
Embedding tables encode a finite (and thus discrete) set of objects with a continuous representation. Vector-quantized variational autoencoders [4] exploit a quantized embedding table, efectively making the representations discrete and achieving outstanding results in image and sound generation tasks. They found that discrete representations are crucial to avoid posterior collapse in variational autoencoders. The categorical reparametrization trick (Gumbel-softmax operator) [8] and its variant, the concrete distribution (Gumbel-sigmoid operator) [9], are continuous relaxations of binary and categorical random variables. Discretization is achieved by annealing a temperature parameter to zero, and it is thus susceptible to a trade-of between the “discreteness” of the representation and the “informativeness” of its gradient.
A notable mention of neuro-symbolic methods expoiting discrete probabilistic latent spaces
is Latplan [
Deep hashing methods [5] deal with eficient information retrieval, exploiting binary latent representations which are then used to address buckets by means of Hamming distance computations. Research directions in this area mainly focus on non-diferentiability issues of discrete representations and the Hamming function, as well as eficiency, in terms of number of training samples, availability of annotations and retrieval performance.
Among neuro-symbolic appraoches, methods feeding combinatorial optimizers with neural information must deal with discrete representations and how to backpropagate through illdefined gradients. A plethora of solutions [ 10, 6, 11, 12] have been proposed to define a gradient which is both well-behaved and useful as a learning signal which allows to exploit information from the downstream task of combinatorial optimization. These techniques however are often method-specific and are dificult to generalize to other tasks (e.g., planning or deduction).
There is extensive literature in philosophy of science concerned with the formal definition of desirable properties for symbols and symbolic reasoning. In the work by Taddeo and Floridi [13], for example, a “truly” symbolic system has to perform grounding while satisfying the zero semantic commitment condition (i.e., the agent must be able to develop its own semantics without external guidance). In the work by Santoro et al. [14], properties are instead defined within a meaning-by-convention framework: symbolic behavior is defined as the set of traits allowing an interpreter to operate within a socio-cultural setting. They identify six desirable properties, possessed by human behaviour but not yet by autonomous agents, and postulate that symbolic fluency can arise as emergent behaviour under meaning-by-convention training regimes, exploiting social interactions with humans, mediated by natural language, and large training sets. According to them, symbolic systems should be: receptive, constructive, embedded, malleable, meaningful and graded. While these properties are certainly intuitive, they are dificult to quantify and, therefore, they cannot be exploited to guide the learning process.
In this work, we took inspiration also from classic principles of knowledge organization [15] and knowledge representation [16, 17].
Symbolic manipulation is almost unanimously assumed to play an important role in reasoning generalization. The binding problem for subsymbolic systems is an open problem concerning the generation and manipulation of symbolic entities (objects) starting from subsymbolic stimuli [18]. Many neuro-symbolic approaches address this problem, by combining symbolic and subsymbolic components, so that manipulation subtasks (notoriously dificult to address by neural networks) are dealt by the former whereas learning and noise-sensitive subtasks (dificult to address symbolically) are solved by the latter. Our idea is to reduce this dichotomy, conferring a reasoning capability to neural networks, so that synergy with symbolic methods can be further strengthened, allowing for interleaved end-to-end neuro-symbolic computation.
Toward this goal, we propose the use of symbolic representations, with the final objective of acting both as an interface between neural and symbolic components, and as computational entities within the neural component. These representations should possess the following desirable properties: (i) non-ambiguity, (ii) purity, (iii) usefulness, (iv) symbol composability, (v) manipulation composability, and (vi) generalizability with respect to diferent tasks.
Non-ambiguity means that diferent entities should not be mapped to the same representation, while purity is its dual: every entity should have the smallest possible number of representations (ideally only one each). A high degree of non-ambiguity increases downstream performance and a high degree of purity implies a stronger robustness to noise. If both properties are optimal, there is a 1-to-1 mapping between entities and their representations.
Usefulness is the capability of improving performance for a downstream symbolic task,
with respect to another representation. According to Newell and Simon [
Symbol composability is the possibility of creating complex constructs (e.g., sequences, expressions, graphs, etc.) from atomic symbols and it is commonly accepted as the key component of any form of reasoning. As both perceptual information and underlying semantics are seldom “flat”, it is often necessary to encode hierarchies of entities. Grounding complex structures can be addressed by a wide spectrum of strategies, ranging from grounding atoms into a fixed structure, to learning part-whole hierarchies from perceptual stimuli [ 18]. Since diferent downstream tasks require diferent abstractions, we advocate for a semi-quantitative evaluation of symbol composability, to determine whether a given technique is beneficial, while acknowledging the dificulty in devising a quantitative metric.
Manipulation composability is the capability of deriving a new symbol by performing a sequence of operations to an existing symbol. There are many reasons justifying the need of manipulating entitites in a learnable way. To give some examples: the initial encoding may be at the “wrong” abstraction level (thus requiring a specialization/generalization operation), there may be the need to enforce a specific property (e.g., when encoding the set of natural numbers, it can be beneficial to enforce that each number is the successor of some other number), incremental construction (like contextual embeddings are computed in a transformer), denoising a representation, and so on. Although distinct concepts, manipulation composability is strongly related with the principle of elaboration tolerance [19] in knowledge representation, as it provides a mechanism to alter (or, conversely, guarantee invariance) a representation in response to external conditioning. As manipulation composability paves the way for a recursive processing, it also provides enough expressive power to allow a certain degree of reasoning capability, thus, just like symbol composability, it is important to quantify such capability.
Other properties are more application-specific, and thus we leave their analysis to future research, among these, generalizability (the capability of reusing the same representations for diferent downstream tasks) is of particular importance, while others are compactness, monotonicity, invertibility (or de-composability), and so on.
Existing representations present some, but not all of these properties. One-hot encoding is the most pure and general representation, however it is also arbitrary (unless it is relaxed into a continuous softmax, it cannot propagate gradients) and non-compact. Manipulation composability is extremely limited, as it can only be performed in terms of permutations, and symbol composability is non-trivial. Continuous embeddings are useful and general, but they lack in purity and may be ambiguous. Manipulation composability is extremely good, if implemented by transformers, and very limited symbol composability can be performed by algebraic operations in embedding space. To the best of our knowledge, symbolic properties of both probabilistic discrete latent representations and deep hashing methods have not been studied yet, and their application outside their traditional domains (variational autoencoders and retrieval systems) is relatively unexplored. We conjecture they both can achieve good generalizability and usefulness, provided they are learned with additional regularizations, but lack either kind of composability. Finally, discrete tensors acting as interfaces between neural and symbolic components, achieve the maximum degree of composability (delegated entirely to the symbolic part), along with the potential for high levels of purity and non-ambiguity (handled by the neural one). However, they have limited generalizability and, unless it is possible to backpropagate through the symbolic component, they may have reduced (or absent) usefulness.
In appendix A, we provide examples on how these properties may be quantified in practice.
Every discretization operation, from the simplest (), to the most complicated, is typically implemented by step functions, that are not well suited for gradient-based optimization methods, since their derivative is zero everywhere except in discontinuity points. This characteristic efectively stops information flow , because the chain rule would cause vanishing gradients. Existing methods mitigating this issue are not general and present specific issues. Residual connections between continuous and discrete representations are a naive and somewhat inefective approach: since gradient only flows along the continuous path, the learning process is encouraged to discard information coming from the discrete one. A temperature parameter scaling inputs for a sigmoid or softmax non-linearity can be annealed to zero during training, providing a progressively smaller gradient. Although simple and potentially efective, this method is sensitive to initial values and annealing schedule [20]. Straight-through estimation replaces a non-diferentiable non-linear activation with the identity function in the backward pass, hence “skipping” the activation during chain rule computation. This approach works well in those cases in which the non-linearity closely approximates the identity, but may provide misleading gradient updates when this assumption does not hold. Methods such as the ones exploited by diferentiable combinatorial optimizers [ 11] are extremely powerful techniques for backpropagation, based on building gradients as finite diferences between the input representation and the minimal perturbation acting as a counterfactual. Although efective and representation-centric, they require binary decision boundaries and are slow to compute.
An additional issue arising in gradient-based methods is the choice of the loss function. Symbolic computations are usually discrete in nature, and therefore it may be challenging to devise both a quantitative evaluation of the representation, and an optimizable continuous relaxation or upper bound of such evaluation. Assignment-based manipulations (e.g., deduction, Hungarian matching, etc.) can be reformulated within a fuzzy or probabilistic framework, allowing for a diferentiable objective, with suitable thresholding operations performed at inference time, which can however present boundary instability issues. Retrieval-based approaches heavily rely on non-diferentiable functions comparing diferent representations (such as the Hamming distance) and often require to devise a continuous upper bound [21].
High-quality training data is often expensive to annotate and the ideal condition in which supervision is provided for every concept associated to each data point is often far from reality. Symbolic representation learning should exploit intrinsic characteristics which reduce the quantity and quality of annotations required, ideally up to the completely unsupervised discovery of symbols and their compositions. In general, one could provide (i) complete supervision on concepts, (ii) supervision on the downstream task only, (iii) no supervision at all, (iv) weak or partial supervision in the form of relations between entities (e.g., monotonicity constraints, ontologies, equivalence classes between entities, etc.).
Knowledge plays an important role both in learning and reasoning. Being able to learn a
representation by means of knowledge injection is not only a way of reducing the amount of
data required to achieve a “good” representation, but also greatly improves usefulness. This
challenge is strongly related to the way in which supervisions are provided, as well as to the
problem of learning through gradient-descent. Knowledge can be injected in the form of (i)
representational constraints (e.g., distances between representations should mirror paths inside
a knowledge graph), (ii) constraints related to the downstream task structure (as it happens
with planning [
Regarding the metrics to be used to evaluate symbolic representations, we do not believe in the existence of a one-size-fits-all metric, in virtue of diferent downstream applications for which diferent subsets of desirable properties are needed. To define a suitable metric, one should consider: (i) the target property, (ii) the type of supervision required, (iii) the interaction with learning. By target property we mean the property quantified by the metric: we provide some examples in Appendix A. An important open question along this dimension is how to quantify composability. The second dimension addresses the availability of ground truth with respect to the four categories of quality of annotations mentioned above. For example, in a metric learning setup only the equivalence relation between inputs is known (category iv), and purity and non-ambiguity could be measured by re-identification metrics such as mAP@k, while in planning from visual stimuli where only feedback on the downstream task is available (category ii), usefulness of representations could be measured by the number of failed plans, or the average length of successful plans. Diferentiable metrics may be also exploited during learning. Along this dimension we can distinguish (i) truly diferentiable metrics, (ii) non-diferentiable metrics, (iii) metrics with diferentiable proxies (e.g., Hamming distance approximated by binary crossentropy), (iv) compliance to logic or combinatorial constraints. In the latter case it may be possible to define, for example, complex purity measures taking into account hierarchies of entities, or logic formulae against which compliance can be evaluated with techniques such as semantic-based regularization [22].
Additionally, specific challenges arise in specific scenarios. For example, in a multi-agent setting, agents may share information in the form of visual stimuli or directly as symbols. In the first case, an additional challenge is to ensure coherence across diferent representations; in the second case, the learning process should pursue consistency, so that the same symbol has the same meaning for all agents, and should also handle the case in which an agent may receive new symbols it never encountered before. In the case of continual learning, additional challenges are posed by handling catastrophic forgetting (not only in the sense of poor performance in forgotten tasks, but also in the sense of forgetting or overwriting the “meaning” of rarely observed representations), guaranteeing elaboration tolerance when new information is discovered, and generally requiring a stronger robustness to noise. As these require case-specific considerations, we leave their discussion to future work.
We now present a preliminary experiment, devised to back up some of our claims, and to pave the way for further investigations. Our aim is to analyze (i) whether symbolic representations can be achieved both by means of supervised classification and metric learning, and (ii) whether learning a symbolic representation via a downstream subsymbolic task is beneficial.
We considered the traditional MNIST digit classification [
Our training data consist of triplets randomly sampled from the MNIST dataset (we extract batches of 64 samples for 20,000 training steps): two images belong to the same class, and the third to a diferent one. In the literature, these samples are usually called anchor ( ), positive () and negative (). We store as training labels the value of the anchor for the single digit classification, and the sum of anchor and negative for the sum classification. 1
As an additional experiment, we consider whether the discrete representations, learned
when the downstream task is the one-digit sum, can be exploited by a decision tree to perform
the same classification tasks. For this scenario, we collect 200 training batches of 64 samples
each (thus 12,800 training samples), we compute the discrete representations, and implement a
standard decision tree using the default parameters of the scikit-learn library [
After training, we measure non-ambiguity and purity of learned representations, exploiting the entropy-based method presented in appendix B for supervised evaluation. Being intrinsic properties of the representation, we measure them once, regardless of the downstream task.
We leave to future work the design and implementation of metrics to validate the other properties presented in Section 3. 1Single digit classification could be improved by also annotating positive and negative samples: however, to balance the number of supervisions available for both tasks, we decided not to.
The discrete representation bottleneck is a layer with neurons which is discretized by means of the sign function, and which provides intermediate values for loss computation as follows: ℓ = · + = sign(ℎ) ℎ′ = ℎ = tanh(ℓ) ℎ 1
+
2 2
where ℓ is a linear combination of the inputs, ℎ applies to ℓ the non-linear activation function
tanh, is the discretized (binary) encoding of ℎ, and ℎ′ rescales the values of ℎ in [
ℒ = · ℒ + ℎ · ℒ + · (ℒ + ℒ) + ‖ℓ‖22 being and ℎ the weights of the representation losses, that of the classification losses ℒ and ℒ (standard cross-entropy), and that of L2-regularization applied to ℓ.
As the sign function has an ill-behaved gradient, the gradient flow across the discrete representation layer is efectively stopped. 2 To demonstrate the importance of receiving information from a downstream task, we implemented a simple strategy via the stop gradient operator: basically, we propagate in the forward pass, and the gradient of ℎ during the backward pass.
Purity and non-ambiguity are computed with a representation matrix associating discrete representations () and true labels.
2When gradient is not back-propagated, then the learned representations are inherently arbitrary.
digit classification with decision trees; (d) sum classification with decision trees.
signal (nograd, = 0, ℎ = 1). The Hamming triplet loss alone is not able to converge to
useful discrete representations (blue dashed and dotted curves in Figures 1c and 1d): this is a
counter intuitive result, however we posit that the multiple steps involved in the computation
(bits approximated by tanh, Hamming distance approximated by binary crossentropy on tanh
rescaled in [
8 12 16 20 No. bits in discrete representation 24 objective alone (dashed blue line) seems to have a regularizing efect on purity, whereby the other cases quickly diverge (the red dashed line represents a random baseline, as no learning objective was enforced on the discrete representation). As for non-ambiguity, it improves in all cases as the number of bits increases. While ultimately converging to zero, each curve has a diferent decreasing rate: as expected, when gradient flows, convergence is faster (grad, solid lines) and the random baseline (dashed red line) is the slowest.
We considered the task of learning symbolic representation, a problem that has been underexplored in the neuro-symbolic AI community. We defined the key properties of discrete symbolic representations and identified challenges associated to the learning problem. As the downstream use of symbolic representations can be applied to an ample and varied set of symbolic tasks, we identified a strong need also for appropriate evaluation metrics.
Multiple interesting research directions can be identified for the future. In particular, we aim to focus on two promising topics: the use of Peano arithmetics to represent the properties of symbolic representations by modeling composability through recursion, and the formalization of the learning problem within a continual learning setting, which also seems a natural way to enable the composability of diferent representations. symbolic boundary, in: Proceedings of the aaai conference on artificial intelligence, volume 32, 2018. [4] A. Van Den Oord, O. Vinyals, et al., Neural discrete representation learning, Advances in neural information processing systems 30 (2017). [5] X. Luo, H. Wang, D. Wu, C. Chen, M. Deng, J. Huang, X.-S. Hua, A survey on deep hashing methods, ACM Transactions on Knowledge Discovery from Data (TKDD) (2020). [6] M. V. Pogančić, A. Paulus, V. Musil, G. Martius, M. Rolinek, Diferentiation of blackbox combinatorial solvers, in: International Conference on Learning Representations, 2020. [7] W. Stammer, M. Memmel, P. Schramowski, K. Kersting, Interactive disentanglement: Learning concepts by interacting with their prototype representations, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 10317–10328. [8] E. Jang, S. Gu, B. Poole, Categorical reparameterization with gumbel-softmax, arXiv preprint arXiv:1611.01144 (2016). [9] C. J. Maddison, A. Mnih, Y. W. Teh, The concrete distribution: A continuous relaxation of discrete random variables, arXiv preprint arXiv:1611.00712 (2016). [10] A. Agrawal, B. Amos, S. Barratt, S. Boyd, S. Diamond, J. Z. Kolter, Diferentiable convex optimization layers, Advances in neural information processing systems 32 (2019). [11] M. Fredrikson, K. Lu, S. Vijayakumar, S. Jha, V. Ganesh, Z. Wang, Learning modulo theories, arXiv preprint arXiv:2301.11435 (2023). [12] P.-W. Wang, P. Donti, B. Wilder, Z. Kolter, Satnet: Bridging deep learning and logical reasoning using a diferentiable satisfiability solver, in: International Conference on Machine Learning, PMLR, 2019, pp. 6545–6554. [13] M. Taddeo, L. Floridi, Solving the symbol grounding problem: a critical review of fifteen years of research, Journal of Experimental & Theoretical Artificial Intelligence 17 (2005) 419–445. [14] A. Santoro, A. Lampinen, K. Mathewson, T. Lillicrap, D. Raposo, Symbolic behaviour in artificial intelligence, arXiv preprint arXiv:2102.03406 (2021). [15] B. Hjørland, Nine principles of knowledge organization (1994). [16] R. Davis, H. Shrobe, P. Szolovits, What is a knowledge representation?, AI magazine 14 (1993) 17–17. [17] J. McCarthy, From here to human-level ai, Artificial Intelligence 171 (2007) 1174–1182. [18] K. Gref, S. Van Steenkiste, J. Schmidhuber, On the binding problem in artificial neural networks, arXiv preprint arXiv:2012.05208 (2020). [19] J. McCarthy, Elaboration tolerance, in: Common sense, volume 98, 1998, p. 2. [20] Ł. Kaiser, S. Bengio, Discrete autoencoders for sequence models, arXiv preprint arXiv:1801.09797 (2018). [21] M. Norouzi, D. J. Fleet, R. R. Salakhutdinov, Hamming distance metric learning, Advances in neural information processing systems 25 (2012). [22] M. Diligenti, M. Gori, C. Sacca, Semantic-based regularization for learning and inference,
Artificial Intelligence 244 (2017) 143–165. [23] M. Lombardi, M. Milano, A. Bartolini, Empirical decision model learning, Artificial
Intelligence 244 (2017) 343–367. [24] E. Misino, G. Marra, E. Sansone, Vael: Bridging variational autoencoders and probabilistic
The definitions we provided in Section 3.1 are general enough to be applicable to a broad variety of tasks. We hereby instantiate those definitions to some exemplar metrics, applicable in specific contexts. We will mainly focus on the supervised setting in which labels are available for each of the entities to be encoded, and we will try to generalize where possible. It is important to note how more interesting cases, such as the unsupervised setting, present challenges which make quantification dificult. In Section 3.2 we discussed a possible taxonomy of metrics, to better navigate such complex landscape.
In the supervised case, discrete representations of discrete entities allow to build a representation matrix, where rows are associated with entities and columns with representations. Each entry ℛ counts the times in which representation is used to encode entity . In general this matrix is rectangular, and it can be seen as a generalization of the confusion matrix, where predicted labels are replaced by their representations.
Purity and non-ambiguity can be defined in terms of the representation matrix, and the ideal condition would correspond to a matrix where every row and every column have exactly one non-zero entry (such as Table 3). Purity can be naively computed on the representation matrix as a pseudo-recall, but a more robust metric would be the column-wise average of the entropies computed for each row (after a row-wise normalization of counts). Likewise, non-ambiguity can be computed as a pseudo-precision, or as the row-wise average of the entropies for each column (normalized column-wise). For both metrics, zero is the optimal value, and larger positive values are progressively worse. Appendix B contains an example on how to compute purity and non-ambiguity from a representation matrix.
Usefulness is strongly coupled with the nature of the downstream task, thus there are no general metrics. Moreover, it follows from the definition that a suitable metric should compare two diferent representations. Since downstream tasks can be monitored by means of performance metrics (even in the case of weakly or unsupervised symbols), we propose to quantify usefulness as a direct comparison of the two performance scores. This comparison can be instantiated as a diference, ratio, log-ratio, or any other suitable function.
Symbol and manipulation composability are challenging to quantify, however it is still important to define an order relation between methods. To give a motivating example for this claim, consider the case of two systems, one of which combines symbols by concatenating their representations, and the other constructing a causal graph where nodes are grounded with the representations: these two approaches clearly possess a diferent degree of symbol composability. Likewise, for manipulation composability, the repeated application of the same multi-layer perceptron block a fixed number of times, transformer-like blocks, and the unbounded iteration of a recurrent architecture where gates are conditioned by external inputs, have diferent expressivity. We do not have a practical solution, however we advocate for the need of a (preferably formal) hierarchy of methods, which reflects the intuitive comparisons made above. We postulate that a Chomsky-like hierarchy, where classes are defined by constraints on the allowed manipulations, may serve the purpose.
Generalizability can be evaluated for multi-task settings or when it is important to assess
robustness with respect to domain shifts. We suggest three possible approaches for quantifying
generalizability: a naive direct comparison (similar to usefulness), the use of multi-task
benchmarks (possibly with an aggregated final score), or the use of continual learning experimental
setups [
In this section we show how to compute purity and non-ambiguity using the entropy-based metrics proposed in Appendix A. For this example we perform computations with base 2 logarithms and assume a numpy-like indexing (rows-first, indexes go from 0 to − 1).
Let us assume a task where supervision is given for five symbols (e.g., classes), , , , , associated with six possible representations, , , , , , (note that we abstract from the specific encoding used, i.e., could be associated with a string, a sequence of bits or an ordinal number, the only consideration here is that is diferent than , and so on). After training, each symbol is mapped to the representations with a number of occurrences indicated by Table 1. Note how there is symbol imbalance ( is represented 11 times, while only twice).
Also note how for symbols and , there is only one possible representation (if considered alone their representation would be perfectly pure), and how representation is associated only to symbol (so it is a perfectly non-ambiguous representation). The purity and non-ambiguity metrics proposed try to quantify such behavior globally.
Let us now compute purity. We first normalize the counts by row, which produces the matrix on the left in Table 2. We then compute the entropy of each row as follows: 5− 1 = ∑︁ − · log2( ).
=0 j A 0 B 1 C 0 D 0
E 0 Total 1
For non-ambiguity, the process is the same, but rows and columns are switched (we call the matrix normalized by columns). Entropies are computed column-wise as follows: 5− 1 = ∑︁ − · log2( )
Table 3 is a diferent example, where an encoding perfectly matches symbols to representations (this can be the case of a one-hot encoding output by a perfect classifier, for example). Note how, in this case, normalizing row-wise and column-wise will both produce a permutation of the identity matrix, and thus how entropy is zero, no matter from which direction it is computed.
An important point to keep in mind when using a representation matrix is sparsity, for two reasons. The first one is storage or computation eficiency, as large representation sets grow exponentially with the degrees of freedom of representations (e.g., 32-bit binary representations for 10 classes have a 10 × 232 representation matrix, which cannot be stored densely). Since the validation and test sets of a machine learning setup are typically assumed to be finite in size, it may be more eficient to store occurrences as associations (e.g., dictionaries mapping entities and representations, which will only grow linearly with the dataset and representation sizes), but this may result in slower computations for denser representations. The other consideration is related to the normalization term in the final average of non-ambiguity computations, as it can be computed as the total number of representations, or as the number of representations with non-zero occurrence. The former choice can lead to an underestimation of non-ambiguity in the case of a sparse representation space (where many entries in the representation matrix are zero).3 It can be argued, however, that many symbolic representations, especially those rooted in logic, work better with concise (and thus dense) representations, and that the latter choice, of averaging with respect to non-zero entries, would break the symmetry with respect to purity computation.
In Tables 4 and 5 we report the performance, in terms of F1-score, achieved by neural classifiers and, respectively, decision trees, on the digit (D) and sum (S) classification tasks, with the considered combinations of regularization hyper-parameters in the loss function. These are the same data represented in graphical form in Figure 1. Similarly, Table 6 reports the values of purity and non-ambiguity as a function of the number of bits used for the representations.
P N P N 8 12 16 20 24