ion about phenomena in the environment being observed in the layers below. Through an iterative process, the diferences between the elements in successive iterations within a given layer are captured as transformations between the elements and used for identification and recognition of objects as well as prediction and verification of the environment in future iterations. A bridge between the subsymbolic and symbolic levels can be built by successively adding layers at ever more sophisticated levels of abstraction. This approach aims to benefit from subsymbolic learning, while harnessing the abstraction and reasoning powers of classical symbolic AI techniques.
stage : stage + :
attention −−−−−−−−−−−→
attention −−−−−−−−−−−→ identification −−−−−−−−−−−→ recognition −−−−−−−−−−−→ 1 1 Figure 1: Initial image at subsymbolic level showing the identification of areas of interest in one stage, and the subsequent recognition of objects at a later stage.
Suppose we want to describe in a symbolic way a scene perceived as a sequence of still images, each represented as a binary matrix of × picture elements (pixels) = 0, 1, 2, …. For simplicity, let us assume that these elements are binary, i.e., 0 or 1 (resp., ‘white’ – background, and ‘black’ – foreground). We can interpret each matrix as a snapshot of the world at a particular point in time and the sequence of matrices as the world’s evolution over time. This paper describes what is involved, aiming to provide a blueprint for investigation.
In the first stage, we distinguish objects in the scene and associate symbols to them (an ongoing problem
both in philosophy [
Consider the matrices in Figure 1. It is not dificult to identify at the subsymbolic (pixel) level the areas in
red representing structured elements (or patterns), which we refer to as objects, within our unstructured
input space. Let us assume that this can be done in an unsupervised manner using, e.g., an attention
function [
Our first objective is therefore to create a set of symbols and link these with a subsymbolic representation.
To this end, the representation of an object, which we call a signature, exists in a more abstract space [
By comparing distinct objects (in the same image) we can learn about similarities and diferences [
1 0 −−−−−−−−→ 0
Assuming we have mechanisms for uniquely identifying and recognising objects, we want to represent how objects change over time, aiming to express temporal transformations symbolically, retaining the association between the transformation and the representation. To be clear, if we perceive a change in the AoIs associated with a recognised object in successive images, we also want to associate a symbol with this change, allowing future reasoning about the transformation’s ‘meaning’.
Consider the transformation 0 of matrix 0 into 1 on the left of Figure 2. At the subsymbolic level, we can consider the transformation applied to 0 as a whole, but to reason about the objects in the image, we are more interested in the transformations applied to the AoIs 1, 2 and 3, denoted by 01, 02, and 03, respectively (see right hand side of Figure 2). Assume that the area is associated with object in matrix . For example, 01 is associated with object 1 in 0 and 11 with object 1 in 1, etc. Our task is to define the commutative diagram in Figure 3, so that ( ) is indeed a faithful representation of object in .
The ∗ in the commutative diagram of Figure 3, which shows the relationship between subsymbolic and symbolic levels, is intentional because in general multiple translations may be necessary to achieve the right level of abstraction at the symbolic level. In addition, the dotted lines indicate that the transformations may not necessarily be precise (see Section 4). This potential need for multiple translations is especially important in the generalisation of transformations involving the same object through a sequence of matrices. atomic symbol (
Consider the problem of defining the concept of moving an object ‘up’ in a matrix and then generalising this notion to other objects. At the subsymbolic level, this operation transforms the region 01 (of 0) into 11 (of 1). We could give this transformation a symbol so that, e.g., the transformation 01 is represented by the 01), but we would not be able to describe (in symbolic terms) what this transformation entails, hence it will be impossible to generalise it or to reason about it in more abstract terms. This means that in general we want ( 01) not to be atomic in our symbolic language and, consequently, ( 01) should be expressed in terms of how it afects some specific properties of objects (in this case 1’s ‘location’). ( 01) must provide a faithful representation of 01, meaning that the transformation from ( 01) into ( 11) must be such that ( 11) indeed represents 11, i.e., efectively how 1 would be identified/recognised in 1 (hence commuting the diagram of Figure 3). Because of the complexity of the operations involved, it may be suficient for this process to simply be a “good enough” approximation.
The next question is then what should the appropriate level of granularity of these representations be for our exercise? The answer to this question is not simple, but in principle, we want the granularity to reflect the level of reasoning we hope to be able to capture. As an example, suppose that we need the relative positioning of the “boundaries” of 1 within 0 and 1 (a reasonable assumption if we need to express the “movement” of objects). This means that we need to assume some coordinate/geometrical primitives which are available to us at the subsymbolic level, so that we can understand how to represent them symbolically. Thus, the identification process of 1 must include these elements yielding a signature for 1 that is not only suficient for the recognition of 1 in diferent matrices, but that also contains the basic ingredients for the description of the properties of 1 that we need at the symbolic level. One such property could be, for example, 1’s relative position with respect to a common coordinate or to another area of interest (e.g., another object). Once we are happy with the basic components of 1’s signature we can associate it with 01 and our job will be complete if ( 01) is capable of producing a faithful symbolic representation of the signature 11 in the form of ( 11).
We gloss over a number of complications for now, but it should be easy to understand that we can only describe concepts at the symbolic level that we can somehow capture at the subsymbolic one. In practice, this means that what we associate with 1 in Figure 1 at the subsymbolic level needs to capture enough of the relationship of 1 with the rest of so that we can describe it at the symbolic level in suficient detail for the application in mind.
Consider the game of Pong mentioned in Section 1, which is represented as a sequence of matrices = 0, 1, … at the subsymbolic level. Some areas in the matrices are associated with objects, i.e., the ball and paddles. The objects may change in the sequence, e.g., by appearing in diferent locations in the matrices. These changes represent the notion of movement of the objects and can be used to describe how the scene evolves through time. We can perceive the changes between the matrices as transformations 0, 1, etc (see Figure 4). The challenge is to find a mechanism that describes the transformations in a way that is adequate for the application at hand. For example, we would like to describe 0 as a change in the relative -coordinate of the left paddle and of the -coordinate of the ball in 1 with respect to their values in 0 while still being able to associate the corresponding AoIs in 1 with the ‘same’ objects identified in 0. Eventually, we want to produce a sequence of representations ( 0), ( 1), etc, that describe the objects in a symbolic language, leading to symbolic representations ( 0), ( 1), …, etc, of the matrices themselves. 0 1 2 3 4 0 −−→0 1 −−→1 2 Figure 4: Transformations across a sequence of images. 2 −−→ 3 3 −−→ 4
To a large extent, we have only considered objects in isolation, but in general the subsymbolic representation may encode domain information about the objects’ relationships that is valuable for symbolic reasoning. In our example, information embedded in the matrices , such as the total number of objects, or their direction of travel with respect to each other, will not generally be captured by individual signatures of objects identified. An important consideration is therefore how to incorporate general knowledge Γ that we need to add to ( ) s.t. Γ ∪ (
) is also faithful with respect to for the particular application. Note that
this notion of general knowledge is related (but distinct) to that of common sense priors mentioned in [
The transformations should eventually allow us to perform basic predictions about what future matrices should look like, e.g., through simulating the application of transformations to generate the next state. Verification of the predictions against actual inputs should provide opportunities for revision that generate more accurate translations with time. This results in a temporal aspect of the whole process as well. This dimension is depicted along the horizontal axis of the diagram of Figure 5. 0
In this paper, we proposed the conceptualisation of a multi-layered framework for neuro-symbolic integration with learning. Although we have not provided a concrete instantiation, this level of separation between symbolic and sub-symbolic reasoning, and the proposed mode of integration, is novel and generalises other approaches by providing an initial scafold upon which to employ specific techniques.
The initial focus is on how to define the basic ingredients with which a generic neuro-symbolic process can be devised that allows the association of subsymbolic components to symbolic counterparts, in a way that avoids hand-crafted symbols. We envisage the definition of object ‘signatures’ associating areas of interest in an image, the process that recognises them, and other components derived over time, and we postulate that comparisons between signatures can allow for the definition of some primitive concepts, such as properties, transformations, etc. More complex concepts can then be built from these in layers at increasing levels of abstraction, eventually leading to the development of a symbolic language over time arising from completely subsymbolic inputs.
Finally, we see revisions of the model arising through comparisons of predictions of future states with actual inputs. In future work, we will investigate how interactions and relationships between objects can be captured symbolically using these ideas.
This work was supported by UK Research and Innovation grant number EP/S023356/1, in the UKRI Centre for Doctoral Training in Safe and Trusted Artificial Intelligence ( https://www.safeandtrustedai.org).