In this paper, we present a synopsis of the PlayGround project. Through neural-symbolic learning and reasoning, the PlayGround project assumes that high-level concepts and reasoning processes can be used to advance both symbol grounding and object afordance inference. However, a prerequisite for reasoning about objects and their afordances is integrated object representations that concurrently maintain symbolic values (e.g., high-level concepts), and sub-symbolic features (e.g., spatial aspects of objects). Integrated representations that, preferably, should be based upon neural-symbolic computation such that neural-symbolic models can, subsequently, be used for high-level reasoning processes. Nevertheless, reasoning processes for symbol grounding and afordance inference often require multiple inference steps. Taking inspiration from the cognitive prospects in simulation semantics, the PlayGround project further presumes that these reasoning processes can be simulated by neural rendering complementary to high-level reasoning processes.
The meaning of an object goes beyond the symbol used to refer to it. One of the tenets of
intelligence is the ability to solve the symbol grounding problem, also known as the representation
grounding problem. In symbol grounding [
1 2 3 ???? ?
Afordability inference is a task that comes relatively easily to us humans. Our ability to
perform afordability inferences stems from the fact that we are able to index words and phrases
to objects in the world to prototypical symbols of those objects. Once we derive afordances
from those objects, such afordances constrain the way objects can be coherently combined
– they determine what is possible and what is not. Afordability inference is, however, a task
that, in many ways, is challenging, saying the least, for robotic systems, as exemplified in
Figure 1. Cognitive scientists have long advocated that the mechanisms by which afordability
inference is possible are through a process called simulation semantics – the process by which
we understand and reason about utterances by simulating their content, using similar constructs
to both perception and control. Reasoning about afordances can, arguably, proceed at a high
level using symbolic representations. However, sub-symbolic representations are needed to
capture the 3D spatial aspect of objects, as 2D representations are inherently limited in both
capturing adequate dimensional features and features that are invariant to camera motions [
An underlying assumption of the PlayGround project is that high-level concepts and reasoning processes can be leveraged to improve symbol grounding and afordance inference. This assumption implies integrated representations, administrated by a symbolic – sub-symbolic framework, that can cope with both the high-level concepts (symbolic), as well as 3D spatial aspects of objects (sub-symbolic). Furthermore, this symbolic – sub-symbolic framework should, intuitively, be based upon neural-symbolic computation such that neural-symbolic models, subsequently, can be used for high-level reasoning processes. Based on the observation that reasoning processes for symbol grounding and afordance inference often require multiple inference steps, a further assumption of the PlayGround project is that these processes can be simulated (in alignment with the cognitive perspective on simulation semantics). It is, therefore, natural to develop a simulation framework for reasoning about symbol grounding and afordances. The simulation framework will use neural rendering to generate possible 3D scenes and sequences of such scenes, given language statements and instructions. This semantic simulation renders low-level 3D scenes and is, hence, complementary to the high-level reasoning processes, supported by the neural-symbolic approach.
In summary, the overall goal of the PlayGround project is to “[. . . ] contribute novel techniques for afordance inference and for symbol grounding that are based on 1) an integrated symbolic – sub-symbolic framework, and 2) a semantic simulation framework.”
Symbol grounding for physically embedded systems has followed several diferent tracks. One
track learns the meaning of words in the sensorimotor space of the robot using neural networks.
Typically, features are extracted from the perceptual data, and the output of the network is
tightly coupled to the exact motor configuration of the robot [
Learning object afordances has been reported in correlation to both semantic object
recognition [
This section presents a selection of previous results. Results that have paved the way for the PlayGround project, which we therefore also count as preliminary results of PlayGround.
In previous work [12], we have presented ProbAnch – a modular data-driven probabilistic anchoring framework. The novelty of this framework is the integration of data-driven bottom-up anchoring [13], together with probabilistic reasoning based on dynamic distributional clauses (DDC)[14]. This integration allows the ProbAnch framework not only to create and maintain representations of objects (i.e., anchors) based on perceptual observations (derived from subsymbolic sensory data), but also to reason about objects in the absence of perceptual inputs (e.g., in the case of object occlusions), using a combination of logical, probabilistic, and neuralsymbolic methods. In other words, ProbAnch is a framework for handling semantic world modeling with an extension for semantic relational object tracking [15, 16], as seen in Figure 2.
PROBANCH
In parallel with the work that resulted in ProbAnch, we have additionally presented initial work on learning generative image manipulations from language instructions using a semantic simulation framework [17]. This work has explored whether a perceptual visual system can simulate human-like cognitive capabilities by training a computational model to predict the output of actions expressed through language instructions. Using a combination of language instructions and images pairs of objects before and after state, as the efects of manipulation actions, the computational model was trained in the settings of a generative adversarial network (GAN)[18] in order to generate simulated images that visualize the efect of an action on a given object, i.e., a synthetic generated image that demonstrates the efect of a certain basic manipulation action (e.g., move, remove, add, and replace). Aiming to bridge the gap between simulation and the real world, the trained computational model was subsequently tested in real-world scenarios, as illustrated in Figure 3.
Language instructions
Real-world image pairs
Generated images
Target images “replace the green small cube with a blue big pyramid” “add a green small cube on top of red big sphere”
A natural direction of future work within the PlayGround project would be to integrate both frameworks, as outlined in the previous section. Such integration would allow the symbolic – sub-symbolic framework to utilize the semantic simulator, in combination with language instructions, to predict the future whereabouts of objects and thereby inject positional probability distributions to support the subsequent anchoring of the objects (once the objects are perceived through sensory observations). However, the data used to train the generative model of the semantic simulator was neither realistic (image-wise), nor expressive (language-wise). As a result, the model failed to generate representative images while tested in real-world settings (as seen by the generated images in Figure 3). Taking inspiration from the CLEVR [19] and CLEVRER [20] approaches to rendering visual representations together with language questions, an initial objective of PlayGround is to develop a synthetic generator for generating realistic synthetic scenarios. This generator should be qualified to generate realistic (and expressive) scenarios so that the scenarios can be transferred to real-world settings. Essential for the PlayGround project is that the generator is also incorporating the notion of afordances, both in terms of afordances given visual representations, as well as afordances in language instructions. Generated synthetic scenarios can, thereby, be used to advance the development of novel techniques for both afordance inference and symbol grounding. Given a qualified synthetic generator, the long-term objectives of PlayGround are, subsequently, to develop integrated symbolic – sub-symbolic representations for supporting high-level reasoning processes, as well as a semantic simulator utilizing neural rendering techniques.
In this paper, we have presented a summary of the PlayGround project. In PlayGround, we emphasize symbol grounding and afordance inference using neural-symbolic reasoning and semantic simulation. Inspired by CLEVR [19] and CLEVRER [20], we promote the use of a synthetic generator for generating scenarios representative of symbol grounding and afordance inference problems. Based on generated scenarios, the grander ambition of PlayGround is thereafter to develop both a symbolic – sub-symbolic learning and reasoning framework (i.e., a neural-symbolic framework), and a semantic simulator framework. Furthermore, as both frameworks are tightly connected and likewise intended for symbol grounding and afordance inference, we expect to additionally be able to exploit synergies between neural-symbolic reasoning and semantic simulation.
First of all, we would like to acknowledge the larger consortium of this project that, besides the authors of this short paper, consists of Prof. Luc De Raedt and two senior researchers, Marjan Alirezaie and Martin Längkvist. Thank you for all the work on the proposal that realized the PlayGround project. In addition, this project is funded and supported by the Swedish Research Council (sv. Vetenskapsrådet), grant number: 2021-05229. [11] I. Vendrov, R. Kiros, S. Fidler, R. Urtasun, Order-embeddings of images and language, arXiv preprint arXiv:1511.06361 (2015). [12] A. Persson, P. M. Zuidberg Dos Martires, L. De Raedt, A. Loutfi, Probanch: a modular probabilistic anchoring framework, in: Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence„ International Joint Conferences on Artificial Intelligence, 2020, pp. 5285–5287. [13] A. Loutfi, S. Coradeschi, A. Safiotti, Maintaining coherent perceptual information using anchoring, in: Proc. of the 19th IJCAI Conf., Edinburgh, UK, 2005, pp. 1477–1482. [14] D. Nitti, T. De Laet, L. De Raedt, Probabilistic logic programming for hybrid relational domains, Machine Learning 103 (2016) 407–449. [15] A. Persson, P. Z. Dos Martires, L. De Raedt, A. Loutfi, Semantic relational object tracking,
IEEE Transactions on Cognitive and Developmental Systems 12 (2020) 84–97. [16] P. Zuidberg Dos Martires, N. Kumar, A. Persson, A. Loutfi, L. De Raedt, Symbolic learning and reasoning with noisy data for probabilistic anchoring, Frontiers in Robotics and AI 7 (2020) 100. [17] M. Längkvist, A. Persson, A. Loutfi, Learning generative image manipulations from language instructions, in: Concepts in Action: Representation, Learning, and Application (CARLA 2020), Virtual workshop, September 22-23, 2020, 2020. [18] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial nets, in: Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, K. Q. Weinberger (Eds.), Advances in Neural Information Processing Systems 27, Curran Associates, Inc., 2014, pp. 2672–2680. URL: http://papers.nips.cc/paper/ 5423-generative-adversarial-nets.pdf. [19] J. Johnson, B. Hariharan, L. Van Der Maaten, L. Fei-Fei, C. Lawrence Zitnick, R. Girshick, Clevr: A diagnostic dataset for compositional language and elementary visual reasoning, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2901–2910. [20] K. Yi, C. Gan, Y. Li, P. Kohli, J. Wu, A. Torralba, J. B. Tenenbaum, Clevrer: Collision events for video representation and reasoning, in: International Conference on Learning Representations, 2019.