Inferring Dispositions from Object Shape and Material with Physics Game Engine Modelling Abhijit Vyas1 , Daniel Beßler1 and Michael Beetz1 1 Department of Artificial Intelligence, University of Bremen, 28359 Bremen, Germany Abstract Dispositional qualities are the characteristics of an object that are attributed to it’s properties, such as mass, color, shape, material, etc. Understanding how the design of an object affords such qualities is a crucial task in robotics. Such as a cup being functionally designed to hold or contain something, and structurally designed to be carried or grasped by its handle. Dispositions tend to be more independent of an environment than affordances, since they are related to fundamental characteristics of an object. Whereas, affordances define the action possibilities with the object in the given environment with an agent capable of manipulating them, such as a bottle of water affords drinking possibility to an adult but it is hard for an infant to open the bottle cap in order to drink from it. The topic of affordances is widely explored in the domain of robotics where it plays a vital role for basic object manipulation skills. In this paper, we present an approach for disposition learning about an object from it’s shape and material information provided by a physics engine. We postulate our hypothesis around the current state of the art game engines which have complex object rendering and modelling techniques. The modelling of shape and material information about the object can be harvested as a source of knowledge for the given object in the environment. An intelligent agent thus benefits from having prior information about such objects in the world. Keywords dispositions learning, affordances, ontology population, autonomous robotics 1. Introduction Performing everyday activities such as cleaning, setting up the table and cooking comes naturally to us as humans even in an unknown environment. However, this is something we humans have learned through multiple experiments over our lifetime. In addition, we are able to use our common sense knowledge to derive certain facts that can be helpful for performing tasks efficiently. Such knowledge is not easily available or transferable to artificial agents. It is widely accepted that the agent’s capability to manipulate objects in unknown environments is an indicator for intelligent behaviour. Although, doing such tasks under realistic conditions is a big challenge for artificial agents. In computer vision, object recognition techniques are mainly focused on visual appearance and in some extent shape detection of objects. However the classification of objects can be done RobOntics 2021: 2nd International Workshop on Ontologies for Autonomous Robotics, held at JOWO 2021: Episode VII The Bolzano Summer of Knowledge, September 11–18, 2021 " avyas@uni-bremen.de (A. Vyas); danielb@uni-bremen.de (D. Beßler); beetz@cs.uni-bremen.de (M. Beetz) ~ https://ai.uni-bremen.de/team/abhijit_vyas (A. Vyas); https://ai.uni-bremen.de/team/daniel_bessler (D. Beßler); https://ai.uni-bremen.de/team/michael_beetz (M. Beetz) © 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) in many different ways. Gibson argued that the main objective of such vision techniques should be to understand the possible interactions between an agent and an environment in a scene rather than merely identifying contents of the scene [1]. He put forth an idea of affordances in order to understand such interactions: “The affordances of the environment are what it offers the animal, what it provides or furnishes, either for good or ill. The word affordance implies the complementarity of the animal and the environment. [1]” Gibson’s affordance theory forms the relationship between an action, an object, and the effects of the action on the object. An agent can use exploratory behaviors when facing a new object while interacting with the world. It can be stated that the properties of an object that an agent is likely to learn are directly linked to the behavioral procedures an agent follows [2]. Identifying such properties can be relatively straight forward, since the agent can follow a sequence of exploratory behaviors and observe their effects. In order to realize an affordance via interaction, two complementary dispositional match should occur, where bearer and trigger roles are assumed by various participants in the event. Such as, during a cutting action, a knief will be a cutter and a piece of bread will a cuttie. Learning about such complementary dispositions is useful to detect situations under which potential of interaction can be explored. However, there are many other views on the interpretation of the term affordances and ontological categorisation of it. Some researchers consider affordances as an event and others say it is a quality of an object. In this paper, we commit to the Descriptive Affordance Theory [3] which states that the affordance-manifestation is a situation, and the affordance is the description of this manifestation. Here, Disposition is defined as an object quality that allows an object to participate in events that realize an affordance [4]. This is inspired by Turvey’s [5] notion of affordances as dispositional characteristics of an object which states: "An affordance is a particular kind of disposition, one whose complement is a dispositional property of an organism". It is evident that the basic dispositional qualities of an object are tightly linked to its shape and material attributes. In this work, we try to shade some light on how dispositional qualities can be grounded in such attributes, and how this knowledge can be incorporated with an existing ontological model. Our approach is to exploit the detailed shape and material models used in state of the art game engines. Modern game engines, such as the Unreal Engine and Unity, have enabled users to create virtual environments with realistic modelling of 3D objects. In this work, we study the 3D object modeling in the Unreal Engine, and explore ways to translate them into dispositional qualities. The main contributions of this paper are the following ones: (1) A classification of different types of object dispositions; and (2) a methodological approach for object classification based on material and shape information from the Unreal Engine. In the following, we will first provide an overview of different related works concerning learning of affordances. Thereafter, we will investigate object modelling in the Unreal Engine which can be utilized for inferring dispositions. After that, we propose our approach for using information from Unreal and incorporate this knowledge in an ontological model. At last, we will discuss preliminary results and the future work. 2. Related Work Affordance modelling has been investigated for many years. There has been many efforts for an affordance detection of an object while observing it’s physical characteristics such as color, shape, material and so on. Schmidt et al. [6] have used associative as well as estimation approaches for finding me- chanical properties such as stiffness of various 3D modelled objects using Blender 2.76. They have performed various experiments in order to prove the effect of material and shape changes on stiffness. They conclude that both shape and material characteristics of an object play an important role for stiffness measurement under varying conditions although the effect of those two can be dependant on the situation under which an observer perceives an object. From a design point of view, Andries et al. [7] have explored how the functional design of an object plays a role in shaping the object. Each object is designed to fulfil certain functional re- quirements which can be interpreted as affordances. They have used an artificial neural network to learn the function to form (shape) mapping from affordance labelled data about the objects. Then, they have used various shapes that can provide one or more desired affordances and tried to generate the shape which can provide all of the desired affordances. They were successful in finding out the shape that can have either contain-ability, sit-ability and supportability or all of them. Tool substitution is the problem which addresses the case where an intelligent agent is supposed to perform the task with a specific tool, but in the absence of such tool, it has to look for an alternative tool within the environment. E.g. if the robot can not find the lid of a vessel to cover, then it should use the plate instead. Shrivatsav et al. [8] have introduced shape and material reasoning to effectively identify the substitute tool for the given task. They train two separate twin neural networks, also known as siamese neural network, with shape and material properties of an object. These properties are responsible for providing the specific affordance of an object for a given task, and later combine the output of both networks to rank various tools for the given task. Ardón et al. [9] have considered the role of an environment while learning about object’s affordances. They discuss about a social study on the development of human cognition. This study helps us to understand that the interactive learning with objects within our environment is not only the result of our past experiences but also our capacity of inferring based on the context of the environment where these objects are placed. This is useful for creating a relationship between the object, the context or the situation under which an object is perceived, and the set of possible actions that can be performed in order to interact with it. Moldovan et al. [10] have explored relational affordance models for multi-object manipulation tasks. The authors use statistical relational learning (SRL) which allows to build a single model to support inference for multi objects affordances. They claim that their approach with SRL provides ease of modelling by allowing the transfer of the model structure to other domains. They use the iCub robot platform for experimenting with a multi-object shopping shelf scenario. Montesano et al. [11] have presented the general affordance model based on Bayesian net- works, which links actions, object features and action effects. They use object’s shape, size, color, velocity, and other parameters such as an agent’s action, hand velocity, and distance from object to train their affordance network. The network learns through interaction with the environment by an agent which is similar to an exploratory behaviour approach we have discussed in the previous section. Tenorth et al. [12] have developed a system to identify functional parts of an object from CAD models as a source of knowledge while performing logical inference with Prolog rules. The system was able to identify geometric primitive shapes such as planes, cylinders and spheres. By recognizing such components, they were able to answer affordance related questions such as which object should I use to pour one liter of liquid? Or which part of the bottle should one grasp in order to open it? 3. Physics Engine Object Modelling In this section, we present an overview of the modelling approaches that we think are useful to infer dispositions of objects based on their shape, material and physical attributes provided by the Unreal Engine. One of the advantages of using a physics engine is that the modeling of objects can be done realistically using physical properties such as mass, center of mass, density, friction, restitution, and so on. 3.1. Shape and Size When an agent interacts with various shapes presented in the environment, it can gather information about the physical nature of such shapes. For example, spherical objects are easy to roll, and boxes are easy to slide. During such interactions, an agent can also learn about the ideal posture of the object such as that a WineGlass should be held upwards as shown in Figure 1. If it is placed sideways, then it can slide or roll down the table. Through such Figure 1: A wine glass in upward position. experimentation in a game environment, an agent can learn about how various shapes impose constraints on the pose of an object, and its stability. The Unreal Engine provides modelling of basic primitive types such as spheres, boxes, capsules, convex elements, and tapered capsules 1 . Such shapes can be useful in deriving dispositions such as roll-ability, slid-ability and containment. Shape and size also influences material attributes of an object such as rigidity or brittleness. One can argue, from a material point of view, that objects made from glass tend to be more brittle, but it is the shape and size which contributes the degree of brittleness. An example is the high storey building windows made from glass which are as strong as concrete. Although in a everyday kitchen scenario, we do not encounter such complex cases, hence, it is out of scope for this work. 3.2. Physical Material Humans have adapted themselves to identify visual cues from material, and can associate those cues with different material characteristics. We can identify if an object is rigid, brittle, rough or slippery based on our observations of the material surface. However, material information is something which is not readily available to an agent in the real world. On the other hand, with virtual reality environments it is needed to model such information in order to render as well as process interactions among various players and objects. Within the Unreal Engine, we can define various surface types 2 such as metallic, wood, plastic, paper, rubber, skin, etc. As can be seen from Table 1, one can also set the friction, restitution and density related parameters of the physical material in order to simulate those surface behaviours. We use this information to assign dispositions such as rigidness, plasticity, elasticity, and brittleness to the given virtual object. Unreal Property Description Friction The friction value of the surface, which controls how easily things can slide on this surface. Restitution Determines how "bouncy" the surface is, or how much energy it retains when it collides with another surface. Density Used with the shape of the object to calculate its mass proper- ties. The higher the number, the heavier the object. Measured as g per cubic cm. Surface Type Defined as an enum value to be used in the engine for defining any number of things, from what sound plays as a character walks across a surface, to the type of decal an explosion should leave on different surfaces. Table 1 Physical material 1 https://docs.unrealengine.com/4.26/en-US/InteractiveExperiences/Physics/PhysicsBodies/Reference/ 2 https://docs.unrealengine.com/4.26/en-US/InteractiveExperiences/Physics/PhysicalMaterials/ PhysMatUserGuide/ Unreal Property Description Mass Mass of the object computed in Kg. Center of Mass offset User defined offset of center of mass for given object. Table 2 Physics attributes 3.3. Physical Attributes It is rather difficult to estimate physical attributes of an object such as mass, and center of mass via visual inspection. One can gain insights only by interacting with the object. For each object in the Unreal virtual world, we can define physical attributes specific to the object. We only consider a few selected physical parameters, as shown in Table 2, such as mass and center of mass. This information along with shape information can be vital for finding balanced poses of the object under the influence of gravity. 4. Approach This section describes the approach we consider to relate dispositional qualities of an object to material and shape information obtained from the Unreal Engine. We employ the information obtained from the physics engine object modelling and link it with the object representation in the ontology model. The SOMA ontology model defines a Disposition concept as an extrinsic physical quality of an object. Extrinsic means here that it depends on relationships to other things, for example, the color of an object depends on light conditions in the environment. We define such relationships by adding new axioms to our model. An example is to check if an object, or a part of it has been made from metal, wood, plastic, rubber, glass or paper. One can add relationships between objects and material dispositions such as that an object or a part of it is rigid, elastic, plastic or brittle. Dispositions Shape and Size Material Physical Attributes roll-ability ✓ ✓ slid-ability ✓ ✓ ✓ Structural Dispositions containment ✓ liquid-containment ✓ ✓ rigidity ✓ ✓ elasticity ✓ ✓ Material Dispositions plasticity ✓ ✓ brittleness ✓ ✓ Table 3 Dispositions provided by shape, size, material and physical attributes We classify dispositions into two main groups: structural dispositions, and material disposi- tions. Structural dispositions such as roll, slide, support and containment are dependant upon Class : LiquidContainment C l a s s : WineGlass SubClassOf : SubClassOf : Containment , Glass , DesignedContainer Structural Disposition h a s D i s p o s i t i o n some a f f o r d s B e a r e r only C o n t a i n e r LiquidContainment a f f o r d s T r i g g e r only L i q u i d h a s M a t e r i a l R e g i o n some G l a s s s y hasComponent some Bowl and hasComponent some Base and hasComponent some Stem Class : Glassy SubClassOf : C l a s s : Bowl MaterialRegion SubClassOf : h a s D i s p o s i t i o n some B r i t t l e n e s s DesignedComponent h a s D i s p o s i t i o n some Containment C l a s s : Base C l a s s : Stem SubClassOf : SubClassOf : DesignedComponent DesignedComponent h a s D i s p o s i t i o n some S u p p o r t h a s D i s p o s i t i o n some G r a s p i n g h a s D i s p o s i t i o n some C o n n e c t i n g Figure 2: An example of how an object and its dispositions are represented ontologically. Figure 3: IAI Virtual Kitchen with objects shape of the object and other physics attributes. The second category of dispositions are ma- terial dispositions which are inherently derived from material characteristics, such as rigidity, elasticity, plasticity and brittleness. Table 3 lists all the dispositions that we consider for our experiment. Figure 2 depicts an ontological representation of object shape and material characteristics Figure 4: An Unreal Physical Material: Rubber of wine glass. We define the Wine Glass concept as a subclass of Glass which is broadly defined as a sub-concept of Designed Container. Thus, each Wine Glass instance inherits a containment disposition from being a container, and a more specific liquid containment disposition. Additionally, Figure 1 depicts that a Wine Glass can have other functional parts such as a base and a stem, which can provide support and grasp dispositions respectively. One should also note that, a wine glass is made of Glassy material which by nature possess brittleness or fragility as a material disposition. We have established an experimental setup with the Unreal Engine resembling a domestic kitchen environment, as shown in Figure 3. Kitchens are challenging environments, partly due to the amount of different object types that can be encountered, and various degrees of usefulness to apply them in different situational contexts. We expect this setup to be a valuable source for dispositional learning in our future work. For our test domain, we have modelled different objects with material information, and we have defined various material surface types such as wood, plastic, paper, glass, clay, metal, rubber and created physical materials for each of those surfaces. An example of such physical material parameters is shown in Figure 4 for the material type rubber. So far, we were able to achieve the following objectives: • modeling of objects with material information in the Unreal Engine; and • ontological definition of several material types, and their relation to object dispositions. We were also able to create an interface for querying the game engine data from external components which is important for connecting our approach to the goals of a robotic agent. 5. Discussion and Future work We have presented an approach to infer dispositional qualities from physics engine object modelling. Table 4 presents the objects shown in Figure 3 with their dispositions and material information gathered from the Unreal Engine. We have used material surface types to categorize objects based on material, and have inferred corresponding material dispositions. Object Material Disposition Material Spoon Rigidity Metal Fork Rigidity Metal Butter Knife Rigidity Metal Cork Screw Rigidity Metal, Wood Steak Knife Rigidity Metal, Hard-Plastic Wooden Spatula Rigidity Wood Pizza cutter Rigidity Metal Silicon Spatula Elasticity Rubber Steel Spatula Rigidity Metal Peeling Knife Rigidity Metal Plate Brittleness Hard clay Big bowl Brittleness Hard clay Cup Brittleness, Hard clay Wine glass Brittleness Glass Short glass Brittleness Glass Blue rubber bowl Elasticity Rubber Plastic container Plasticity Plastic Plastic container lid Plasticity Plastic Butter milk bottle Plasticity Plastic Salt bottle Rigidity Hard Plastic Milk carton Elastic-Plastic [13] Paperboard Table 4 List of objects and their material dispositions along with material type As a future task, we are looking into decomposing object meshes into several functional parts and learn potential dispositions of an object. This process will help us to classify objects with various structural dispositions. In order to do that, we need to model each Unreal object with a mesh which can be decomposed into various functional parts such as that a wine glass can be decomposed into bowl, stem, and base. Then, we plan to create a model which can decompose meshes into several primitive shapes. At last, we plan to update the ontology model to incorporate such primitive shapes with corresponding dispositions such as roll-ability, slid- ability, containment, etc. Furthermore, we will draw inspiration from the approach suggested by Tenorth et al. [12] in order to relate primitive types with the functional parts of the object mesh. We will also evaluate our results against some benchmarks such as PartNet [14]. The level of depth at which such object descriptions are useful for robotics tasks are highly influenced by competence and performance measures. We are interested in creating a set of competency questions and will explore at which level of detail an object description should be to answer these questions. 6. Acknowledgements This work was funded by the German Research Foundation (DFG) as part of Collaborative Research Center (CRC) 1320 EASE – Everyday Activity Science and Engineering, University of Bremen (http://www.ease-crc.org/) References [1] J. J. Gibson, The Ecological Approach to Visual Perception, Psychology Press Classic Editions, 1979. [2] A. Stoytchev, Toward Learning the Binding Affordances of Objects: A Behavior-Grounded Approach, 2005. [3] D. Beßler, R. Porzel, M. Pomarlan, M. Beetz, R. Malaka, J. A. Bateman, A formal model of affordances for flexible robotic task execution, in: G. D. Giacomo, A. Catalá, B. Dilkina, M. Milano, S. Barro, A. Bugarín, J. Lang (Eds.), ECAI 2020 - 24th European Conference on Artificial Intelligence, volume 325 of Frontiers in Artificial Intelligence and Applications, IOS Press, 2020, pp. 2425–2432. URL: https://doi.org/10.3233/FAIA200374. doi:10.3233/ FAIA200374. [4] D. Beßler, R. Porzel, M. Pomarlan, A. Vyas, S. Hoefner, M. Beetz, R. Malaka, J. Bateman, Foundations of the Socio-physical Modelof Activities (SOMA) for Autonomous Robotic Agents, in: 12th International Conference on Formal Ontology in Information Systems, 2021. [5] M. T. Turvey, Affordances and prospective control: An outline of the ontology, Ecological psychology, vol. 4, no. 3, pp. 173–187 (1992). [6] F. Schmidt, V. C. Paulun, J. J. R. van Assen, R. Fleming, Inferring the stiffness of unfamiliar objects from optical, shape, and motion cues, Journal of Vision 17(3):18, 1–17 (2017). [7] A. D. Mihai Andries, J. Santos-Victor, Automatic generation of object shapes with desired affordances using voxelgrid representation, Front. Neurorobot. 14:22. doi: 10.3389/fn- bot.2020.00022 (2020). [8] N. S. Srikanth, L. Nair, S. Chernova, Tool substitution with shape and material reasoning using dual neural networks, CoRR abs/1911.04521 (2019). URL: http://arxiv.org/abs/1911. 04521. arXiv:1911.04521. [9] K. S. L. Paola Ardón, S. Ramamoorthy, Object Affordances by Inferring on the Surroundings, in: IEEE Workshop on Advanced Robotics and its Social Impacts (ARSO), 2018. [10] B. Moldovan, P. Moreno, M. van Otterlo, J. Santos-Victor, L. De Raedt, Learning relational affordance models for robots in multi-object manipulation tasks, in: 2012 IEEE International Conference on Robotics and Automation, 2012, pp. 4373–4378. doi:10.1109/ICRA.2012. 6225042. [11] L. Montesano, M. Lopes, A. Bernardino, J. Santos-Victor, Modeling affordances using bayesian networks, in: 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2007, pp. 4102–4107. doi:10.1109/IROS.2007.4399511. [12] F. B.-B. Moritz Tenorth, Stefan Profanter, M. Beetz, Decomposing cad models of objects of daily use and reasoning about their functional parts, 2013. [13] Y. Li, S. E. Stapleton, S. Reese, J.-W. Simon, Anisotropic elastic-plastic deformation of paper: Out-of-plane model, International Journal of Solids and Structures 130-131 (2018) 172–182. URL: https://www.sciencedirect.com/science/article/pii/S0020768317304560. doi:https: //doi.org/10.1016/j.ijsolstr.2017.10.003. [14] K. Mo, S. Zhu, A. X. Chang, L. Yi, S. Tripathi, L. J. Guibas, H. Su, Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding, 2018. arXiv:1812.02713.