=Paper=
{{Paper
|id=Vol-2969/paper79-RobOntics
|storemode=property
|title=Inferring Dispositions from Object Shape and Material with Physics Game Engine Modelling
|pdfUrl=https://ceur-ws.org/Vol-2969/paper79-RobOntics.pdf
|volume=Vol-2969
|authors=Abhijit Vyas,Daniel Beßler,Michael Beetz
|dblpUrl=https://dblp.org/rec/conf/jowo/VyasBB21
}}
==Inferring Dispositions from Object Shape and Material with Physics Game Engine Modelling==
https://ceur-ws.org/Vol-2969/paper79-RobOntics.pdf
Inferring Dispositions from Object Shape and
Material with Physics Game Engine Modelling
Abhijit Vyas1 , Daniel Beßler1 and Michael Beetz1
1
Department of Artificial Intelligence, University of Bremen, 28359 Bremen, Germany
Abstract
Dispositional qualities are the characteristics of an object that are attributed to it’s properties, such as
mass, color, shape, material, etc. Understanding how the design of an object affords such qualities is a
crucial task in robotics. Such as a cup being functionally designed to hold or contain something, and
structurally designed to be carried or grasped by its handle. Dispositions tend to be more independent
of an environment than affordances, since they are related to fundamental characteristics of an object.
Whereas, affordances define the action possibilities with the object in the given environment with an
agent capable of manipulating them, such as a bottle of water affords drinking possibility to an adult but
it is hard for an infant to open the bottle cap in order to drink from it. The topic of affordances is widely
explored in the domain of robotics where it plays a vital role for basic object manipulation skills. In
this paper, we present an approach for disposition learning about an object from it’s shape and material
information provided by a physics engine. We postulate our hypothesis around the current state of
the art game engines which have complex object rendering and modelling techniques. The modelling
of shape and material information about the object can be harvested as a source of knowledge for the
given object in the environment. An intelligent agent thus benefits from having prior information about
such objects in the world.
Keywords
dispositions learning, affordances, ontology population, autonomous robotics
1. Introduction
Performing everyday activities such as cleaning, setting up the table and cooking comes naturally
to us as humans even in an unknown environment. However, this is something we humans
have learned through multiple experiments over our lifetime. In addition, we are able to use
our common sense knowledge to derive certain facts that can be helpful for performing tasks
efficiently. Such knowledge is not easily available or transferable to artificial agents. It is widely
accepted that the agent’s capability to manipulate objects in unknown environments is an
indicator for intelligent behaviour. Although, doing such tasks under realistic conditions is a
big challenge for artificial agents.
In computer vision, object recognition techniques are mainly focused on visual appearance
and in some extent shape detection of objects. However the classification of objects can be done
RobOntics 2021: 2nd International Workshop on Ontologies for Autonomous Robotics, held at JOWO 2021: Episode VII
The Bolzano Summer of Knowledge, September 11–18, 2021
" avyas@uni-bremen.de (A. Vyas); danielb@uni-bremen.de (D. Beßler); beetz@cs.uni-bremen.de (M. Beetz)
~ https://ai.uni-bremen.de/team/abhijit_vyas (A. Vyas); https://ai.uni-bremen.de/team/daniel_bessler (D. Beßler);
https://ai.uni-bremen.de/team/michael_beetz (M. Beetz)
© 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org)
in many different ways. Gibson argued that the main objective of such vision techniques should
be to understand the possible interactions between an agent and an environment in a scene
rather than merely identifying contents of the scene [1]. He put forth an idea of affordances in
order to understand such interactions:
“The affordances of the environment are what it offers the animal, what it provides
or furnishes, either for good or ill. The word affordance implies the complementarity
of the animal and the environment. [1]”
Gibson’s affordance theory forms the relationship between an action, an object, and the
effects of the action on the object. An agent can use exploratory behaviors when facing a new
object while interacting with the world. It can be stated that the properties of an object that
an agent is likely to learn are directly linked to the behavioral procedures an agent follows
[2]. Identifying such properties can be relatively straight forward, since the agent can follow a
sequence of exploratory behaviors and observe their effects. In order to realize an affordance
via interaction, two complementary dispositional match should occur, where bearer and trigger
roles are assumed by various participants in the event. Such as, during a cutting action, a
knief will be a cutter and a piece of bread will a cuttie. Learning about such complementary
dispositions is useful to detect situations under which potential of interaction can be explored.
However, there are many other views on the interpretation of the term affordances and
ontological categorisation of it. Some researchers consider affordances as an event and others
say it is a quality of an object. In this paper, we commit to the Descriptive Affordance Theory [3]
which states that the affordance-manifestation is a situation, and the affordance is the description
of this manifestation. Here, Disposition is defined as an object quality that allows an object
to participate in events that realize an affordance [4]. This is inspired by Turvey’s [5] notion
of affordances as dispositional characteristics of an object which states: "An affordance is a
particular kind of disposition, one whose complement is a dispositional property of an organism".
It is evident that the basic dispositional qualities of an object are tightly linked to its shape and
material attributes. In this work, we try to shade some light on how dispositional qualities can
be grounded in such attributes, and how this knowledge can be incorporated with an existing
ontological model. Our approach is to exploit the detailed shape and material models used in
state of the art game engines. Modern game engines, such as the Unreal Engine and Unity, have
enabled users to create virtual environments with realistic modelling of 3D objects. In this work,
we study the 3D object modeling in the Unreal Engine, and explore ways to translate them into
dispositional qualities.
The main contributions of this paper are the following ones: (1) A classification of different
types of object dispositions; and (2) a methodological approach for object classification based
on material and shape information from the Unreal Engine.
In the following, we will first provide an overview of different related works concerning
learning of affordances. Thereafter, we will investigate object modelling in the Unreal Engine
which can be utilized for inferring dispositions. After that, we propose our approach for using
information from Unreal and incorporate this knowledge in an ontological model. At last, we
will discuss preliminary results and the future work.
2. Related Work
Affordance modelling has been investigated for many years. There has been many efforts for
an affordance detection of an object while observing it’s physical characteristics such as color,
shape, material and so on.
Schmidt et al. [6] have used associative as well as estimation approaches for finding me-
chanical properties such as stiffness of various 3D modelled objects using Blender 2.76. They
have performed various experiments in order to prove the effect of material and shape changes
on stiffness. They conclude that both shape and material characteristics of an object play an
important role for stiffness measurement under varying conditions although the effect of those
two can be dependant on the situation under which an observer perceives an object.
From a design point of view, Andries et al. [7] have explored how the functional design of an
object plays a role in shaping the object. Each object is designed to fulfil certain functional re-
quirements which can be interpreted as affordances. They have used an artificial neural network
to learn the function to form (shape) mapping from affordance labelled data about the objects.
Then, they have used various shapes that can provide one or more desired affordances and tried
to generate the shape which can provide all of the desired affordances. They were successful in
finding out the shape that can have either contain-ability, sit-ability and supportability or all of
them.
Tool substitution is the problem which addresses the case where an intelligent agent is
supposed to perform the task with a specific tool, but in the absence of such tool, it has to look
for an alternative tool within the environment. E.g. if the robot can not find the lid of a vessel
to cover, then it should use the plate instead. Shrivatsav et al. [8] have introduced shape and
material reasoning to effectively identify the substitute tool for the given task. They train two
separate twin neural networks, also known as siamese neural network, with shape and material
properties of an object. These properties are responsible for providing the specific affordance of
an object for a given task, and later combine the output of both networks to rank various tools
for the given task.
Ardón et al. [9] have considered the role of an environment while learning about object’s
affordances. They discuss about a social study on the development of human cognition. This
study helps us to understand that the interactive learning with objects within our environment is
not only the result of our past experiences but also our capacity of inferring based on the context
of the environment where these objects are placed. This is useful for creating a relationship
between the object, the context or the situation under which an object is perceived, and the set
of possible actions that can be performed in order to interact with it.
Moldovan et al. [10] have explored relational affordance models for multi-object manipulation
tasks. The authors use statistical relational learning (SRL) which allows to build a single model
to support inference for multi objects affordances. They claim that their approach with SRL
provides ease of modelling by allowing the transfer of the model structure to other domains.
They use the iCub robot platform for experimenting with a multi-object shopping shelf scenario.
Montesano et al. [11] have presented the general affordance model based on Bayesian net-
works, which links actions, object features and action effects. They use object’s shape, size,
color, velocity, and other parameters such as an agent’s action, hand velocity, and distance
from object to train their affordance network. The network learns through interaction with
the environment by an agent which is similar to an exploratory behaviour approach we have
discussed in the previous section.
Tenorth et al. [12] have developed a system to identify functional parts of an object from CAD
models as a source of knowledge while performing logical inference with Prolog rules. The
system was able to identify geometric primitive shapes such as planes, cylinders and spheres.
By recognizing such components, they were able to answer affordance related questions such
as which object should I use to pour one liter of liquid? Or which part of the bottle should one
grasp in order to open it?
3. Physics Engine Object Modelling
In this section, we present an overview of the modelling approaches that we think are useful
to infer dispositions of objects based on their shape, material and physical attributes provided
by the Unreal Engine. One of the advantages of using a physics engine is that the modeling of
objects can be done realistically using physical properties such as mass, center of mass, density,
friction, restitution, and so on.
3.1. Shape and Size
When an agent interacts with various shapes presented in the environment, it can gather
information about the physical nature of such shapes. For example, spherical objects are easy
to roll, and boxes are easy to slide. During such interactions, an agent can also learn about
the ideal posture of the object such as that a WineGlass should be held upwards as shown
in Figure 1. If it is placed sideways, then it can slide or roll down the table. Through such
Figure 1: A wine glass in upward position.
experimentation in a game environment, an agent can learn about how various shapes impose
constraints on the pose of an object, and its stability.
The Unreal Engine provides modelling of basic primitive types such as spheres, boxes, capsules,
convex elements, and tapered capsules 1 . Such shapes can be useful in deriving dispositions
such as roll-ability, slid-ability and containment.
Shape and size also influences material attributes of an object such as rigidity or brittleness.
One can argue, from a material point of view, that objects made from glass tend to be more
brittle, but it is the shape and size which contributes the degree of brittleness. An example is
the high storey building windows made from glass which are as strong as concrete. Although in
a everyday kitchen scenario, we do not encounter such complex cases, hence, it is out of scope
for this work.
3.2. Physical Material
Humans have adapted themselves to identify visual cues from material, and can associate those
cues with different material characteristics. We can identify if an object is rigid, brittle, rough
or slippery based on our observations of the material surface. However, material information is
something which is not readily available to an agent in the real world. On the other hand, with
virtual reality environments it is needed to model such information in order to render as well
as process interactions among various players and objects.
Within the Unreal Engine, we can define various surface types 2 such as metallic, wood,
plastic, paper, rubber, skin, etc. As can be seen from Table 1, one can also set the friction,
restitution and density related parameters of the physical material in order to simulate those
surface behaviours. We use this information to assign dispositions such as rigidness, plasticity,
elasticity, and brittleness to the given virtual object.
Unreal Property Description
Friction The friction value of the surface, which controls how easily
things can slide on this surface.
Restitution Determines how "bouncy" the surface is, or how much energy
it retains when it collides with another surface.
Density Used with the shape of the object to calculate its mass proper-
ties. The higher the number, the heavier the object. Measured
as g per cubic cm.
Surface Type Defined as an enum value to be used in the engine for defining
any number of things, from what sound plays as a character
walks across a surface, to the type of decal an explosion should
leave on different surfaces.
Table 1
Physical material
1
https://docs.unrealengine.com/4.26/en-US/InteractiveExperiences/Physics/PhysicsBodies/Reference/
2
https://docs.unrealengine.com/4.26/en-US/InteractiveExperiences/Physics/PhysicalMaterials/
PhysMatUserGuide/
Unreal Property Description
Mass Mass of the object computed in Kg.
Center of Mass offset User defined offset of center of mass for given object.
Table 2
Physics attributes
3.3. Physical Attributes
It is rather difficult to estimate physical attributes of an object such as mass, and center of mass
via visual inspection. One can gain insights only by interacting with the object. For each object
in the Unreal virtual world, we can define physical attributes specific to the object. We only
consider a few selected physical parameters, as shown in Table 2, such as mass and center of
mass. This information along with shape information can be vital for finding balanced poses of
the object under the influence of gravity.
4. Approach
This section describes the approach we consider to relate dispositional qualities of an object to
material and shape information obtained from the Unreal Engine. We employ the information
obtained from the physics engine object modelling and link it with the object representation in
the ontology model. The SOMA ontology model defines a Disposition concept as an extrinsic
physical quality of an object. Extrinsic means here that it depends on relationships to other
things, for example, the color of an object depends on light conditions in the environment. We
define such relationships by adding new axioms to our model. An example is to check if an
object, or a part of it has been made from metal, wood, plastic, rubber, glass or paper. One can
add relationships between objects and material dispositions such as that an object or a part of it
is rigid, elastic, plastic or brittle.
Dispositions Shape and Size Material Physical Attributes
roll-ability ✓ ✓
slid-ability ✓ ✓ ✓
Structural Dispositions
containment ✓
liquid-containment ✓ ✓
rigidity ✓ ✓
elasticity ✓ ✓
Material Dispositions
plasticity ✓ ✓
brittleness ✓ ✓
Table 3
Dispositions provided by shape, size, material and physical attributes
We classify dispositions into two main groups: structural dispositions, and material disposi-
tions. Structural dispositions such as roll, slide, support and containment are dependant upon
Class : LiquidContainment C l a s s : WineGlass
SubClassOf : SubClassOf :
Containment , Glass , DesignedContainer
Structural Disposition h a s D i s p o s i t i o n some
a f f o r d s B e a r e r only C o n t a i n e r LiquidContainment
a f f o r d s T r i g g e r only L i q u i d h a s M a t e r i a l R e g i o n some G l a s s s y
hasComponent some Bowl
and hasComponent some Base
and hasComponent some Stem
Class : Glassy
SubClassOf :
C l a s s : Bowl
MaterialRegion
SubClassOf :
h a s D i s p o s i t i o n some B r i t t l e n e s s
DesignedComponent
h a s D i s p o s i t i o n some Containment
C l a s s : Base C l a s s : Stem
SubClassOf : SubClassOf :
DesignedComponent DesignedComponent
h a s D i s p o s i t i o n some S u p p o r t h a s D i s p o s i t i o n some G r a s p i n g
h a s D i s p o s i t i o n some C o n n e c t i n g
Figure 2: An example of how an object and its dispositions are represented ontologically.
Figure 3: IAI Virtual Kitchen with objects
shape of the object and other physics attributes. The second category of dispositions are ma-
terial dispositions which are inherently derived from material characteristics, such as rigidity,
elasticity, plasticity and brittleness. Table 3 lists all the dispositions that we consider for our
experiment.
Figure 2 depicts an ontological representation of object shape and material characteristics
Figure 4: An Unreal Physical Material: Rubber
of wine glass. We define the Wine Glass concept as a subclass of Glass which is broadly
defined as a sub-concept of Designed Container. Thus, each Wine Glass instance inherits
a containment disposition from being a container, and a more specific liquid containment
disposition. Additionally, Figure 1 depicts that a Wine Glass can have other functional parts
such as a base and a stem, which can provide support and grasp dispositions respectively.
One should also note that, a wine glass is made of Glassy material which by nature possess
brittleness or fragility as a material disposition.
We have established an experimental setup with the Unreal Engine resembling a domestic
kitchen environment, as shown in Figure 3. Kitchens are challenging environments, partly
due to the amount of different object types that can be encountered, and various degrees of
usefulness to apply them in different situational contexts. We expect this setup to be a valuable
source for dispositional learning in our future work.
For our test domain, we have modelled different objects with material information, and we
have defined various material surface types such as wood, plastic, paper, glass, clay, metal,
rubber and created physical materials for each of those surfaces. An example of such physical
material parameters is shown in Figure 4 for the material type rubber.
So far, we were able to achieve the following objectives:
• modeling of objects with material information in the Unreal Engine; and
• ontological definition of several material types, and their relation to object dispositions.
We were also able to create an interface for querying the game engine data from external
components which is important for connecting our approach to the goals of a robotic agent.
5. Discussion and Future work
We have presented an approach to infer dispositional qualities from physics engine object
modelling. Table 4 presents the objects shown in Figure 3 with their dispositions and material
information gathered from the Unreal Engine. We have used material surface types to categorize
objects based on material, and have inferred corresponding material dispositions.
Object Material Disposition Material
Spoon Rigidity Metal
Fork Rigidity Metal
Butter Knife Rigidity Metal
Cork Screw Rigidity Metal, Wood
Steak Knife Rigidity Metal, Hard-Plastic
Wooden Spatula Rigidity Wood
Pizza cutter Rigidity Metal
Silicon Spatula Elasticity Rubber
Steel Spatula Rigidity Metal
Peeling Knife Rigidity Metal
Plate Brittleness Hard clay
Big bowl Brittleness Hard clay
Cup Brittleness, Hard clay
Wine glass Brittleness Glass
Short glass Brittleness Glass
Blue rubber bowl Elasticity Rubber
Plastic container Plasticity Plastic
Plastic container lid Plasticity Plastic
Butter milk bottle Plasticity Plastic
Salt bottle Rigidity Hard Plastic
Milk carton Elastic-Plastic [13] Paperboard
Table 4
List of objects and their material dispositions along with material type
As a future task, we are looking into decomposing object meshes into several functional
parts and learn potential dispositions of an object. This process will help us to classify objects
with various structural dispositions. In order to do that, we need to model each Unreal object
with a mesh which can be decomposed into various functional parts such as that a wine glass
can be decomposed into bowl, stem, and base. Then, we plan to create a model which can
decompose meshes into several primitive shapes. At last, we plan to update the ontology model
to incorporate such primitive shapes with corresponding dispositions such as roll-ability, slid-
ability, containment, etc. Furthermore, we will draw inspiration from the approach suggested
by Tenorth et al. [12] in order to relate primitive types with the functional parts of the object
mesh. We will also evaluate our results against some benchmarks such as PartNet [14].
The level of depth at which such object descriptions are useful for robotics tasks are highly
influenced by competence and performance measures. We are interested in creating a set of
competency questions and will explore at which level of detail an object description should be
to answer these questions.
6. Acknowledgements
This work was funded by the German Research Foundation (DFG) as part of Collaborative
Research Center (CRC) 1320 EASE – Everyday Activity Science and Engineering, University of
Bremen (http://www.ease-crc.org/)
References
[1] J. J. Gibson, The Ecological Approach to Visual Perception, Psychology Press Classic
Editions, 1979.
[2] A. Stoytchev, Toward Learning the Binding Affordances of Objects: A Behavior-Grounded
Approach, 2005.
[3] D. Beßler, R. Porzel, M. Pomarlan, M. Beetz, R. Malaka, J. A. Bateman, A formal model of
affordances for flexible robotic task execution, in: G. D. Giacomo, A. Catalá, B. Dilkina,
M. Milano, S. Barro, A. Bugarín, J. Lang (Eds.), ECAI 2020 - 24th European Conference
on Artificial Intelligence, volume 325 of Frontiers in Artificial Intelligence and Applications,
IOS Press, 2020, pp. 2425–2432. URL: https://doi.org/10.3233/FAIA200374. doi:10.3233/
FAIA200374.
[4] D. Beßler, R. Porzel, M. Pomarlan, A. Vyas, S. Hoefner, M. Beetz, R. Malaka, J. Bateman,
Foundations of the Socio-physical Modelof Activities (SOMA) for Autonomous Robotic
Agents, in: 12th International Conference on Formal Ontology in Information Systems,
2021.
[5] M. T. Turvey, Affordances and prospective control: An outline of the ontology, Ecological
psychology, vol. 4, no. 3, pp. 173–187 (1992).
[6] F. Schmidt, V. C. Paulun, J. J. R. van Assen, R. Fleming, Inferring the stiffness of unfamiliar
objects from optical, shape, and motion cues, Journal of Vision 17(3):18, 1–17 (2017).
[7] A. D. Mihai Andries, J. Santos-Victor, Automatic generation of object shapes with desired
affordances using voxelgrid representation, Front. Neurorobot. 14:22. doi: 10.3389/fn-
bot.2020.00022 (2020).
[8] N. S. Srikanth, L. Nair, S. Chernova, Tool substitution with shape and material reasoning
using dual neural networks, CoRR abs/1911.04521 (2019). URL: http://arxiv.org/abs/1911.
04521. arXiv:1911.04521.
[9] K. S. L. Paola Ardón, S. Ramamoorthy, Object Affordances by Inferring on the Surroundings,
in: IEEE Workshop on Advanced Robotics and its Social Impacts (ARSO), 2018.
[10] B. Moldovan, P. Moreno, M. van Otterlo, J. Santos-Victor, L. De Raedt, Learning relational
affordance models for robots in multi-object manipulation tasks, in: 2012 IEEE International
Conference on Robotics and Automation, 2012, pp. 4373–4378. doi:10.1109/ICRA.2012.
6225042.
[11] L. Montesano, M. Lopes, A. Bernardino, J. Santos-Victor, Modeling affordances using
bayesian networks, in: 2007 IEEE/RSJ International Conference on Intelligent Robots and
Systems, 2007, pp. 4102–4107. doi:10.1109/IROS.2007.4399511.
[12] F. B.-B. Moritz Tenorth, Stefan Profanter, M. Beetz, Decomposing cad models of objects of
daily use and reasoning about their functional parts, 2013.
[13] Y. Li, S. E. Stapleton, S. Reese, J.-W. Simon, Anisotropic elastic-plastic deformation of paper:
Out-of-plane model, International Journal of Solids and Structures 130-131 (2018) 172–182.
URL: https://www.sciencedirect.com/science/article/pii/S0020768317304560. doi:https:
//doi.org/10.1016/j.ijsolstr.2017.10.003.
[14] K. Mo, S. Zhu, A. X. Chang, L. Yi, S. Tripathi, L. J. Guibas, H. Su, Partnet: A large-scale
benchmark for fine-grained and hierarchical part-level 3d object understanding, 2018.
arXiv:1812.02713.