Dispositional qualities are the characteristics of an object that are attributed to it's properties, such as mass, color, shape, material, etc. Understanding how the design of an object afords such qualities is a crucial task in robotics. Such as a cup being functionally designed to hold or contain something, and structurally designed to be carried or grasped by its handle. Dispositions tend to be more independent of an environment than afordances, since they are related to fundamental characteristics of an object. Whereas, afordances define the action possibilities with the object in the given environment with an agent capable of manipulating them, such as a bottle of water afords drinking possibility to an adult but it is hard for an infant to open the bottle cap in order to drink from it. The topic of afordances is widely explored in the domain of robotics where it plays a vital role for basic object manipulation skills. In this paper, we present an approach for disposition learning about an object from it's shape and material information provided by a physics engine. We postulate our hypothesis around the current state of the art game engines which have complex object rendering and modelling techniques. The modelling of shape and material information about the object can be harvested as a source of knowledge for the given object in the environment. An intelligent agent thus benefits from having prior information about such objects in the world.
Performing everyday activities such as cleaning, setting up the table and cooking comes naturally to us as humans even in an unknown environment. However, this is something we humans have learned through multiple experiments over our lifetime. In addition, we are able to use our common sense knowledge to derive certain facts that can be helpful for performing tasks eficiently. Such knowledge is not easily available or transferable to artificial agents. It is widely accepted that the agent’s capability to manipulate objects in unknown environments is an indicator for intelligent behaviour. Although, doing such tasks under realistic conditions is a big challenge for artificial agents.
In computer vision, object recognition techniques are mainly focused on visual appearance
and in some extent shape detection of objects. However the classification of objects can be done
in many diferent ways. Gibson argued that the main objective of such vision techniques should
be to understand the possible interactions between an agent and an environment in a scene
rather than merely identifying contents of the scene [
Gibson’s afordance theory forms the relationship between an action, an object, and the
efects of the action on the object. An agent can use exploratory behaviors when facing a new
object while interacting with the world. It can be stated that the properties of an object that
an agent is likely to learn are directly linked to the behavioral procedures an agent follows
[
However, there are many other views on the interpretation of the term afordances and
ontological categorisation of it. Some researchers consider afordances as an event and others
say it is a quality of an object. In this paper, we commit to the Descriptive Afordance Theory [
It is evident that the basic dispositional qualities of an object are tightly linked to its shape and material attributes. In this work, we try to shade some light on how dispositional qualities can be grounded in such attributes, and how this knowledge can be incorporated with an existing ontological model. Our approach is to exploit the detailed shape and material models used in state of the art game engines. Modern game engines, such as the Unreal Engine and Unity, have enabled users to create virtual environments with realistic modelling of 3D objects. In this work, we study the 3D object modeling in the Unreal Engine, and explore ways to translate them into dispositional qualities.
The main contributions of this paper are the following ones: (1) A classification of diferent types of object dispositions; and (2) a methodological approach for object classification based on material and shape information from the Unreal Engine.
In the following, we will first provide an overview of diferent related works concerning learning of afordances. Thereafter, we will investigate object modelling in the Unreal Engine which can be utilized for inferring dispositions. After that, we propose our approach for using information from Unreal and incorporate this knowledge in an ontological model. At last, we will discuss preliminary results and the future work.
Afordance modelling has been investigated for many years. There has been many eforts for an afordance detection of an object while observing it’s physical characteristics such as color, shape, material and so on.
Schmidt et al. [
From a design point of view, Andries et al. [
Tool substitution is the problem which addresses the case where an intelligent agent is
supposed to perform the task with a specific tool, but in the absence of such tool, it has to look
for an alternative tool within the environment. E.g. if the robot can not find the lid of a vessel
to cover, then it should use the plate instead. Shrivatsav et al. [
Ardón et al. [
Moldovan et al. [
Montesano et al. [11] have presented the general afordance model based on Bayesian networks, which links actions, object features and action efects. They use object’s shape, size, color, velocity, and other parameters such as an agent’s action, hand velocity, and distance from object to train their afordance network. The network learns through interaction with the environment by an agent which is similar to an exploratory behaviour approach we have discussed in the previous section.
Tenorth et al. [12] have developed a system to identify functional parts of an object from CAD models as a source of knowledge while performing logical inference with Prolog rules. The system was able to identify geometric primitive shapes such as planes, cylinders and spheres. By recognizing such components, they were able to answer afordance related questions such as which object should I use to pour one liter of liquid? Or which part of the bottle should one grasp in order to open it?
In this section, we present an overview of the modelling approaches that we think are useful to infer dispositions of objects based on their shape, material and physical attributes provided by the Unreal Engine. One of the advantages of using a physics engine is that the modeling of objects can be done realistically using physical properties such as mass, center of mass, density, friction, restitution, and so on.
When an agent interacts with various shapes presented in the environment, it can gather information about the physical nature of such shapes. For example, spherical objects are easy to roll, and boxes are easy to slide. During such interactions, an agent can also learn about the ideal posture of the object such as that a WineGlass should be held upwards as shown in Figure 1. If it is placed sideways, then it can slide or roll down the table. Through such experimentation in a game environment, an agent can learn about how various shapes impose constraints on the pose of an object, and its stability.
The Unreal Engine provides modelling of basic primitive types such as spheres, boxes, capsules, convex elements, and tapered capsules 1. Such shapes can be useful in deriving dispositions such as roll-ability, slid-ability and containment.
Shape and size also influences material attributes of an object such as rigidity or brittleness. One can argue, from a material point of view, that objects made from glass tend to be more brittle, but it is the shape and size which contributes the degree of brittleness. An example is the high storey building windows made from glass which are as strong as concrete. Although in a everyday kitchen scenario, we do not encounter such complex cases, hence, it is out of scope for this work.
Humans have adapted themselves to identify visual cues from material, and can associate those cues with diferent material characteristics. We can identify if an object is rigid, brittle, rough or slippery based on our observations of the material surface. However, material information is something which is not readily available to an agent in the real world. On the other hand, with virtual reality environments it is needed to model such information in order to render as well as process interactions among various players and objects.
Within the Unreal Engine, we can define various surface types 2 such as metallic, wood, plastic, paper, rubber, skin, etc. As can be seen from Table 1, one can also set the friction, restitution and density related parameters of the physical material in order to simulate those surface behaviours. We use this information to assign dispositions such as rigidness, plasticity, elasticity, and brittleness to the given virtual object.
things can slide on this surface.
Determines how "bouncy" the surface is, or how much energy it retains when it collides with another surface.
Used with the shape of the object to calculate its mass properties. The higher the number, the heavier the object. Measured as g per cubic cm.
Defined as an enum value to be used in the engine for defining any number of things, from what sound plays as a character walks across a surface, to the type of decal an explosion should leave on diferent surfaces. 1https://docs.unrealengine.com/4.26/en-US/InteractiveExperiences/Physics/PhysicsBodies/Reference/ 2https://docs.unrealengine.com/4.26/en-US/InteractiveExperiences/Physics/PhysicalMaterials/ PhysMatUserGuide/ Mass
Center of Mass ofset
User defined ofset of center of mass for given object.
It is rather dificult to estimate physical attributes of an object such as mass, and center of mass via visual inspection. One can gain insights only by interacting with the object. For each object in the Unreal virtual world, we can define physical attributes specific to the object. We only consider a few selected physical parameters, as shown in Table 2, such as mass and center of mass. This information along with shape information can be vital for finding balanced poses of the object under the influence of gravity.
This section describes the approach we consider to relate dispositional qualities of an object to material and shape information obtained from the Unreal Engine. We employ the information obtained from the physics engine object modelling and link it with the object representation in the ontology model. The SOMA ontology model defines a Disposition concept as an extrinsic physical quality of an object. Extrinsic means here that it depends on relationships to other things, for example, the color of an object depends on light conditions in the environment. We define such relationships by adding new axioms to our model. An example is to check if an object, or a part of it has been made from metal, wood, plastic, rubber, glass or paper. One can add relationships between objects and material dispositions such as that an object or a part of it is rigid, elastic, plastic or brittle.
Material Dispositions roll-ability slid-ability containment liquid-containment rigidity elasticity plasticity brittleness ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
We classify dispositions into two main groups: structural dispositions, and material dispositions. Structural dispositions such as roll, slide, support and containment are dependant upon Class : L i q u i d C o n t a i n m e n t SubClassOf :
Containment , S t r u c t u r a l D i s p o s i t i o n a f f o r d s B e a r e r only C o n t a i n e r a f f o r d s T r i g g e r only L i q u i d Class : G l a s s y SubClassOf :
M a t e r i a l R e g i o n h a s D i s p o s i t i o n some B r i t t l e n e s s Class : Base SubClassOf :
DesignedComponent h a s D i s p o s i t i o n some S u p p o r t
Class : WineGlass SubClassOf :
G l a s s , D e s i g n e d C o n t a i n e r h a s D i s p o s i t i o n some
L i q u i d C o n t a i n m e n t h a s M a t e r i a l R e g i o n some G l a s s s y hasComponent some Bowl and hasComponent some Base and hasComponent some Stem Class : Bowl SubClassOf :
DesignedComponent h a s D i s p o s i t i o n some Containment Class : Stem SubClassOf :
DesignedComponent h a s D i s p o s i t i o n some G r a s p i n g h a s D i s p o s i t i o n some C o n n e c t i n g shape of the object and other physics attributes. The second category of dispositions are material dispositions which are inherently derived from material characteristics, such as rigidity, elasticity, plasticity and brittleness. Table 3 lists all the dispositions that we consider for our experiment.
Figure 2 depicts an ontological representation of object shape and material characteristics of wine glass. We define the Wine Glass concept as a subclass of Glass which is broadly defined as a sub-concept of Designed Container. Thus, each Wine Glass instance inherits a containment disposition from being a container, and a more specific liquid containment disposition. Additionally, Figure 1 depicts that a Wine Glass can have other functional parts such as a base and a stem, which can provide support and grasp dispositions respectively. One should also note that, a wine glass is made of Glassy material which by nature possess brittleness or fragility as a material disposition.
We have established an experimental setup with the Unreal Engine resembling a domestic kitchen environment, as shown in Figure 3. Kitchens are challenging environments, partly due to the amount of diferent object types that can be encountered, and various degrees of usefulness to apply them in diferent situational contexts. We expect this setup to be a valuable source for dispositional learning in our future work.
For our test domain, we have modelled diferent objects with material information, and we have defined various material surface types such as wood, plastic, paper, glass, clay, metal, rubber and created physical materials for each of those surfaces. An example of such physical material parameters is shown in Figure 4 for the material type rubber.
So far, we were able to achieve the following objectives: • modeling of objects with material information in the Unreal Engine; and • ontological definition of several material types, and their relation to object dispositions.
We were also able to create an interface for querying the game engine data from external components which is important for connecting our approach to the goals of a robotic agent.
We have presented an approach to infer dispositional qualities from physics engine object modelling. Table 4 presents the objects shown in Figure 3 with their dispositions and material information gathered from the Unreal Engine. We have used material surface types to categorize objects based on material, and have inferred corresponding material dispositions.
As a future task, we are looking into decomposing object meshes into several functional parts and learn potential dispositions of an object. This process will help us to classify objects with various structural dispositions. In order to do that, we need to model each Unreal object with a mesh which can be decomposed into various functional parts such as that a wine glass can be decomposed into bowl, stem, and base. Then, we plan to create a model which can decompose meshes into several primitive shapes. At last, we plan to update the ontology model to incorporate such primitive shapes with corresponding dispositions such as roll-ability, slidability, containment, etc. Furthermore, we will draw inspiration from the approach suggested by Tenorth et al. [12] in order to relate primitive types with the functional parts of the object mesh. We will also evaluate our results against some benchmarks such as PartNet [14].
The level of depth at which such object descriptions are useful for robotics tasks are highly influenced by competence and performance measures. We are interested in creating a set of competency questions and will explore at which level of detail an object description should be to answer these questions.
This work was funded by the German Research Foundation (DFG) as part of Collaborative Research Center (CRC) 1320 EASE – Everyday Activity Science and Engineering, University of Bremen (http://www.ease-crc.org/)