When a robot is operating in a dynamic environment, it cannot be assumed that a tool required to solve a given task will always be available. In case of a missing tool, an ideal response would be to nd a substitute to complete the task. In this paper, we present a proof of concept of a grounded knowledgebased approach to tool substitution. In order to validate the suitability of a substitute, we conducted experiments involving 22 substitution scenarios. The substitutes computed by the proposed approach were validated on the basis of the experts' choices for each scenario. Our evaluation showed, in 20 out of 22 scenarios (91%), the approach identi ed the same substitutes as experts.
The sophistication pertaining to tool-use in humans involves not just the dexterity in manipulating a tool, but also the diversity in tool exploitation. The ability to exploit the tools has enabled humans to adapt and thus exert control over an uncertain environment, Copyright c by the paper's authors. Copying permitted for private and academic purposes. especially when they are faced with unfavorable situations. For example, if we don't nd a hammer to hammer a nail into a wall, we will use a heel of a shoe or a rock or if a tray is unavailable for serving the drinks, we will use a plate for serving. In situations like these, humans seem to know - either from the past experience or from observations or from the \necessity is the mother of improvisation (invention)" type approach what kind of object is needed as a substitute.
On the contrary, consider a robot performing a task that involves tool use. When a robot is operating in a dynamic environment, it can not be assumed that a tool required in the task will always be available. In situations like these, an e ective way for a robot would be to nd an alternative as humans do, for example, use an eating plate for serving, rather than wait until a tray becomes available. This skill is signi cant when operating in a dynamic, uncertain environment because it allows a robot to adapt to unforeseen situations to a degree. The question is how can a robot determine which object in the environment is a viable candidate for a substitute? A possible approach would be by interacting with an object in a manner missing tool is maneuvered. However, it would be time consuming if a robot interacts with every single object in the environment to determine a viability which makes this approach less practical.
In this prototypical work, we propose a non-invasive approach that identi es viable candidate/s from the existing objects in the environment. This paper makes the following contributions: 1) An approach to create grounded knowledge about objects expressed in terms of their properties (Sec. 5.2), 2) an approach to identify relevant properties of a missing tool and determine a substitute on the basis of them (Sec. 5.3). 2
Typically, a substitute for a missing tool is determined by means of knowledge base that provides knowledge about objects and similarity measures to determine the similarity between a missing tool and a potential substitute. In the following, in addition to the approaches to determine a substitute, we also report the literature related to existing knowledge bases developed for robotic applications.
We reviewed in [
We noted that the majority of the knowledge bases relied on the external human-centric commonsense knowledge bases such as WordNet, Cyc, OpenCyc, and some either relied on the hand-coded knowledge or on the knowledge acquired by human-robot interaction. The main issue, we believe is that, the depth and breadth of the human-centric knowledge base is not observable by a robot in its entirety due to its limited sensing capabilities. This causes a disconnect between human-centric knowledge and robot-centric perception. To de ect this issue, we aim to acquire the robot-centric perceptual data for di erent properties of objects. Such property data can then be used to generate grounded knowledge about objects (see Sec. 5.1).
One of the closest areas that study the usability of an object is a ordancs of tools where the primary focus is to examine various functional abilities of an object by exploring what actions can be performed on the object and observing its responses. As such, using a substitute in place of a missing tool can also be seen as transferring of an a ordance of the missing tool to the substitute after determining similarity between them.
In [
How to characterize similarity between a missing tool and a potential substitute: A candidate for a substitute is expected to be similar to a missing tool to some degree to ensure a substitutability. The notion of similarity can be understood in various forms, for instance, a distance between two objects denoted by two points in a multi-dimensional space or two objects belonging to the same cluster or aspects of the objects that are identi ed as shared. In this work, the question will be addressed in a broader sense: it is not merely about identifying a similar object by deploying some similarity measure, instead, it is about gaining an access to what aspects of the objects were found to be shared between the similar objects.
What kind of knowledge is required to determine the
similarity: It has been demonstrated in the literature
on tool use in humans and animals alike that in order
to use an object in tasks one needs to have knowledge
about objects [
How to acquire the necessary knowledge: The acquisition of such conceptual knowledge is not without challenge. From a robot stand-point, it is a tradeo between what needs to be known and what can be known. The trade-o is a direct consequence of the limited perception capabilities of a robot which often leads to partial understanding of the environment. While deploying a multi-modal perception to extract the required knowledge about objects would be an ideal solution, however, it carries its own set of complexities such as noisy sensors, dynamicity of the environment, complexities of the composition of an object. For this prototypical work, the necessary knowledge is acquired using human-centric as well as machine-centric methods.
How to maneuver a substitute as a missing tool: Once the substitute has been identi ed, a robot is expected to use it in place of a missing tool and achieve the same result as the missing tool in the task. The challenge to estimate the maneuver as well as grasping of a substitute is two fold: to determine whether the maneuver and grasping knowledge of a missing tool can be transferred and utilized on a substitute, else, estimate the maneuver and grasping for a substitute such that it can be used as a missing tool in the task.
For this work, we have focused on the rst three challenges and have developed a prototypical system called ERSATZ (German word for a substitute or alternative) where the focus is to identify the required knowledge to determine a substitute and develop a system that computes a substitute for a missing tool. 4
The proposed approach distinguishes a tool from a substitute where a tool is de ned as an artifact that is designed, manufactured and maneuvered in accordance with its designated purpose in the tasks such as hammer for hammering, tray for serving etc., while a substitute is seen as an extension of a missing tool. Within the context of a designated purpose, the relationship between a tool and a substitute is symmetric, for instance, for hammering, a hammer can be replaced by a heeled shoe and vice versa. However, it may not always be the case once you step outside the context, for instance, a hammer can not replace a heeled shoe. Our research work, therefore, focuses on searching for a substitute for a conventional tool required in the ongoing task as opposed to determining a substitute for itself.
Consider a scenario in which a robot has to choose between a plate and a mouse pad as an alternative for a tray. A tray can be de ned as a rigid, rectangular, at, wooden, brown colored object while a plate can be de ned as a rigid, circular, semi- at, white colored object and a mouse pad as soft, rectangular, at, leather-based object. Bear in mind, however, that some properties are more relevant than others with respect to the designated purpose of the tool. For a tray whose designated purpose is to carry, rigid and at are more relevant to carry than a material or a color of a tray. Consequently, to nd the most appropriate substitute, the relevant properties of the unavailable tool need to correspond to as large a degree as possible to the properties of the possible choices for a substitute.
The proposed approach performs conceptual knowledge-driven computation to identify the relevant properties of the missing tool and determines the most similar substitute on the basis of those properties. Besides identifying the most similar object as a substitute for a missing tool, the proposed approach grants an explicit access to the relevant properties of the missing tool which carries twofold advantages: rstly, knowing which properties are primarily required in the potential substitute narrows down the search space and secondly, in case of an unknown object instance, only the relevant properties will have to be learned to determine a substitute.
The conceptual knowledge considered in this work primarily involves properties of the objects. The properties considered are divided into physical and functional properties where physical properties describe the physicality of the objects such as rigidity, weight, hollowness while the functional properties ascribe the (functional) abilities or a ordances to the objects such as containment, blockage, support. The functional properties in the proposed approach play a primary role in identifying the relevant properties of a missing tool (see Sec. 5.3).
The functional properties considered in this work
are derived from the theory of image schemas [
Our ultimate objective is to acquire machine centric data from which property speci c data can be extracted. Such property data will then be used to generate grounded knowledge about objects. As a rst step, our initial property acquisition focuses on the composite of a machine-centric and a human-centric method. In the machine-centric approach, geometrical properties are acquired using a non-invasive vision-based technique while non-geometric properties are acquired by sampling from the data from the expert generated intuitive model for the properties.
Machine Generated Properties: In this paper,
we introduce a state-of-art data-driven approach that
unsupervisedly conceptualizes shape according to
commonalities within object point clouds which is
discussed in detail in our work [
In our previous work on shape concept learning [
An object shape representation is introduced that gradually encodes observed objects symbol compositions (see B in Fig. 1): from local components to component groups that may represent object parts or objects as a whole. The proposed shape representation incorporates aspects of exemplar, respectively, prototype theory since we believe that the richness of a prototype provides an unaltered perspective on the characteristics of object instances. Based on the proposed symbolic shape representation we analyze topology and structure within the encoded symbol compositions in order to discover persistent patterns that may represent shape concepts.
We introduce an iterative ltering process [
Human Generated Properties The geometric properties alone o er a very limited scope of the physicality as well as the functionality of an object. Therefore, to compensate the gap, we also considered nongeometrical properties such as weight, rigid, hollowness as physical and support, blockage, containment as functional. Note that, in general, these properties are challenging and cumbersome to extract solely from non-invasive visuoperceptual approaches. Consequently, extracting such properties via multi-modal or manipulation capabilities is needed, but this is beyond the scope of this paper. In the generation process, a set of labeled prototype objects selected from the Washington dataset (see Table 1) were taken into account. The distribution of each property for particular object labels (cf. Table 1) was approximated by an expert to resemble the scope for the variations in the values of the property in general. Consequently, given an object and its label, a sample value was drawn from the a-priori generated property distribution. 5.2
Knowledge about objects is spread across three levels: the rst level consists of the data about the machinegenerated as well as human-generated properties, the second level consists of qualitative knowledge about individual object instances, while the third level consists of the aggregated qualitative fuzzy knowledge about respective classes of object instances. The fuzzy formalism is used to model the intra-class variations in the objects. In the following, we discuss the formal description of the methodology deployed to create grounded knowledge about objects.
Consider O as a given set of object class labels where (by abuse of notation) each object class is identi ed with its label. Let each object class O 2 O be a given set of its instances. Let S O be a union of all object classes such that j S Oj = n. Let P and F be the given sets of physical properties' labels and a set of functional properties' labels respectively. By abuse of notation, each physical and functional property is identi ed with its label. For each physical property P 2 P as well as for a functional property F 2 F, sensory data is acquired from each object instance o 2 S O. Let Pn and Fn represent sets of n number of extracted sensory values from n number of object instances for a physical property P 2 P and a functional property F 2 F respectively.
The sub-categorization process is performed to form (more intuitive) qualitative measures to represent the degree with which a property is re ected by an object instance. It is the rst step in creating symbolic knowledge about object classes where the symbols representing the qualitative measures of a physical or a functional property re ected in an object instance are generated unsupervisedly by a clustering mechanism. A qualitative measure of a physical property is referred to as a physical quality and that of a functional property as a functional quality.
In this process, Pn and Fn representing measurements of a physical property P 2 P and a functional property F 2 F respectively extracted from n number of object instances is categorized into a given number of discrete clusters using a clustering algorithm. Let rP and rF be partitions of the sets Pn and Fn after performing clustering on them. Let P and F be the sets of labels, expressing physical qualities and functional qualities, generated for a physical property P 2 P and a functional property F 2 F respectively. Given the label for a property, the quality labels are generated by combining a property label P and a cluster label (created by the clustering algorithm). For instance, the quality labels for a property size are represented as fsize 1; size 2; size 3; size 4g. At the end of the sub-categorization process, the clusters are mapped to the generated symbolic labels for qualitative measures.
Note that the number of clusters essentially describes the granularity with which each property can qualitatively be represented. The higher number of clusters suggest that an object is described in a ner detail which may obstruct the selection of a substitute since it may not be possible to nd a substitute which is similar to a missing tool down to the ner details. For example, in size = fsmall; medium; big; biggerg, size is a physical property and small, medium, big, bigger are its physical qualities. The semantic terms given above are meant for the readers to understand the qualitative measures of the properties.
The attribution process generates knowledge about each object instance by aggregating all the physical and functional qualities assigned to the object instance by the sub-categorization step. In other terms, the knowledge about an instance consists of the physical as well as functional qualities re ected in the instance. Let P and F be the families of sets containing the physical quality labels P and the functional quality labels F for each physical property P 2 P and functional property F 2 F respectively. Thus, each object instance o 2 S O is represented as a set of all the physical as well as functional qualities attributed to it which are expressed by a symbol holds as: holds S O (P [ F ) For example, knowledge about the instance plate1 of a plate class can be given as, holds(plate1; medium), holds(plate1; harder), holds(plate1; can support) where medium is a physical quality of size property, harder is a physical quality of rigidity property and can support is a functional quality of support property.
The conceptualization process aggregates the knowledge about all the instances of an object class. The aggregated knowledge is regarded as conceptual knowledge about an object class.
Let OKB be a knowledge base about object classes where each object class O 2 O. Given the knowledge about all the instances of an object class O, in the conceptualization process, the knowledge about the object class OK 2 OKB is expressed as a set of tuples consisting of a physical or a functional quality and its proportion (membership) value in the object class. A tuple is expressed as hO; t; mi where t 2 P [ F and a proportion value m is calculated using the following membership function: m = P (holds(o; t)jo 2 O). The proportion value allows to model the intra-class variations in the objects.
For example, knowledge about object class table can be expressed as: fhplate, harder, 0.6 i, hplate, light weight, 0.75 i, hplate, less hollow, 0.67 i, hplate, hollow, 0.33 i, hplate, more support, 0.71 ig, where the numbers indicate that, for instance, physical quality harder was observed in 60% instances of object class plate. At the end of the conceptualization process, conceptual knowledge about an object class is created which is represented in a symbolic fuzzy form and grounded into the human-generated or machinegenerated data about the properties of objects. The knowledge about objects is then used to determine a substitute from the existing objects in the environment.
In addition to conceptual knowledge about objects, Conceptualization process also creates knowledge about functional quality, termed as a function model, by associating the occurrence of physical qualities in an object instance with the occurrence of a functional quality in the instance and aggregating the result of such concurrent occurrences. The role of a functional model is discussed later in the section 5.3. Given the knowledge about the object instances, a function model fd of a functional quality f 2 F is expressed as a set of tuples containing a functional quality f 2 F , a physical quality p 2 P and a proportion value d. A tuple is represented as hf; p; di where f 2 F ; p 2 P and a proportion value d is computed as, d = P (holds(o; p)jholds(o; f )) For example, a function model for a functional quality more support is given as, f hmore support, harder,0.8 i, hmore support, softer, 0.2 i where the number indicates that, for instance, functional quality more support and a physical quality harder co-occurred in the knowledge about the object instances 80% of the time. 5.3
Fig. 2 illustrates a process ow consisting of the primary operations involved in determining a substitute. The ow o ers an approximated aerial view for the prototypical model of ERSATZ. When ERSATZ is queried to nd a substitute for a missing tool x from the set of available objects Y the system checks if the substitution model for x exists in the knowledge base. If the substitution model does not exist, then the reasoner computes the relevant functional and physical properties of the queried tool.
A representative physical model and a representative functional model of an object consists of the physical or functional qualities, respectively, that are regarded as representative qualities of the object class, while the qualities which do not fall under representative qualities are regarded as exceptional or uncommon qualities.
Let O 2 O be an object class of a missing tool and let is a representative model threshold which quali es a physical or a functional quality as stereotypical or representative to the object class O. Orp is called as a representative physical model of an object class O such that Orp = fp : implies(O; p) ; p 2 P g and Orf is called as a representative functional model of an object O such that Orf = ff : implies(O; f ) ; f 2 F g. Similarly, let fd be a function model of functional quality f , then frp is called as a representative physical model of a functional quality f such that frp = fp : implies(f; p) ; p 2 P g
Due to the abstract nature of an image schema and
by extension a corresponding functional property, it
can subsume various purposes of objects, for example,
a functional property support which can subsume the
purposes place on, sit on and serve on of the a table,
a chair and a tray respectively. It is suggested in [
The relevance of a representative functional quality is decided by examining whether the physical characterization of the function model of the representative functional quality of a tool are in a close proximity to the physical characterization of a representative physical model of the tool. The close proximity between a functional quality and the object class of the tool is determined using Jaccard Index. Jaccard Index determines a similarity and dissimilarity between the two sets A and B where the similarity is calculated by dividing the magnitude of the intersection of A and B by the magnitude of the union of A and B.
Let Orp and frp be the representative physical models of an object class O of the missing tool and of a function model fd of a representative functional quality f 2 F of the object class O respectively. Let be a Minimum Similarity Tolerance threshold for similarity. Then, Jaccard Index of Orp and frp is computed as: J (Orp; frp) = jOrp\frpj . A representative functional jOrp[frpj quality f of an object class O is regarded as relevant if J (Orp; frp) > . Let OF 0 be a set of all relevant functional qualities of an object class O. Let frp be a representative physical model of a function model fd of a relevant functional quality f 2 OF 0 . Let Orp be a representative physical model of O. Then, the relevant physical qualities of an object class O, expressed by a set OP 0 = (Orp \ frp).
Let O 2 O be an object class of a missing tool and let O 2 O be an object class of a possible candidate for a substitute. Let OP 0 be a set of relevant physical qualities of O and let Orp be a representative physical model of O . Let be a Minimum Similarity Tolerance threshold for similarity. The substitutability of a candidate is determined by measuring the similarity between OP 0 and Orp using Jaccard's Index. O is termed as a substitute, expressed as O +, if J (OP 0 ; Orp) > , else it is regarded as not a substitute and expressed as O . Given the set of relevant physical qualities OP 0 , the set of relevant functional qualities OF 0 and a positive substitute O +, and a negative substitute O , a substitution model of O is expressed as a tuple: hOP 0 ; OF 0 ; O +; O i. The knowledge about object O 2 O is then extended in OKB to accommodate its substitution model. 6
The objective of the experimental evaluation of
ERSATZ is to validate the suitability of the substitutes
computed by ERSATZ by comparing the results with
that of human experts. For the experimental
evaluation, we used the images from the Washington
Dataset [
The results of both experiments were plotted as a heat map where the y-axis shows missing tools and xaxis shows the available objects illustrated in Fig. 3. The grayed cells mean the corresponding object categories were not available in the respective query. The cells that are marked with represents substitutes selected by experts and ERSATZ. Out of 22 scenarios, . t s n I .
t sn Isn
ERSATZ and the experts identi ed the same substitutes in 20 scenarios (91%). 7
The paper presents a prototypical system to determine a substitute for a missing tool using the grounded knowledge about objects. The approach has drawn inspiration from symbol grounding, the theory of a ordances and the theory of image schemas to represent the grounded knowledge and to determine a substitute. This is an ongoing research with a focus on the Fuzzy Conceptual Knowledge - Object Classes Bivariate Joint Frequency Distributions
Fuzzy Conceptual Knowledge - Object Instances Clustering Method
Functional Property Data - Object Instances Aggregation
Physical Property Data - Object Instances Property Extraction Methods
Sensory Data - Object Instances following aspects.
Our immediate goal focuses on the fuzzi cation of
the clustering method and the reasoning method to
combat the migration of the data points within
clusters. Moreover, we have derived three functional
properties, namely, contain, support, block from the image
schemas Containment, Support and Blockage
respectively. However, further investigation is needed to
formalize the identi cation of additional functional
properties to be derived from the existing image schema.
For robot-centric property acquisition, we are
currently developing a framework that allows a robot to
extract properties of individual objects and build a
knowledge base in a bottom-up manner such that the
knowledge about properties of objects is constructed
on the basis of what is sensed (see Fig. 4). We have
proposed the preliminary framework in [