Household robots need semantics to understand that a detergent is a cleaning product that can be used to clean physical objects like a table, but laundry detergent is only used to clean/wash laundry. A safely acting autonomous robot should also know that both will not be used as ingredients for meal preparation. We propose a new approach to connect robot sensor data to Linked Data in order to give robotic agents semantic product information about objects that can be found in their environment so that the action to be performed with a given object can be inferred. For this, we use the robot's belief state when recognizing a product and link it to a product ontology that follows Semantic Web standards. We then use the product class information to fetch further information from external sources like Wikidata or ConceptNet that contain action information (e.g. laundry detergent is used for laundering). At last, the action results are mapped to internally known actions of the robotic agent so that it knows which action can be performed with the perceived object.
Representing common sense knowledge in cognitive robots is a widely tackled research
challenge as it could help in tasks such as action planning. Action planning tasks often
involve autonomous robotic agents operating in human environments to help perform
mundane tasks like vacuum-cleaning or washing the dishes. In these environments, the
need for robotic agents to operate competently and safely becomes even more
important. Unfortunately, household robots today are far from operating autonomously when it
comes to more complex tasks [
The necessity to integrate semantics into robot knowledge bases has been addressed
in relevant cognitive robotics literature [
This work proposes an approach to include implicit knowledge such as product
information of objects a robotic agent can find in a household environment by using Linked
Data and information from the Semantic Web . The Semantic Web [
Entities and relationships in Semantic Web format are represented in the form of
a knowledge graph [
In this work, we connect robot sensor data to a product ontology, which again links to data sources from the Web. Figure 1 visualizes the links created in an example use-case. A household robot operating in the kitchen perceives an object, which it reasons about in its belief state. The object has a related pose and a barcode scanned by the robot. The barcode also contains the European Article Number (EAN), which can be linked to further information about the object in the linked product ontology. Through this link the agent already knows that the object is a laundry detergent. By linking this product class to existing knowledge graphs like Wikidata or ConceptNet our agent can find the information that laundry detergent can be used for laundering ((<laundry detergent> <use> <laundering>)4 or (<Things that require> <laundry detergent>: <washing your clothes>)5). These Linked Data sources offer links to even more sources (e.g. WordNet or DBpedia).
The example in Figure 1 reflects the goal of building a knowledge graph in this
project. We use a knowledge graph to acquire and integrate information from the Web
and represent this into an ontology so that a reasoner can be used to derive new
knowledge [
Each robotic agent uses different sensors to make sense of its environment. The effort
for implementing common sense into a cognitive robot is a highly considered topic in
the field of cognitive robots. Different sources have been able to provide a variety of
information to robots, including WordNet, OpenCyc, Freebase, ConceptNet, and OMICS
[
The KnowRob system, first introduced in 2009 and later extended to a newer version
in 2018, is at the forefront of cognitive robot work in the household domain in terms of
the extent of information its knowledge base represents [
For this work, we will focus on the robot belief state retrieved by the KnowRob system and link it to knowledge graphs. To simplify the situation, we will only focus on products that can be identified via barcode. A barcode can easily be recognized even by simple sensors and still uniquely identifies a product. This makes us focus on ontologies instead of object detection.
Since KnowRob already is an ontology-based system, a robotic household agent will store its belief state including object position and its barcode in an Web Ontology 4https://www.wikidata.org/wiki/Q910284 5http://conceptnet.io/c/en/laundry detergent Language (OWL) format. This file is used to link to the product ontology described in the following section.
From the robot belief state, we get the product EAN as a unique product identifier. It is
easy to find product information on the Web once a EAN is given. From this information
we created a product ontology/ knowledge graph following the Linked Data standards
set by Bizer, Heath et al. [
The implementation can be seen in the following excerpt of the product ontology. It shows all definitions of the detergent class as well as one instance of it. For this project, we created instances for 186 household products that are available in our lab and can be perceived by robotic agents operating there.
Prefix declaration to be used throughout the document. <?xml version="1.0"?> <rdf:RDF xmlns:gr="http://purl.org/goodrelations/v1#" xmlns:products="http://knowrob.org/kb/dmproducts.owl#" xmlns:po="http://www.productontology.org/id"# xmlns:xsd="http://www.w3.org/2001/XMLSchema#"> <owl:Ontology rdf:about="http://knowrob.org/kb/dmproducts.owl"/>
Class declaration including link to product ontology. <!-- ////////////////////////////////////////// // Classes ////////////////////////////////////////// --> <!-- products#putzen-und-reinigen --> <owl:Class rdf:about="products#putzen-und-reinigen">
<rdfs:subClassOf rdf:resource="products#haushalt"/> 6http://www.productontology.org/ 7http://www.heppnetz.de/ontologies/goodrelations/v1
<owl:sameAs rdf:resource="po:Detergent"/> </owl:Class>
Instance declaration including Good Relations web vocabulary. <!-- ////////////////////////////////////////// // Individuals ////////////////////////////////////////// --> <!-- products#
sagrotan-allzweckreiniger-reine-frische-p4002448128533 --> <owl:NamedIndividual rdf:about="products#
sagrotan-allzweckreiniger-reine-frische-p4002448128533"> <rdf:type rdf:resource="products#putzen-und-reinigen"/> <gr:hasBrand rdf:resource="products#sagrotan"/> <gr:hasEAN UCC-13 rdf:xsd="datatype#
integer">4002448128533</gr:hasEAN UCC-13> <gr:name rdf:xsd="datatype#
Literal">Sagrotan Allzweckreiniger Reine Frische</gr:name> </owl:NamedIndividual>
Creating links to external knowledge graphs. <!-- ////////////////////////////////////////// // Annotations ////////////////////////////////////////// --> <rdf:Description rdf:about="products#putzen-und-reinigen"> <rdfs:label rdf:xsd="datatype#
Literal">Detergent;@en</rdfs:label> <owl:sameAs rdf:resource="https://www.wikidata.org/entity/Q334637"/> <owl:sameAs rdf:resource="http://conceptnet.io/c/en/detergent> <rdfs:label rdf:xsd="datatype#
Literal">Reinigungsmittel;@de</rdfs:label> </rdf:Description> </rdf:RDF>
Figure 2 shows the taxonomy of the created product ontology. These are main product categories extracted from drugstore websites and might need to be adapted when applied to different domains. Although this classification could be broken down even further, for the application in this project the 103 created classes seem sufficient to solve the given problem of linking product information to action information.
It was already stated in the previous section that each class definition in the product ontology is described to be <class> <owl:sameAs> <other Linked Data source class>.
If we look at the Wikidata page for “laundry detergent” shown in Figure 2, we can see the previously stated product to action relation <laundry detergent> <use> <laundering>.
The link to ConceptNet results in: <Things that require> <laundry detergent>: <washing your clothes>
These links close the gap to link the initial robot sensor data to an action by using the product ontology. The cognitive robotic agent who perceives a product can send its EAN to the knowledge graph and query for an action to be performed with it. Then, depending on the query implementation, the agent can get either of the mentioned results, or any other Linked Data source result.
The last step is to make the robot understand this result by mapping the linked action to internally known actions of the robotic agent. Agents need to be told that an internally known action "washing clothes" is the same as the given results "washing your clothes" or "laundering".
This paper introduced an approach to help a cognitive robotic agent make sense of actions that can be performed with objects they perceive. We concentrated on products, i.e. objects that can be identified through a barcode, to simplify the situation. These products and their unique identifier EAN can be accessed through the robot’s belief state. We then use the EAN to classify the products in a product ontology following Semantic Web standards. Lastly, the product class information links to other data sources like Wikidata or ConceptNet to find actions related to the product classes. These actions need to be mapped to internally known actions of the robotic agent so that it knows which action to perform with a given product.
The preliminary findings are very promising for discovering possible actions to be performed with products that are linked to Linked Data sources. The results of this work will be further investigated in future work to include more Linked Data sources, along with a thorough evaluation of the completeness and accuracy of the resulting knowledge graph. Additionally, it would be interesting to include objects without a barcode, as well as other semantic object information like object depositories.