=Paper=
{{Paper
|id=Vol-2708/robontics2
|storemode=property
|title=Using Linked Data to Help Robots Understand Product-related Actions
|pdfUrl=https://ceur-ws.org/Vol-2708/robontics2.pdf
|volume=Vol-2708
|authors=Michaela Kümpel,Anna de Groot,Ilaria Tiddi,Michael Beetz
|dblpUrl=https://dblp.org/rec/conf/jowo/KumpelGTB20
}}
==Using Linked Data to Help Robots Understand Product-related Actions==
https://ceur-ws.org/Vol-2708/robontics2.pdf
Using Linked Data to Help Robots
Understand Product-related Actions1,2
Michaela KÜMPEL a,3 and Anna DE GROOT b and Ilaria TIDDI b and
Michael BEETZ a
a Institute of Artificial Intelligence, University of Bremen, Germany
b Knowledge Representation and Reasoning Group, Vrije Universiteit Amsterdam,
Netherlands
Abstract. Household robots need semantics to understand that a detergent is a
cleaning product that can be used to clean physical objects like a table, but laundry
detergent is only used to clean/wash laundry. A safely acting autonomous robot
should also know that both will not be used as ingredients for meal preparation. We
propose a new approach to connect robot sensor data to Linked Data in order to
give robotic agents semantic product information about objects that can be found
in their environment so that the action to be performed with a given object can be
inferred. For this, we use the robot’s belief state when recognizing a product and
link it to a product ontology that follows Semantic Web standards. We then use the
product class information to fetch further information from external sources like
Wikidata or ConceptNet that contain action information (e.g. laundry detergent is
used for laundering). At last, the action results are mapped to internally known ac-
tions of the robotic agent so that it knows which action can be performed with the
perceived object.
Keywords. Knowledge Graph, Linked Data, product ontology, knowledge representation,
knowledge acquisition
1. Introduction
Representing common sense knowledge in cognitive robots is a widely tackled research
challenge as it could help in tasks such as action planning. Action planning tasks often
involve autonomous robotic agents operating in human environments to help perform
mundane tasks like vacuum-cleaning or washing the dishes. In these environments, the
need for robotic agents to operate competently and safely becomes even more impor-
1 Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution
4.0 International (CC BY 4.0).
2 The research reported in this paper has been partially supported by the Federal Ministry for
Economic Affairs and Energy BMWi within the Knowledge4Retail project, subproject semantic Digital
Twin 01MK20001M (https://knowledge4retail.org), the European Commission within the H2020 REFILLS
projectID n. 731590, as well as the German Research Foundation DFG, as part of Collaborative Research
Center (Sonderforschungsbereich) 1320 “EASE - Everyday Activity Science and Engineering”, University of
Bremen (http://www.ease-crc.org/).
3 Corresponding Author: Michaela Kümpel is with the Institute of Artificial Intelligence, University of
Bremen, Am Fallturm 1, 28359 Bremen, Germany; E-mail: kording@uni-bremen.de.
Figure 1. Information linking approach in this work.
tant. Unfortunately, household robots today are far from operating autonomously when it
comes to more complex tasks [1]. One main reason for this lies in the fact that they miss
basic semantic knowledge. An agent does not know that a spoon can be used for eating
soup or that detergent can be used to clean a bowl but not as a replacement for milk in
breakfast cereals.
The necessity to integrate semantics into robot knowledge bases has been addressed
in relevant cognitive robotics literature [2,3,4]. An autonomous cognitive robot must have
encyclopedic knowledge about non-disputable things in its environment. It must also be
able to reason about more implicit knowledge, such as knowing that detergent is used for
cleaning.
This work proposes an approach to include implicit knowledge such as product in-
formation of objects a robotic agent can find in a household environment by using Linked
Data and information from the Semantic Web . The Semantic Web [5] aims at bringing
structure in the form of knowledge graphs and ontologies to the content of web pages.
This is achieved by using keywords and standardized formats to represent entities and
their relationships so that machines, software agents or robots can easily understand and
carry out tasks for users.
Entities and relationships in Semantic Web format are represented in the form of
a knowledge graph [6]. In this work, we refer to a knowledge graph as composed of
a factual layer as well as an ontology layer, with the ontology layer giving meaning
to its facts and allowing reasoning on it, thereby making it utilizable for performing
different tasks. Moreover, knowledge graphs are useful for data representation due to
their scalability, format, and ability to be used for automatic extraction of information
[7]. When knowledge graphs are linked to each other, we refer to them as Linked Data
[8,6]. There are a handful Linked Data sources available which are related to common
sense knowledge [9].
In this work, we connect robot sensor data to a product ontology, which again
links to data sources from the Web. Figure 1 visualizes the links created in an exam-
ple use-case. A household robot operating in the kitchen perceives an object, which it
reasons about in its belief state. The object has a related pose and a barcode scanned
by the robot. The barcode also contains the European Article Number (EAN), which
can be linked to further information about the object in the linked product ontology.
Through this link the agent already knows that the object is a laundry detergent. By
linking this product class to existing knowledge graphs like Wikidata or ConceptNet
our agent can find the information that laundry detergent can be used for laundering
(( )4 or (
: )5 ). These Linked Data sources
offer links to even more sources (e.g. WordNet or DBpedia).
The example in Figure 1 reflects the goal of building a knowledge graph in this
project. We use a knowledge graph to acquire and integrate information from the Web
and represent this into an ontology so that a reasoner can be used to derive new knowl-
edge [10]. The project aims at utilizing web information for autonomous robots in house-
hold environments. We do not claim to already integrate all interesting data from the
Web. The emphasis of this paper is to showcase the effect of linking product information
to action information and can be seen as a starting point for further investigation. The
main contributions of this paper are: first, we connect Linked Data to the KnowRob on-
tology [11,4]. Second, we create a product ontology based on standard Semantic Web vo-
cabulary. Third, we link the product ontology to Linked Data sources like Wikidata and
ConceptNet to close the semantic gaps from object over product information to action.
2. Robot Belief State
Each robotic agent uses different sensors to make sense of its environment. The effort
for implementing common sense into a cognitive robot is a highly considered topic in
the field of cognitive robots. Different sources have been able to provide a variety of in-
formation to robots, including WordNet, OpenCyc, Freebase, ConceptNet, and OMICS
[12]. WordNet has been useful to help robots understand natural language [13,3,14],
while ConceptNet is a widely used knowledge base for achieving non-trivial encyclope-
dic knowledge in common sense driven projects [15]. Image sources, such as ShapeNet,
have been useful for knowledge grounding in object recognition tasks [16].
The KnowRob system, first introduced in 2009 and later extended to a newer version
in 2018, is at the forefront of cognitive robot work in the household domain in terms of
the extent of information its knowledge base represents [12]. It can be seen as one of
the currently most influential ontology-based knowledge representation and processing
systems [17,12] that also includes virtual environment models and links encyclopedic
knowledge to other input information in order for autonomous robots to successfully per-
form tasks with missing information. The system works by combining knowledge from
a robot’s sensor input and the encyclopedic and common sense knowledge its knowledge
base holds. This allows the robot to answer queries about the location of objects based
on its function, about action positions, and about finishing incomplete instructions.
For this work, we will focus on the robot belief state retrieved by the KnowRob
system and link it to knowledge graphs. To simplify the situation, we will only focus on
products that can be identified via barcode. A barcode can easily be recognized even by
simple sensors and still uniquely identifies a product. This makes us focus on ontologies
instead of object detection.
Since KnowRob already is an ontology-based system, a robotic household agent
will store its belief state including object position and its barcode in an Web Ontology
4 https://www.wikidata.org/wiki/Q910284
5 http://conceptnet.io/c/en/laundry detergent
Language (OWL) format. This file is used to link to the product ontology described in
the following section.
3. Product Ontology
From the robot belief state, we get the product EAN as a unique product identifier. It is
easy to find product information on the Web once a EAN is given. From this information
we created a product ontology/ knowledge graph following the Linked Data standards
set by Bizer, Heath et al. [8], which are:
• use the RDF data model to publish structured data on the web. A RDF triple can
be represented in the following way: . A
set of RDF triples is called a RDF graph.
• use RDF links to interlink data from different data sources. Linkage of the data
generates a graph, in which the nodes are Uniform Resource Identifiers (URIs) of
the represented entities and edges resemble the relation between two nodes.
• re-use as many existing URIs as possible and unique identifiers like EAN as part
of the URI. We will mostly be using the product ontology classes6 .
• re-use existing terms/vocabularies if possible. The W3C provides a set of standard
vocabularies. Since we are building a product ontology, we use the Good Relations
web vocabulary for E-Commerce7 .
• use the owl:sameAs property to interlink two data sources. In our case we will
use this property to link product classes to Wikidata and ConceptNet, which offer
links to other Linked Data sources like WordNet or DBPedia.
The implementation can be seen in the following excerpt of the product ontology. It
shows all definitions of the detergent class as well as one instance of it. For this project,
we created instances for 186 household products that are available in our lab and can be
perceived by robotic agents operating there.
Prefix declaration to be used throughout the document.
Instance declaration including Good Relations web vocabulary.
4002448128533
Sagrotan Allzweckreiniger Reine Frische
Creating links to external knowledge graphs.
Detergent;@en
Reinigungsmittel;@de
Figure 2 shows the taxonomy of the created product ontology. These are main prod-
uct categories extracted from drugstore websites and might need to be adapted when ap-
plied to different domains. Although this classification could be broken down even fur-
ther, for the application in this project the 103 created classes seem sufficient to solve the
given problem of linking product information to action information.
4. Linking to External Knowledge Graphs
It was already stated in the previous section that each class definition in the product
ontology is described to be
.
If we look at the Wikidata page for “laundry detergent” shown in Figure 2, we can see
the previously stated product to action relation
Figure 2. Product taxonomy linking EAN to product class information in the product taxonomy on the left,
which is then linked to action information on the right example webpage.
.
The link to ConceptNet results in:
:
These links close the gap to link the initial robot sensor data to an action by using the
product ontology. The cognitive robotic agent who perceives a product can send its EAN
to the knowledge graph and query for an action to be performed with it. Then, depending
on the query implementation, the agent can get either of the mentioned results, or any
other Linked Data source result.
The last step is to make the robot understand this result by mapping the linked action
to internally known actions of the robotic agent. Agents need to be told that an internally
known action "washing clothes" is the same as the given results "washing your
clothes" or "laundering".
5. Conclusion
This paper introduced an approach to help a cognitive robotic agent make sense of ac-
tions that can be performed with objects they perceive. We concentrated on products, i.e.
objects that can be identified through a barcode, to simplify the situation. These prod-
ucts and their unique identifier EAN can be accessed through the robot’s belief state. We
then use the EAN to classify the products in a product ontology following Semantic Web
standards. Lastly, the product class information links to other data sources like Wikidata
or ConceptNet to find actions related to the product classes. These actions need to be
mapped to internally known actions of the robotic agent so that it knows which action to
perform with a given product.
The preliminary findings are very promising for discovering possible actions to be
performed with products that are linked to Linked Data sources. The results of this work
will be further investigated in future work to include more Linked Data sources, along
with a thorough evaluation of the completeness and accuracy of the resulting knowledge
graph. Additionally, it would be interesting to include objects without a barcode, as well
as other semantic object information like object depositories.
References
[1] Marcus G, Davis E. Rebooting AI: Building artificial intelligence we can trust. Pantheon; 2019.
[2] Tenorth M, Jain D, Beetz M. Knowledge processing for cognitive robots. KI-Künstliche Intelligenz.
2010;24(3):233–240.
[3] Kunze L, Tenorth M, Beetz M. Putting people’s common sense into knowledge bases of household
robots. In: Annual Conference on Artificial Intelligence. Springer; 2010. p. 151–159.
[4] Beetz M, Beßler D, Haidu A, Pomarlan M, Bozcuoğlu AK, Bartels G. Know rob 2.0—a 2nd generation
knowledge processing framework for cognition-enabled robotic agents. In: 2018 IEEE International
Conference on Robotics and Automation (ICRA). IEEE; 2018. p. 512–519.
[5] Berners-Lee T, Hendler J, Lassila O, et al. The semantic web. Scientific american. 2001;284(5):28–37.
[6] Paulheim H. Knowledge graph refinement: A survey of approaches and evaluation methods. Semantic
web. 2017;8(3):489–508.
[7] Ehrlinger L, Wöß W. Towards a Definition of Knowledge Graphs. SEMANTiCS (Posters, Demos,
SuCCESS). 2016;48:1–4.
[8] Heath T, Hausenblas M, Bizer C, Cyganiak R, Hartig O. How to publish linked data on the web. In:
Tutorial in the 7th International Semantic Web Conference, Karlsruhe, Germany; 2008. .
[9] Heist N, Hertling S, Ringler D, Paulheim H. Knowledge Graphs on the Web–an Overview. arXiv
preprint arXiv:200300719. 2020.
[10] Ehrlinger L, Wöß W. Towards a Definition of Knowledge Graphs. In: SEMANTiCS (Posters, Demos,
SuCCESS); 2016. .
[11] Tenorth M, Beetz M. KnowRob – Knowledge Processing for Autonomous Personal Robots. In:
IEEE/RSJ International Conference on Intelligent Robots and Systems; 2009. p. 4261–4266.
[12] Thosar M, Zug S, Skaria AM, Jain A. A Review of Knowledge Bases for Service Robots in Household
Environments. In: AIC; 2018. p. 98–110.
[13] Moritz T, Ulrich K, Dejan P, Michael B. Web-enabled robots-robots that use the web as an information
resource. IEEE Robotics &Automation Magazine. 2011;18(2):58–68.
[14] Tandon N, Hariman C, Urbani J, Rohrbach A, Rohrbach M, Weikum G. Commonsense in parts: Min-
ing part-whole relations from the web and image tags. In: Thirtieth AAAI Conference on Artificial
Intelligence; 2016. .
[15] Ostermann S, Roth M, Modi A, Thater S, Pinkal M. Semeval-2018 task 11: Machine comprehension
using commonsense knowledge. In: Proceedings of the 12th International Workshop on semantic eval-
uation; 2018. p. 747–757.
[16] Daga ACGBE, Motta E. Improving Object Recognition for Mobile Robots by Leveraging Common-
sense Knowledge Bases;.
[17] Barreto M, Alberto OA, Beßler D, Bermejo J, Diab M, Jan R, et al. A Review and Comparison of
Ontology-based Approaches to Robot Autonomy. The Knowledge Engineering Review. 2019 08.