Using Linked Data to Help Robots Understand Product-related Actions1,2 Michaela KÜMPEL a,3 and Anna DE GROOT b and Ilaria TIDDI b and Michael BEETZ a a Institute of Artificial Intelligence, University of Bremen, Germany b Knowledge Representation and Reasoning Group, Vrije Universiteit Amsterdam, Netherlands Abstract. Household robots need semantics to understand that a detergent is a cleaning product that can be used to clean physical objects like a table, but laundry detergent is only used to clean/wash laundry. A safely acting autonomous robot should also know that both will not be used as ingredients for meal preparation. We propose a new approach to connect robot sensor data to Linked Data in order to give robotic agents semantic product information about objects that can be found in their environment so that the action to be performed with a given object can be inferred. For this, we use the robot’s belief state when recognizing a product and link it to a product ontology that follows Semantic Web standards. We then use the product class information to fetch further information from external sources like Wikidata or ConceptNet that contain action information (e.g. laundry detergent is used for laundering). At last, the action results are mapped to internally known ac- tions of the robotic agent so that it knows which action can be performed with the perceived object. Keywords. Knowledge Graph, Linked Data, product ontology, knowledge representation, knowledge acquisition 1. Introduction Representing common sense knowledge in cognitive robots is a widely tackled research challenge as it could help in tasks such as action planning. Action planning tasks often involve autonomous robotic agents operating in human environments to help perform mundane tasks like vacuum-cleaning or washing the dishes. In these environments, the need for robotic agents to operate competently and safely becomes even more impor- 1 Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 2 The research reported in this paper has been partially supported by the Federal Ministry for Economic Affairs and Energy BMWi within the Knowledge4Retail project, subproject semantic Digital Twin 01MK20001M (https://knowledge4retail.org), the European Commission within the H2020 REFILLS projectID n. 731590, as well as the German Research Foundation DFG, as part of Collaborative Research Center (Sonderforschungsbereich) 1320 “EASE - Everyday Activity Science and Engineering”, University of Bremen (http://www.ease-crc.org/). 3 Corresponding Author: Michaela Kümpel is with the Institute of Artificial Intelligence, University of Bremen, Am Fallturm 1, 28359 Bremen, Germany; E-mail: kording@uni-bremen.de. Figure 1. Information linking approach in this work. tant. Unfortunately, household robots today are far from operating autonomously when it comes to more complex tasks [1]. One main reason for this lies in the fact that they miss basic semantic knowledge. An agent does not know that a spoon can be used for eating soup or that detergent can be used to clean a bowl but not as a replacement for milk in breakfast cereals. The necessity to integrate semantics into robot knowledge bases has been addressed in relevant cognitive robotics literature [2,3,4]. An autonomous cognitive robot must have encyclopedic knowledge about non-disputable things in its environment. It must also be able to reason about more implicit knowledge, such as knowing that detergent is used for cleaning. This work proposes an approach to include implicit knowledge such as product in- formation of objects a robotic agent can find in a household environment by using Linked Data and information from the Semantic Web . The Semantic Web [5] aims at bringing structure in the form of knowledge graphs and ontologies to the content of web pages. This is achieved by using keywords and standardized formats to represent entities and their relationships so that machines, software agents or robots can easily understand and carry out tasks for users. Entities and relationships in Semantic Web format are represented in the form of a knowledge graph [6]. In this work, we refer to a knowledge graph as composed of a factual layer as well as an ontology layer, with the ontology layer giving meaning to its facts and allowing reasoning on it, thereby making it utilizable for performing different tasks. Moreover, knowledge graphs are useful for data representation due to their scalability, format, and ability to be used for automatic extraction of information [7]. When knowledge graphs are linked to each other, we refer to them as Linked Data [8,6]. There are a handful Linked Data sources available which are related to common sense knowledge [9]. In this work, we connect robot sensor data to a product ontology, which again links to data sources from the Web. Figure 1 visualizes the links created in an exam- ple use-case. A household robot operating in the kitchen perceives an object, which it reasons about in its belief state. The object has a related pose and a barcode scanned by the robot. The barcode also contains the European Article Number (EAN), which can be linked to further information about the object in the linked product ontology. Through this link the agent already knows that the object is a laundry detergent. By linking this product class to existing knowledge graphs like Wikidata or ConceptNet our agent can find the information that laundry detergent can be used for laundering (( )4 or ( : )5 ). These Linked Data sources offer links to even more sources (e.g. WordNet or DBpedia). The example in Figure 1 reflects the goal of building a knowledge graph in this project. We use a knowledge graph to acquire and integrate information from the Web and represent this into an ontology so that a reasoner can be used to derive new knowl- edge [10]. The project aims at utilizing web information for autonomous robots in house- hold environments. We do not claim to already integrate all interesting data from the Web. The emphasis of this paper is to showcase the effect of linking product information to action information and can be seen as a starting point for further investigation. The main contributions of this paper are: first, we connect Linked Data to the KnowRob on- tology [11,4]. Second, we create a product ontology based on standard Semantic Web vo- cabulary. Third, we link the product ontology to Linked Data sources like Wikidata and ConceptNet to close the semantic gaps from object over product information to action. 2. Robot Belief State Each robotic agent uses different sensors to make sense of its environment. The effort for implementing common sense into a cognitive robot is a highly considered topic in the field of cognitive robots. Different sources have been able to provide a variety of in- formation to robots, including WordNet, OpenCyc, Freebase, ConceptNet, and OMICS [12]. WordNet has been useful to help robots understand natural language [13,3,14], while ConceptNet is a widely used knowledge base for achieving non-trivial encyclope- dic knowledge in common sense driven projects [15]. Image sources, such as ShapeNet, have been useful for knowledge grounding in object recognition tasks [16]. The KnowRob system, first introduced in 2009 and later extended to a newer version in 2018, is at the forefront of cognitive robot work in the household domain in terms of the extent of information its knowledge base represents [12]. It can be seen as one of the currently most influential ontology-based knowledge representation and processing systems [17,12] that also includes virtual environment models and links encyclopedic knowledge to other input information in order for autonomous robots to successfully per- form tasks with missing information. The system works by combining knowledge from a robot’s sensor input and the encyclopedic and common sense knowledge its knowledge base holds. This allows the robot to answer queries about the location of objects based on its function, about action positions, and about finishing incomplete instructions. For this work, we will focus on the robot belief state retrieved by the KnowRob system and link it to knowledge graphs. To simplify the situation, we will only focus on products that can be identified via barcode. A barcode can easily be recognized even by simple sensors and still uniquely identifies a product. This makes us focus on ontologies instead of object detection. Since KnowRob already is an ontology-based system, a robotic household agent will store its belief state including object position and its barcode in an Web Ontology 4 https://www.wikidata.org/wiki/Q910284 5 http://conceptnet.io/c/en/laundry detergent Language (OWL) format. This file is used to link to the product ontology described in the following section. 3. Product Ontology From the robot belief state, we get the product EAN as a unique product identifier. It is easy to find product information on the Web once a EAN is given. From this information we created a product ontology/ knowledge graph following the Linked Data standards set by Bizer, Heath et al. [8], which are: • use the RDF data model to publish structured data on the web. A RDF triple can be represented in the following way: . A set of RDF triples is called a RDF graph. • use RDF links to interlink data from different data sources. Linkage of the data generates a graph, in which the nodes are Uniform Resource Identifiers (URIs) of the represented entities and edges resemble the relation between two nodes. • re-use as many existing URIs as possible and unique identifiers like EAN as part of the URI. We will mostly be using the product ontology classes6 . • re-use existing terms/vocabularies if possible. The W3C provides a set of standard vocabularies. Since we are building a product ontology, we use the Good Relations web vocabulary for E-Commerce7 . • use the owl:sameAs property to interlink two data sources. In our case we will use this property to link product classes to Wikidata and ConceptNet, which offer links to other Linked Data sources like WordNet or DBPedia. The implementation can be seen in the following excerpt of the product ontology. It shows all definitions of the detergent class as well as one instance of it. For this project, we created instances for 186 household products that are available in our lab and can be perceived by robotic agents operating there. Prefix declaration to be used throughout the document. Class declaration including link to product ontology. 6 http://www.productontology.org/ 7 http://www.heppnetz.de/ontologies/goodrelations/v1 Instance declaration including Good Relations web vocabulary. 4002448128533 Sagrotan Allzweckreiniger Reine Frische Creating links to external knowledge graphs. Detergent;@en Reinigungsmittel;@de Figure 2 shows the taxonomy of the created product ontology. These are main prod- uct categories extracted from drugstore websites and might need to be adapted when ap- plied to different domains. Although this classification could be broken down even fur- ther, for the application in this project the 103 created classes seem sufficient to solve the given problem of linking product information to action information. 4. Linking to External Knowledge Graphs It was already stated in the previous section that each class definition in the product ontology is described to be . If we look at the Wikidata page for “laundry detergent” shown in Figure 2, we can see the previously stated product to action relation Figure 2. Product taxonomy linking EAN to product class information in the product taxonomy on the left, which is then linked to action information on the right example webpage. . The link to ConceptNet results in: : These links close the gap to link the initial robot sensor data to an action by using the product ontology. The cognitive robotic agent who perceives a product can send its EAN to the knowledge graph and query for an action to be performed with it. Then, depending on the query implementation, the agent can get either of the mentioned results, or any other Linked Data source result. The last step is to make the robot understand this result by mapping the linked action to internally known actions of the robotic agent. Agents need to be told that an internally known action "washing clothes" is the same as the given results "washing your clothes" or "laundering". 5. Conclusion This paper introduced an approach to help a cognitive robotic agent make sense of ac- tions that can be performed with objects they perceive. We concentrated on products, i.e. objects that can be identified through a barcode, to simplify the situation. These prod- ucts and their unique identifier EAN can be accessed through the robot’s belief state. We then use the EAN to classify the products in a product ontology following Semantic Web standards. Lastly, the product class information links to other data sources like Wikidata or ConceptNet to find actions related to the product classes. These actions need to be mapped to internally known actions of the robotic agent so that it knows which action to perform with a given product. The preliminary findings are very promising for discovering possible actions to be performed with products that are linked to Linked Data sources. The results of this work will be further investigated in future work to include more Linked Data sources, along with a thorough evaluation of the completeness and accuracy of the resulting knowledge graph. Additionally, it would be interesting to include objects without a barcode, as well as other semantic object information like object depositories. References [1] Marcus G, Davis E. Rebooting AI: Building artificial intelligence we can trust. Pantheon; 2019. [2] Tenorth M, Jain D, Beetz M. Knowledge processing for cognitive robots. KI-Künstliche Intelligenz. 2010;24(3):233–240. [3] Kunze L, Tenorth M, Beetz M. Putting people’s common sense into knowledge bases of household robots. In: Annual Conference on Artificial Intelligence. Springer; 2010. p. 151–159. [4] Beetz M, Beßler D, Haidu A, Pomarlan M, Bozcuoğlu AK, Bartels G. Know rob 2.0—a 2nd generation knowledge processing framework for cognition-enabled robotic agents. In: 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE; 2018. p. 512–519. [5] Berners-Lee T, Hendler J, Lassila O, et al. The semantic web. Scientific american. 2001;284(5):28–37. [6] Paulheim H. Knowledge graph refinement: A survey of approaches and evaluation methods. Semantic web. 2017;8(3):489–508. [7] Ehrlinger L, Wöß W. Towards a Definition of Knowledge Graphs. SEMANTiCS (Posters, Demos, SuCCESS). 2016;48:1–4. [8] Heath T, Hausenblas M, Bizer C, Cyganiak R, Hartig O. How to publish linked data on the web. In: Tutorial in the 7th International Semantic Web Conference, Karlsruhe, Germany; 2008. . [9] Heist N, Hertling S, Ringler D, Paulheim H. Knowledge Graphs on the Web–an Overview. arXiv preprint arXiv:200300719. 2020. [10] Ehrlinger L, Wöß W. Towards a Definition of Knowledge Graphs. In: SEMANTiCS (Posters, Demos, SuCCESS); 2016. . [11] Tenorth M, Beetz M. KnowRob – Knowledge Processing for Autonomous Personal Robots. In: IEEE/RSJ International Conference on Intelligent Robots and Systems; 2009. p. 4261–4266. [12] Thosar M, Zug S, Skaria AM, Jain A. A Review of Knowledge Bases for Service Robots in Household Environments. In: AIC; 2018. p. 98–110. [13] Moritz T, Ulrich K, Dejan P, Michael B. Web-enabled robots-robots that use the web as an information resource. IEEE Robotics &Automation Magazine. 2011;18(2):58–68. [14] Tandon N, Hariman C, Urbani J, Rohrbach A, Rohrbach M, Weikum G. Commonsense in parts: Min- ing part-whole relations from the web and image tags. In: Thirtieth AAAI Conference on Artificial Intelligence; 2016. . [15] Ostermann S, Roth M, Modi A, Thater S, Pinkal M. Semeval-2018 task 11: Machine comprehension using commonsense knowledge. In: Proceedings of the 12th International Workshop on semantic eval- uation; 2018. p. 747–757. [16] Daga ACGBE, Motta E. Improving Object Recognition for Mobile Robots by Leveraging Common- sense Knowledge Bases;. [17] Barreto M, Alberto OA, Beßler D, Bermejo J, Diab M, Jan R, et al. A Review and Comparison of Ontology-based Approaches to Robot Autonomy. The Knowledge Engineering Review. 2019 08.