In the Semantic Web, ontologies and knowledge graphs have been a research focus for a while, resulting in partially interlinked domain ontologies with different consumer applications. It has been shown that knowledge graphs are a powerful tool that can be used to efficiently answer questions. However, there is a need for research on linking knowledge graphs to real world systems. If knowledge graphs can be grounded in the real world, the contained knowledge becomes perceivable and actionable to users. While we believe that this connection of object to environment information is beneficial for many different domains, in this work we concentrate on product information, which is due to • Environment structure. Products often hold a barcode as identifier and can usually be found in very structured environments: they are stored in shelves or cabinets, which helps in identifying perceived objects. • Availability of environment information. Digital agents can already create environment models of retail stores and household environments autonomously [1,2]. • Availability of Web Information. Product information as well as product images are available in many web stores and consumer websites.
It has been shown that semantic Digital Twins (semDTs) can successfully be used for linking object to environment information in the retail domain for customer support [3]. The concept of a Digital Twin, "the digital equivalent to a physical product" was introduced by Grieves in 2003 [4]. Using Digital Twins to create exact virtual representations of the real world has been a technology trend in digitisation of manufacturing and industry [5]. A semantic Digital Twin can be described as the semantic connection of environment information into a digital representation of an environment. While a semDT offers detailed environment information, it only offers limited object or product information from connected enterprise management systems. We therefore extend the idea and link semDTs to a knowledge graph of product information acquired from Web sources. In this work, we refer to a knowledge graph as an aggregation of ontologies comprising Web information that allow reasoning about the contained facts. With this connection we get access to both detailed environment and vast product information.
This work introduces ProductKG, a knowledge graph that consists of semantic product information from the Semantic Web and the World Wide Web in modular ontologies that are linked to environment information which can be combined and customised for various applications. It enables a customer browsing the aisles with their smartphone to highlight all deodorants that were dermatologically tested but also visualise nutritional information for a product in the fridge, for example. In this work we describe the creation of the ProductKG knowledge graph and its ontologies. We showcase the benefits of using ProductKG in example applications in retail and household environments as depicted exemplary in Figure 1. The applications use the same basic user profile with different implementations like highlighting product preferences, recommending recipes for available products or showing preferred information such as product labels. The novelty of ProductKG lies in the connection of exact location information from a semantic Digital Twin to modular product information acquired from the Web into a knwoldge graph, leading to various applications with consumer focus as an embodiment of the contained knowledge.
The main contributions of this paper are: first, we create ProductKG with its modular ontologies based on Web information, link it to other Semantic Web sources and make it publicly available. Second, we integrate semDT environment information in such a way that it can be used by AR devices as well as robots. Third, we showcase the applicability of ProductKG in daily environments on smartphone and smart glasses, thus making ProductKG perceivable and actionable. These contributions are validated by a system usability study for one of the applications.
On the one hand, recent research has focused on standardised mapping, in particular of retail environments. The benefits of using standardised maps has been identified in [6]. The idea of creating semantic Digital Twins of retail environments as standardised maps has been proposed in [3] for shopping assistance or shop-floor assistance for product refilling. It has additionally been shown how semDTs can be used in retail applications to route a customer to a searched product on different devices such as smartphone and robot [2]. While the shown applications reveal the power of semantic Digital Twins and the included environment information, they lack detailed product information and cannot easily be extended to other applications like finding substitute products in a store or recipes one can prepare with a given product.
On the other hand, there has been research on creation of product knowledge graphs (e.g. [7,8]) and food product knowledge graphs, resulting in partially interlinked domain ontologies with consumer applications ranging from agricultural over health specific to recipe centered (e.g. [9,10,11]). Although the created ontologies, knowledge graphs and their exemplary applications are impressive, they do not connect the contained object or product information to environment information.
This work aims at bringing together both research topics by interlinking semantic Digital Twins with a product knowledge graph.
Knowledge graphs can be created a) manually (which will not be further discussed in the following), b) by accessing structured external (Web) sources and c) by accessing unstructured external (Web) sources [12]. ProductKG consists of a location ontology created by accessing a robot map (as described in [1]) as well as modular product information ontologies that contain structured or unstructured Web information as described in the following.
In order to generalise applications for the use on different devices and in differing applications, we need to have precise, interchangeable environment information. A standardized map is also beneficial for localization of products in a store. We create a location ontology that is linked to precise environment information in a semantic Digital Twin environment model. A location ontology can be generated for any indoor environment following the approach described in [1] and [3]. We additionally derive product dimension information from the perception system. The robot uses its RGB and RGB-D sensors to identify the amount of consecutive products and calculate product width, height and depth.
The positions in the semantic Digital Twin use a fixed reference frame with a given origin, usually set to a randomly chosen corner of the room. In contrast to this, Augmented Reality (AR) devices use the varying device position when starting an application as origin and display its digital content relative to the device position. To solve this discrepancy for a use of the location ontology in both robot and AR applications, we encode all product locations in the location ontology relative to shelf, table or other object positions as described in more detail in [2]. In the AR shopping applications we then use spatial perception and match the world origin of the game to the reference frame origin of the semantic Digital Twin. We also use world anchors for locking positions in the physical space, a technique that has proven effective in other work [13]. World anchors need to be created once and will be loaded in every subsequent run of the application. Thus, product positions can be inferred relative to world anchors and digital content can be displayed.
ProductKG integrates and links to structured Web information. On the one hand, we integrate parts of structured Web information like the FoodKG ontologies [11] for a product-specific recipe recommendation. This is done by reusing the food product classification of the FoodOn ontology [9], which we intertwine with the product classification in the product taxonomy. ProductKG also integrates nutrition information from the Compositional Dietary Nutrition (cdno) ontology [14] as well as nutritional product information like Nutri-Score, product labels or packaging size for food products from Open Food Facts1 . For food classes, we access product class nutrition information from the FoodData central2 from the U.S. Department of agriculture. The information is stored in a food-nutrition ontology, which then links to the FoodOn classification and therefore can be used for food products in the product taxonomy.
On the other hand, we interlink ProductKG to existing data sources by re-using many existing URIs and standard vocabulary as proposed in the Linked Data standards by Bizer, Heath et al. [15]. We do not, however, use the owl:sameAs property but the oboInOwl:hasDbXref annotation property to interlink two data sources, as proposed by the gene ontology (GO) consortium [16]. This is due to the fact that owl:sameAs increases file sizes since it creates duplicate entries while the oboInOwl:hasDbXref not only avoids duplicate entries but also looks more descriptive to a user browsing the ontologies using Protégé. Both the GO and the FoodOn workgroup are part of the OBO foundry [17], a community focussing on the development of interoperable ontologies for the biological sciences. We chose to reuse many of the OBO foundry ontologies (besides FoodOn and cdno we link to the human disease ontology (doid) [18,19] and the human phenotype ontology [20]) since they are openly accessible, are already following the same upper ontologies and have a strong community.
The product classification is further linked to Wikidata and the Product Types Ontology3 wherever possible. We also use the Good Relations web vocabulary for E-Commerce4 . Furthermore, ProductKG links to chemical information about allergens in the Chemical entities of biological interest (ChEBI) 5 ontology. The allergen ontology additionally links to other Semantic Web sources like the National Drug File (NDRF) of BioPortal6 or Drugbank 7 . The advantage of using many Linked Data sources lies in its adaptability to changes. If a concept is changed in a linked source, the new information is immediately available in applications.
For creation of the ProductKG ontologies we additionally acquire unstructured Web information to create structured knowledge sources that machines can query. In order for ProductKG to be usable in various environments, it needs a general product taxonomy that can be used for different domains of daily activities like at home, in drugstores and grocery stores. Most techniques for taxonomy creation aim at properly modeling a domain by using a pattern-based approach for taxonomy induction [21] to learn simple but precise contextual patterns in a Web document. For a more generalised approach we use Information Extraction [22] techniques on online store sitemaps of different retail domains to automatically create a more general product taxonomy.
Since online stores usually offer a sitemap in their robots file 8 , we use this sitemap for Information Extraction to avoid spamming of websites and be more time-efficient. Sitemaps of online stores are usually hidden .xml files that consist of a list of Uniform Resource Locators (URLs) of children websites so that crawlers can easily access them. For a general product classification we create a product class structure from sitemaps of two online stores representing different retail sectors, namely Aldi-Nord (grocery store) and dm (drugstore). We then use Owlready to generate ontologies out of the extracted information. The resulting product taxonomies are merged into one global product taxonomy that additionally integrates parts of the FoodOn food product taxonomy. For this, extracted product categories are standardised to lower case words while special characters and numbers are deleted. This allows for an accurate matching of categories of different stores as will further be explained in Section 4.3.
We further use Information Extraction to acquire product ingredient information from online stores for use in the ingredient ontology and information like brand, awarded labels, weight, filling capacity and country of origin from Codecheck 9 , a consumer-oriented product information website. Again, we standardise the extracted information as before and manually add additional semantic information such as linking a specific label to a label category. What is more, product ingredients are classified as allergens according to information from the U.S. Food & Drug administration (FDA) website 10 .
We continue to explain the contained ontologies in ProductKG as well their accordance to the FAIR (Findable, Accessible, Interoperable, Reusable) principles [23] and how ProductKG can be extended in the following.
As depicted in Figure 2, the product builds the core of ProductKG. The modular ontologies in ProductKG have different foci that can be used to retrieve additional product information for different applications. All ontologies contained in ProductKG with their most relevant properties and their current use in applications as well as the number of axioms are detailed in Table 1. 9 Codecheck website: https://www.codecheck.info/ 10 FDA website: https://www.fda.gov/
Ontologies contained in ProductKG with their most relevant properties and their use in applications. Italic information sources belong to ProductKG while all other mentioned sources are external. One major challenge in ontology alignment of ontologies with such different foci is the use of diverse upper ontologies. While ProductKG is based on the DOLCE+DnS Ultralite (DUL) ontology with its descriptions of relations of actions, objects and agents, the OBO foundry ontologies are based on the Basic Formal Ontology (bfo) defining occurrents and continuants. Thus, external concepts like processes that are defined as occurents in bfo need to be aligned with the definitions of tasks, actions and processes as events from DUL. To simplify ontology reuse in ProductKG, we focus on the domain and solely integrate product information. In particular, the main DUL class ProductKG is based upon is PhysicalObject -the class of all objects perceivable in an environment. In DUL, a 𝑃 ℎ𝑦𝑠𝑖𝑐𝑎𝑙𝑂𝑏𝑗𝑒𝑐𝑡 ∈ 𝑂𝑏𝑗𝑒𝑐𝑡 ∈ 𝐸𝑛𝑡𝑖𝑡𝑦. In bfo, a 𝐹 𝑜𝑜𝑑𝑃 𝑟𝑜𝑑𝑢𝑐𝑡 ∈ 𝐹 𝑜𝑜𝑑𝑀 𝑎𝑡𝑒𝑟𝑖𝑎𝑙 ∈ 𝑀 𝑎𝑡𝑒𝑟𝑖𝑎𝑙𝐸𝑛𝑡𝑖𝑡𝑦 ∈ 𝐶𝑜𝑛𝑡𝑖𝑛𝑢𝑎𝑛𝑡 ∈ 𝐸𝑛𝑖𝑡𝑦. In ProductKG we align these by stating that 𝐹 𝑜𝑜𝑑𝑃 𝑟𝑜𝑑𝑢𝑐𝑡(bfo) ∈ 𝑔𝑟𝑜𝑐𝑒𝑟𝑖𝑒𝑠 ∈ 𝑃 𝑟𝑜𝑑𝑢𝑐𝑡𝑂𝑟𝑆𝑒𝑟𝑣𝑖𝑐𝑒(gr) ∈ 𝑃 ℎ𝑦𝑠𝑖𝑐𝑎𝑙𝑂𝑏𝑗𝑒𝑐𝑡(DUL) a food product of the Food Ontology is a subclass of groceries in ProductKG, which we define as a Product or Service of the Good Relations vocabulary, which then belongs to the physical object class of DUL.
The FAIR guiding principles for scientific data management and stewardship [23] were proposed in 2016 in order to improve the reusability of data holdings. They introduce four principles: Findability, Accessibility, Interoperability, and Reusability that we apply to ProductKG as explained in the following:
• Findability: ProductKG and its application page are openly accessible and can be found at https://michaelakuempel.github.io/ProductKG/. We use persistent URLs (http://purl. org/ProductKG/) for the contained ontologies that redirect the user to the current storage location of the ontologies. • Accessibility: ProductKG is publicly available with all ontologies, a link to its triple store SPARQL endpoint 11 as well as an grlc [25] RESTful API for the example queries 12demonstrated in the shopping assistant applications. • Interoperability: Section 3.2 describes the different vocabularies that are used or linked to in ProductKG. Hence, applications built on those ontologies can easily integrate Produc-tKG information. • Reusability: Entities in the ProductKG ontologies use annotation properties like language labels (which currently are implemented in english and german), comments and links to term defining webpages such as wikidata or the product ontology as descriptors. All ProductKG ontologies come with a clearly stated licence, can be downloaded and reused.
ProductKG can easily be extended if new sources are linked to product names. Using product names for classification has been shown to be quite efficient for linking. For ontologies, Word2Vec can be used to expand the contained knowledge by learning sibling classes [26] and for linking taxonomies one could use either Word2Vec or Node2Vec with good results. If we want to link quite different ontologies such as taxonomies to a brand ontology, the domain of interest changes from linking two "shampoo" classes to linking one "shampoo" class to "Elvital Dream Length Super Strengthening Shampoo", for example. Here, the distinct product name consists of multiple words. Doc2Vec has been shown to efficiently handle multiple words with context [27]. Therefore, we use Doc2Vec for linking of the different ontologies to the product taxonomy.
Even product master data can easily be linked to ProductKG, which is important if different users want to access ProductKG information. For this, we propose to do a preprocessing to have all product names in lower case format. We tested linking product master data containing about 300,000 product names and received a top-1 accuracy of 80% and a top 5 accuracy of 97%. What is more, if the algorithm suggested to have an accuracy of more than 90%, it was actually true in 95% of tested cases.
Using ProductKG, we can answer many complex questions about products, their ingredients, contained allergens, nutrition values, labels, brands or recipes that can be prepared with them, for example. The connection to the semantic Digital Twin additionally enables to locate products in the current environment. It has already been shown in previous work how semantic Digital Twins can successfully be used for shopping assistants to help customers find searched products [2]. The triple store used for storage of the knowledge graph offers a SPARQL endpoint for information access that can be accessed by different agents like robot or smartphone.
The following applications demonstrate the applicability of the ProductKG knowledge graph as one source of data for a range of consumer applications. In omnichannel applications with one source of data, it does not matter if the consumer accesses the information through a Web interface, a smartphone, smart glasses or via interaction with a robot. The following applications show how the same data can be accessed through various channels for different applications. The used devices in the applications are exchangeable.
This openly accessible application 13 highlights all objects that do not comply with a set preference on a HoloLens as depicted in Figure 1 on the left. A user can specify a preference such as an unwanted ingredient and all products that contain the ingredient get a digital overlay, a red X.
To localize the HoloLens in the retail laboratory we use spatial perception and match the world origin of the Unity game to the reference frame origin of the robot environment map as discussed in 3.1. To load an environment map, a user has to scan a QR code that loads the semantic Digital Twin map including the parent anchor. Product positions are inferred relative to this parent anchor. Each product in the game is created as a X game object named according to the ProductKG product names. Thereby, objects can be set visible or invisible automatically according to the query result.
A consumer in the store will see a menu on the wall, displaying options of preferences like "vegan", "eco-friendly" or "no alcohol". Once a preference is chosen, all opposing products will be highlighted as depicted in Figure 1.
This application highlights interesting product information using a smartphone. It uses image targets for image recognition. Therefore, we rely on images or barcodes of products available in daily environments. Hence this application works more reliable in the more static retail environment than in dynamic household environments.
A consumer using the app can scan a product they are interested in to instantly see interesting product information like which label the recognised product has as well as an information slider displaying the name of the recognized product. By clicking on or sliding of the product name, additional product information like contained ingredients and allergens or recipes that can be prepared with the product can be listed. Aside from that, the product can be added to the shopping cart. The smartphone application to visualize interesting product information is depicted in Figure 1 on the right for a recipe recommender and a shopping assistant.
ProductKG offers a basic user profile ontology that can be integrated and instantiated in ontologies like the food-nutrition or ingredient ontology for personalised applications. Figure 3 shows an example nutrition recommender application based on personal dietary preferences 14 . A user can (but does not need to) enter personal information like weight, age, gender and daily activity level. The data is not saved, but the user is instantiated based on the given input values. In Figure 3, a user is searching for products that are rich in vitamin D, nutritional values of coffee and nutrients and products that might treat a headache. The results pages show products in accordance to the preferences and query. Based on this, the user can proceed to see detailed product information or what recipes could be prepared with it. Through connected semDTs, users can also see where a searched product is available or compare prices.
For an assessment of system usability we performed a user study based on the system usability scale, which we extended for a question about the helpfulness of the application, i.e. if the users think that the dietary recommendation will help them in achieving their goals. We performed the user study with n=24 participants by promoting the website and randomly asking website visitors for their feedback based on the following hypotheses:
• We expect that the application is easy to use. • We expect that the idea of the website is understood quickly.
• We expect that the functionality of the website is understood easily.
• We expect the overall application and its subpages to be perceived equally complex.
The performed user study yielded some interesting results, some of which are visualised in Figure 4. Overall, the user study results suggest that users would reuse the application. 14 The Web Applications are available at: http://productkg.informatik.uni-bremen.de/ • As expected, the users did find the application in general easy to use. Surprisingly, this is only true for the general application. Some subpages were assessed as being too complex. • The users were able to quickly grasp the application and its subpages. Again, the overall application received better results than its subpages. • The functionality of the app was evaluated as easily understandable both for the general application and its subpages. • It is interesting to see that the results of the application in general and the subpages offering different information differ. Users were very interested in the symptom page and used it more than other pages. • The nutrition recommender subpage was evaluated as being very helpful.
• Most of the users would reuse the app. The symptom subpage received most interest.
Users said that they were most likely to reuse this part of the application. Those users that would not reuse the app mainly were disappointed in the design of the application.
This work describes the ProductKG knowledge graph with all its ontologies and points out the benefits of linking a knowledge graph to a semantic Digital Twin, making it perceivable and actionable. All sources used for information acquisition and creation of the contained ontologies are described. We show the applicability of the knowledge graph in various applications.
Having a publicly available knowledge graph makes it accessible to different machines, which we demonstrate by querying ProductKG by smart glass (HoloLens) and smartphone. The example applications emphasise the value of the connection of semantic product information to location information, while the power of the applications stem from the use of modular ontologies that can be personalized and queried through a single access point. ProductKG offers:
• A product taxonomy that can be reused for applications in different daily environments.
• A framework of modular ontologies that can be accessed for different consumer needs and applications. • A connection to existing ontologies like wikidata, FoodOn or ChEBI, allowing for further applications based on the contained knowledge. • A connection to exchangeable, standardised Digital Twin environment models, thereby enabling environment-dependent applications, bringing the knowledge to the user.
We believe that ProductKG is a powerful example of connecting knowledge graphs to semantic Digital Twins and will continue to use this approach for different domains such as bookstores as well as more user applications in daily environments. Future work will additionally focus on an integration of the different applications into a single daily activity application. Additionally, we hope to add more semDTs that are openly accessible and we plan on using knowledge graphs for enabling higher robot autonomy in daily environments.
We thank Naser Azizi and Sorin Arion for their support towards this work.
The research reported in this paper has been partially supported by the German Federal Ministy of Education and Research; Project-ID 16DHBKI047 "IntEL4CoRo -Integrated Learning Environment for Cognitive Robotics", University of Bremen, the Federal Ministry for Economic Affairs and Energy BMWi within the Knowledge4Retail project, subproject semantic Digital Twin 01MK20001M (https://knowledge4retail.org) as well as the German Research Foundation DFG, as part of Collaborative Research Center (SFB) 1320 "EASE -Everyday Activity Science and Engineering", University of Bremen (http://www.ease-crc.org/). The research was conducted in subproject R04 "Cognition-enabled execution of everyday actions".