The knowledge graph completion task has gained a lot of attention in recent years, especially with the use of machine learning (ML). However, most of the work has focused on the structure of the graph while ignoring the data it describes. In this demo, we present an approach that does the opposite: it leverages the multimodal data described by a knowledge graph for its completion. We use IBM's Hyperlinked Knowledge Graph framework, which allows nodes of the graph to carry arbitrary data content. This content is processed at query time by user-defined functions which are triggered by rules and whose output is used to decide the materialization of new links, completing the original graph. To demonstrate the approach, we use ML models to classify images of paintings and decide the materialization of links describing their semantics. DEMO
Knowledge Graphs (KGs) are widely used to enrich applications with factual knowledge about
objects in the world. In traditional KG systems, only symbolic content (e.g., concepts, instances)
is represented, while non-symbolic content (e.g., images, videos, scripts) and its integration
with the knowledge stored in the KG have to be handled by external systems. In contrast, in
Multimodal KGs (MMKGs) [
Because of their generality and scope, real-world KGs and MMKGs are large and often
incomplete [
Much of the research in KG completion has focused on predicting what is missing based solely
on the structure of the graph, usually represented in the form of low dimensional spaces (i.e.,
embedding space) [
Our proposal difers from previous approaches in two key aspects: (1) It relies on an extensible rule-based mechanism that completes the graph with new links considering the non-symbolic and symbolic data. (2) It considers human expertise in the loop since the inference of new links is triggered by the execution of user-defined queries. Thus, our work provides an interactive and on-demand MMKG completion solution that can be easily extended by the user.
We instantiate our proposal in IBM’s Hyperlinked Knowledge Graph (HKG) [
The HKG is composed of traditional graph components such as nodes (i.e., circles in Fig. 1) and links but it also ofers specialized components that are particularly interesting for our proposal. We leverage two such specialized components to support extensible query-driven MMKG completion: possibility links and possibility resolvers. A possibility link represents an unasserted link, i.e., a yet-to-be-fulfilled possibility in the KG, which is ignored during regular query evaluation. For example, a possibility link can be used to represent the possibility that a painting , initially without authorship, was painted by Salvador Dalí. Meanwhile, a possibility resolver specifies how the assertion of a possibility link should be resolved. For example, we can use a possibility resolver to indicate how to decide whether was painted by Salvador Dalí. The possibility resolver is represented by an n-ary link whose components act as parameters to solve a given possibility. Its main component is an executable node, i.e., nodes containing a wrapper code that invokes a function to be executed to perform a task on the graph. These nodes can evaluate whether possibility links should be materialized as asserted links.
To illustrate the usage of these components in our approach, consider Figure 1. In this example, we have a set of images of paintings, an ontology of painters and painting styles, and a set of executable nodes containing functions that calls ML models to classify images by painter or painting style. Here we want to be able to answer queries such as “Give me all impressionist paintings” or “List all paintings created by a Spanish painter”.
To answer these queries by leveraging the multimodal knowledge available in this scenario (ontology, images, and ML models), we represent the images as content nodes (i.e., nodes that stand for non-symbolic content) and link them, using possibility links, to instances of concepts of the ontology, such as painting, painting style, and painters. We define executable nodes to invoke an ML model passing the non-symbolic data of a content node as a parameter (e.g., an image).1 Finally, we define possibility resolvers to evaluate the possibility of materializing the possibility links by executing the adequate ML model for each relation defined in the ontology (e.g., ‘hasCreator’ corresponds to the painter of an image, therefore the ML model that identifies the painter would be used to evaluate possibility links of this relation).
A possibility resolver can be formally defined as a statement of the form R( , , , ) establishing that entity is a procedure that decides whether an instance or subclass of is related via predicate to an instance or subclass of . The actual representation of this statement depends on the KG system being used. 2 In HKG, this is represented as a 4-ary relationship among , , , and , as illustrated in Figure 1. Thus, a rule of the form () ∧ ( ) ∧
R( , , , ) →
P (, , ) is used to materialize (to convert from a possibility link to a regular link) the possibility links from the declared possibility resolvers. This rule states that if is an instance or subclass of , is an instance or subclass of , and resolves whether holds from an to a , then there is a possibility link from to with label (predicate) which can be resolved by .
During query evaluation, the links inferred by the above rule are used to decide whether a given relationship exists in the graph. For instance, suppose that the query evaluator needs to decide whether (, ) , that is, whether there is a link with predicate from to . If the link does not exist in the graph and was not inferred, then the result is false. However, if the link exists or was inferred there are two cases. Either the link is a regular link (not a possibility) and in this case, the result is true. Or the link is a possibility link in which case the evaluator calls the associated possibility resolver with arguments , , and and the result of this call is the result of the test. If is deterministic (gives the same result for the same input) then the evaluator can save the trouble of going through this process again by materializing the link.
The entire process starts from a user-determined query execution – hence, human-in-the-loop
and query-driven. HKG provides a query language, the Hyperknowledge Query Language
(HyQL) [
The goal of this demo is to show HKG features that support extensible query-driven MMKG completion through KES and HyQL. The demo uses the Kaggle’s Best Artworks of All Time dataset4, two ML models trained over this dataset, and a simple ontology containing painters (Salvador Dalí, Alfred Sisley, etc.), nationalities (Spain, UK, etc.), art movements (Surrealism, Impressionism, Expressionism, etc.), and themes (landscape, portrait, etc.). The images, ML models, and ontology are used to demonstrate how the HKG can be used to infer links from multimodal data at query time while enabling rich semantic queries.
Video 1: https://ibm.box.com/v/iswc2022keg1 – MMKG Representation with HKG and KES. Video 2: https://ibm.box.com/v/iswc2022keg2 – MMKG Completion with HyQL and KES.