Integration of symbolic representations with multimodal data is an important problem in multiple domains. The Hyperknowledge Framework (HKF) is a multimodal knowledge representation framework that allows users to integrate non-graph data to knowledge graphs. In this particular demo, we will show how HKF API can be used to more easily integrate and consume raw data fragments in knowledge graphs.
model that is independent of an specific media type. It can also complement document
and information description models, such as [
The General Fragment Model defines a formal model for information reference. It describes a conceptual structure that can be instantiated to create resolvable reference names for parts of information artifacts. An information artifact is a codification of some propositional content that realized by some physical or virtual object. Examples are images, text, drawings, sound files, sensor readings, databases and ontologies.
GFM establishes that fragments of information artifacts are specified by anchors. An anchor on an information object o 2 O is defined by an indexer function f (o; d) : O
D ! O0 that maps an arbitrary token d 2 D to a set of parts O0 of O. The tokens in D can be any other information artifacts, especially vectors, dictionaries or strings. For instance, given a text document e, the text fragment e0 between characters 10 and 20 can denoted by the application of an indexer function subtext to the target e and with the argument token [10; 20]. In this case, the function application subtext(e; [10; 20]) is said to be an anchor on e and it is a reference (or a name) to e0.
Anchors can be composed by other fragments. For example, consider an indexer function rect that extract sub-images from figures and an indexer channel that takes an specific color channel of an image. Considering an image img, we can define the fragment
channel(rect(img; [10; 10; 20; 20]); "blue") which takes the blue channel of a 20-pixels square positioned at coordinates 10 10 on img. We can go even further by composing the same indexer function multiple times: xywh(channel(xywh(img; [10; 10; 20; 20]); "blue"); [2; 2; 4; 4]) which denotes a 4-pixel square fragment of the blue channel. Specific implementations might decide on the applicability of a given indexer function on a specific type of information artifact. john depictedBy
rect({x: 10,y: 10, w: 30, h: 230})
picture1
rect({x: 10,y: 34, w: 30, h: 230})
depictedBy
mary
The HypeKnowledge Base (HKB), is a knowledge graph database part of HKF [
All nodes might have associated anchors. Anchors represent a fragment of the inhering node. In our new model, an anchor is identified by what we call Fragment Identifiers, which allows representation of GFM constructs. In the following, we briefly describe (a) the FI language and (b) the anchor resolver API for this service.
Fragment identifiers (FI) is concrete language for entity naming that implements GFM. It allows definition of fragmentation operations based on a JavaScript-like syntax. The basic form is artifact.indexer(token). Artifacts can be any node identifier in HK. Indexers are usually function names. Tokens can be lists or json-like objects. For instance, given a content node identified by document, our previous example of subtext anchor can be specified as:
document:subtext(fstart : 10; end : 20g) FIs allow for anchor composition as well. Considering an information artifact figure representing an image, the following anchor composition is a valid FI: picture1:rect(fx : 10; y : 10; w : 20; h : 20g):channel(fc : "blue"g) HKB currently allows users to specify and resolve FI anchors on content nodes. In practice, these features allow HKB data processing capabilities within the graph database itself. As mentioned before, content nodes are nodes carrying some raw data. Their fragments are represented as FI anchor strings associated to them. Fig. 1 depicts a simple 3-node graph where two KG nodes representing John and Mary are related to their respective depictions in fragments of picture1 specified as FI fragments.
The semantics of these anchors is given by a FI resolution API (Fig. 2). The base API is a REST endpoint implemented within the knowledge graph engine. Currently, we have API bindings for JavaScript and Python, the latter of which we show in this demo.
Indexer Function
Repository Rest API
Resolver
Rect function
Object Storage resolve('picture1.rect({x: 10,y: 34, w: 30, h: 230})')
resolve('picture1') return blob lookupResolver('rect') return rect func. return image blob fetch('picture1') return image blob resolve(picture1, 'rect({x: 10,y: 34, w: 30, h: 230})')
return image blob
The FI resolution endpoint is able to take full FIs (i.e., artifact and multiple anchors) and recursively resolve them to produce the actual fragment of the content nodes in the knowledge graph. Each indexer function is coded as simple pluggable modules1 using a standard API provided by HKB. Fig. 2 shows the resolution of one of the anchors in Fig. 1. 4
In our demonstration, we plan to demonstrate creation and access to media fragments in a simple example using our Python API and a Jupyter notebook: 1. Demonstration of basic Hyperknowledge constructs in KES (HKF’s KG UI); 2. Ingestion of text and image files as content nodes in a knowledge graph via the
Python API; 3. Creation and resolution of simple and composed fragments on the ingested files; 4. Demonstration of the creation of these operations in KES; 5. Association of the created fragments with a simple domain ontology. 1 These modules can be implemented based on existing information and metadata extraction toolkits, such as Apache Tika (https://tika.apache.org) and other components of Apache UIMA (http://uima.apache.org).