<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Hybrid Structured and Similarity Queries over Wikidata plus Embeddings with Kypher-V</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Hans Chalupsky</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Pedro Szekely</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Information Sciences Institute, University of Southern California</institution>
          ,
          <country country="US">USA</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>ylabel 'Zeno of</institution>
        </aff>
      </contrib-group>
      <abstract>
        <p>Augmenting large knowledge graphs such as Wikidata with unstructured sources represented by embedding vectors is important for a number of applications, such as similarity-based search, recommendation systems, question answering, entity linking, etc. However, building such hybrid systems today requires ad hoc combinations of disparate technologies such as structured query languages with approximate nearest neighbor search, since there are no truly hybrid query systems available aimed at simultaneously querying large knowledge graphs and vector data. Moreover, similarity search over large vector data is very resource and main-memory intensive requiring high-end compute servers with large amounts of RAM. This paper introduces Kypher-V, an extension to KGTK's Kypher query language and processor, that allows users to augment Wikidata and other KGs with large-scale vector data and eficiently query it through a hybrid query system using a structured query language based on Cypher. We show that Kypher-V can eficiently store, index and query 40M 1024-d vectors and efectively filter and join results with Wikidata-size datasets on a laptop.</p>
      </abstract>
      <kwd-group>
        <kwd>eol&gt;Wikidata</kwd>
        <kwd>Knowledge Graphs</kwd>
        <kwd>Similarity</kwd>
        <kwd>Embeddings</kwd>
        <kwd>KGTK</kwd>
        <kwd>Kypher</kwd>
        <kwd>Cypher</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>
        Many important tasks and applications such as search, recommendation systems, question
answering, entity linking, link-prediction, etc. require the efective combination of large
amounts of structured and unstructured knowledge. Wikidata [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ] is a popular,
broadcoverage knowledge graph (KG) often used for such applications, which contains circa
100 million entities described with over 1.4 billion statements.1 Despite its impressive
size, however, Wikidata is still incomplete and additional unstructured sources such as
text or images might need to be added to fill gaps and support a particular task.
      </p>
      <p>
        A common approach for representing unstructured data is the use of high-dimensional
embedding vectors that cluster objects by their latent semantic dimensions, and that
provide a uniform representation to support operations such as semantic similarity or
neural net-based machine learning. Recent work on knowledge graph embeddings such
as, for example, TransE [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ] or ComplEx [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ] allows similar vector representations and
operations to also be applied directly to structured data such as knowledge graphs.
      </p>
      <p>Hybrid systems that combine structured and unstructured data have the promise to
amplify the best features of both while compensating for their weaknesses. For example,
structured data is richly described and provides useful constraints for high precision
results, but it lacks a notion of similarity and is generally incomplete. Unstructured
data is often very large-scale, high-coverage, captures most relevant semantic dimensions
automatically and can be used to query by example, but it lacks constraints and becomes
less precise with increasing number of objects.</p>
      <p>
        Efective combination of structured and unstructured sources is still an active research
area and generally requires extensive experimentation. Unfortunately, building such
hybrid systems today is dificult and requires ad hoc combinations of disparate technologies
such as SPARQL or SQL with approximate nearest neighbor search (ANNS) systems
such as Faiss [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ] and others, since there are no truly hybrid query systems available aimed
at simultaneously querying large knowledge graphs and vector data (the KGTK similarity
service [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ] is an example). Similarity search over large vector data is also very resource
and main-memory intensive requiring high-end compute servers with large amounts of
RAM, providing a high barrier to entry for researchers.
      </p>
      <p>
        To support this emerging new area of large KG + vector-based systems, this paper
introduces Kypher-V, which is an extension to Kypher, the query language and processor
of the KGTK Knowledge Graph Toolkit [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ], which allows creating personalized variants
of Wikidata on a laptop, and enables running analytic queries faster than a Wikidata
SPARQL endpoint [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ]. The new Kypher-V allows users to augment Wikidata and other
KGs with large-scale vector data and eficiently query it through a hybrid query system
using a familiar structured query language based on Cypher. Both Kypher and Kypher-V
use the KGTK representation and can therefore be used not just on Wikidata but also
RDF KGs, for example, DBpedia [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ], and others. We show that Kypher-V can eficiently
store, index and query 40M 1024-d vectors and efectively filter and join results with
Wikidata-size datasets on a standard laptop.
      </p>
      <p>The rest of the paper is structured as follows. Section 2 describes the KGTK toolkit
and gives a brief introduction to the Kypher query language. Section 3 describes vector
representation and storage used by Kypher-V, Section 4 introduces the new types of
query patterns supported by Kypher-V, Section 5 presents related work, and Section 6
presents conclusions, discussion of the results and directions for future work.</p>
    </sec>
    <sec id="sec-2">
      <title>2. Background</title>
      <p>
        The Knowledge Graph Toolkit (KGTK) [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ] is a comprehensive framework for the creation
and exploitation of large hyper-relational KGs, designed for ease of use, scalability,
and speed. KGTK represents KGs in tab-separated (TSV) files with four columns:
edge-identifier, head, edge-label, and tail. All KGTK commands consume and produce
KGs represented in this format, so they can be composed into pipelines to perform
complex transformations on KGs. KGTK provides a suite of import commands to import
Wikidata, RDF and popular graph representations into the KGTK format. A rich
collection of transformation commands make it easy to clean, union, filter, and sort KGs;
graph combination commands support eficient intersection, subtraction, and joining
of large KGs; graph analytics commands support scalable computation of centrality
metrics such as PageRank, degrees, connected components and shortest paths; advanced
commands support lexicalization of graph nodes, and computation of multiple variants
of text and graph embeddings over the whole graph. In addition, a suite of export
commands supports the transformation of KGTK KGs into commonly used formats,
including the Wikidata JSON format, RDF triples, JSON documents for ElasticSearch
indexing and graph-tool.2 Finally, KGTK allows browsing KGs in a UI using a variant of
SQID;3 and includes a development environment using Jupyter notebooks that provides
seamless integration with Python Pandas. KGTK can process Wikidata-sized KGs, with
billions of edges, on a laptop.
      </p>
      <p>
        Kypher (kgtk query) is one of 55 commands available in KGTK. Kypher stands for
KGTK Cypher [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ]. Cypher [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ] is a declarative graph query language originally developed
at Neo4j. OpenCypher4 is a corresponding open-source development efort for Cypher
which forms the basis of the new Graph Query Language (GCL).5 We chose Cypher since
its ASCII-art pattern language makes it easy even for novices to express complex queries
over graph data.
      </p>
      <p>
        Kypher adopts many aspects of Cypher’s query language, but has some important
diferences. Most importantly, KGTK and therefore Kypher use a hyper-relational
quadbased data model that explicitly represents edges as first-class objects which is more
general than the labeled property graph (LPG) model used by Cypher. For this reason
Cypher and Neo4j cannot be used to express queries over all aspects of KGTK data (see
[
        <xref ref-type="bibr" rid="ref10">10</xref>
        ] for a discussion of a unifying data model similar to KGTK’s that can encompass
LPG’s, RDF and RDF* ). Other diferences are that Kypher only implements a subset of
the Cypher commands (for example, no update commands) and has some diferences in
syntax, for example, to support naming and querying over multiple graphs.
      </p>
      <p>
        To implement Kypher queries, we translate them into SQL and execute them on
SQLite, a lightweight file-based SQL database. Kypher queries are designed to look and
feel very similar to other file-based KGTK commands. They take tabular file data as
input and produce tabular data as output. There are no servers and accounts to set up,
and the user does not need to know that there is in fact a database used underneath to
implement the queries. A cache mechanism makes multiple queries over the same KGTK
ifles very eficient. Kypher has been successfully tested on Wikidata-scale graphs with
1.5B edges where queries executing on a standard laptop run in milliseconds to minutes
depending on selectivity and result sizes [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ]. Additional information about Kypher and
its capabilities can be found online. 6
      </p>
      <sec id="sec-2-1">
        <title>2https://graph-tool.skewed.de</title>
        <p>3https://sqid.toolforge.org/
4https://opencypher.org/
5https://www.gqlstandards.org/home
6https://kgtk.readthedocs.io/en/latest/transform/query/#overview</p>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>3. Vector representation and storage</title>
      <p>Kypher-V represents vectors as special kinds of literals that are linked to KG objects via
edges in KGTK format. For example, the following edge in KGTK’s TSV format with
edge ID e4925 associates node Q40 (or Austria) with a text embedding vector as node2
of the edge (there is nothing special about this representation pattern, any scheme can
be used that associates a node or edge ID with a vector):</p>
      <p>When vector data is queried, it is imported and cached if necessary just like any other
KGTK data processed by Kypher. The main diference is that for faster query processing,
vector literals get translated into binary format first before they are stored in the Kypher
graph cache based on an indexing directive. Various up-front transformations such as
data type conversions, normalization, storage type, etc. can also be controlled here.
Finally, vector files are treated as large contiguous and homogeneous arrays, so that
vectors can be referenced and accessed eficiently by vector table row ID instead of having
to use an index.</p>
      <p>
        This scheme stores vectors on disk and brings them into main memory on demand
during a query just as any other KG data. Since vector literals can be quite large, it is
important that these accesses are fast and minimized to maximize query speed. For more
limited applications, it is possible to precompute similarities and only store the results
instead of the full vector data (see, for example, the RDFSim browser [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ]). However, we
chose to explicitly store vectors to allow more open-ended queries, comparison against
dynamic data (e.g., vectors from user queries), as well as vector management in general
which can facilitate downstream applications such as machine learning.
      </p>
      <p>Kypher-V can store and simultaneously query any number of diferent types of
embedding vectors. The only requirement is that when two vectors are compared in a query,
they must have the same dimensionality and should come from the same vector set, since
it does not make sense, for example, to compute a similarity between a ComplEx and
TransE vector. Just as the core Kypher system it is based on, Kypher-V is optimized
for read-only processing. Nevertheless, updates and dynamic inputs can be handled but
should be limited to small-batch data for best performance.</p>
    </sec>
    <sec id="sec-4">
      <title>4. Similarity queries</title>
      <p>The following types of vector queries are supported by Kypher-V:
1. Vector computation: given one or more vectors accessed in a query, compute a
function based on those vectors. Currently supported vector functions are inner
product, cosine similarity and Euclidean distance.
2. Vector search: traditional similarity search that finds the top-  similar vectors
and associated nodes for a given input. Only cosine similarity is supported at the
moment.
3. Vector join: a generalized form of vector search that filters input vectors and output
vectors based on arbitrary Kypher query patterns.
4. Multi-vector computations: a planned extension is to be able to aggregate
similarities from multiple vector types, as needed, for example, by the topsim computation
of KGTK’s similarity service.</p>
      <p>Below we will illustrate the diferent types of queries through examples and describe
any pertinent system aspects needed to support them.</p>
      <sec id="sec-4-1">
        <title>4.1. Experimental setup</title>
        <p>All queries shown below use one or more of the embedding sets listed in Table 1 as an
input, together with a custom KGTK version of Wikidata that has all scholarly articles
removed, which leaves about 55M entities and 500M statements. Provided query and
command execution times are intended to be rough indicators of performance, but they
were not determined in a rigorously controlled environment. Queries were measured on
the command line, which includes overhead from loading KGTK and needed ANNS index
ifles. These overheads can be eliminated by using Kypher-V through its Python API. All
queries and commands were executed on a Lenovo Thinkpad W541 laptop with 8 cores,
30GB of RAM and a 2TB SSD disk drive.</p>
      </sec>
      <sec id="sec-4-2">
        <title>4.2. Vector computation queries</title>
        <p>sim
0.84781</p>
        <p>We can also use these computations to compute top- similarities for a node in a
bruteforce way. For example, in Figure 2 we find the top-5 similar philosophers for Socrates
by first finding all nodes y with an occupation link (P106) to philosopher (Q4964182),
then computing the similarity between the respective vectors and the vector for Socrates,
and finally sorting and reporting the top-5 results.</p>
        <p>!$kypher -i short_abstracts_textemb -i labels -i claims
--match 'emb: (x:Q913)-[]-&gt;(xv), (y)-[]-&gt;(yv),
claims: (y)-[:P106]-&gt;(:Q4964182),
labels: (x)-[]-&gt;(xl), (y)-[]-&gt;(yl)'
--return 'xl as xlabel, yl as ylabel, kvec_cos_sim(xv, yv) as sim'
--order 'sim desc' --limit 5</p>
        <p>In this example enumerating a full set and then computing similarity for each of its
members was feasible and still fast (about 1 second), since the set was small (only about
10,000 elements in our main Wikidata claims table). On a larger set, for example all
humans in Wikidata with about 9 million elements, this becomes very ineficient (4
minutes) and we will use nearest neighbor indexing described below to speed things up.
Note also that diferent embeddings will yield diferent results with similarity values that
are not directly comparable across embedding types. For example, here we show the
top-5 similars based on 100-d ComplEx graph embeddings:
!$kypher -i complexemb -i labels -i claims
--match 'emb: (x:Q913)-[]-&gt;(xv), (y)-[]-&gt;(yv),
claims: (y)-[:P106]-&gt;(:Q4964182),
labels: (x)-[]-&gt;(xl), (y)-[]-&gt;(yl)'
--return 'xl as xlabel, yl as ylabel, kvec_cos_sim(xv, yv) as sim'
--order 'sim desc' --limit 5
sim
1.00000
0.82710
0.82499
0.82058
0.81729</p>
      </sec>
      <sec id="sec-4-3">
        <title>4.3. Vector search queries</title>
        <p>Figure 4 shows a vector search operation expressed in Kypher-V. It finds the top- 
similar Wikidata nodes for Socrates (Q913) according to Wikipedia short abstract text
embeddings. Vector search is expressed via a virtual graph edge kvec_topk_cos_sim
which is implemented via an SQLite virtual table (a custom computation that can
generate multiple rows and behaves like a regular table). Virtual edges are a Kypher-V
extension to Cypher that take a number of user-specified input parameters expressed
in Cypher property syntax. The parameters here are  controlling how many results to
return, and  which specifies how many quantizer cells should be searched (more
on that below). Note that the results here are slightly diferent than on a similar query
above, since we do not require that the results have philosopher as their occupation which
accounts for the diferences.</p>
        <p>!$kypher -i short_abstracts_textemb -i labels
--match 'emb: (x:Q913)-[]-&gt;(xv),</p>
        <p>(xv)-[r:kvec_topk_cos_sim {k: 5, nprobe: 4}]-&gt;(y),
labels: (x)-[]-&gt;(xl), (y)-[]-&gt;(yl)'
--return 'xl as xlabel, yl as ylabel, r.similarity as sim'</p>
      </sec>
      <sec id="sec-4-4">
        <title>4.4. ANNS index creation and use in Kypher-V</title>
        <p>
          In a naive brute force implementation as the one from Figure 2, the query in Figure 4
would have to scan all Wikidata nodes to find those with highest similarity (about 52
million for the version we are using). Instead, we are using a custom approximate nearest
neighbor search (ANNS) index based on Faiss [
          <xref ref-type="bibr" rid="ref4">4</xref>
          ] which executes this query in less than
a second.
        </p>
        <p>Generic Faiss assumes ANNS indexes and their vectors are created and loaded into
RAM for best search performance, and building and using of disk-based indexes has
relatively limited support. Unfortunately, this requires very large amounts of RAM and
leads to high startup cost due to index loading time, which would require Kypher-V to
use a server architecture to amortize the cost. Instead, we are using a diferent approach
that leverages the underlying database to store quantization information so relevant
vectors can be loaded lazily from disk as needed by a query instead of eagerly at system
startup. Index creation and use proceed in the following steps:
1. K-means clustering of the vectors of a vector set using the standard Faiss API.</p>
        <p>We sample the input to limit main memory use and clustering run time based
on user-provided resource limits. Our currently largest vector set that requires
approximately 180GB to store can be clustered with 25GB of RAM. The resulting
cluster centroids form the basis of the quantization cells (q-cells) vectors will be
assigned to in the next step. Centroids are saved to a file as a standard Faiss flat
ifle index which is generally small and can be loaded quickly if needed for a vector
search query.
2. Quantization assigns vectors to the q-cell centroid they are closest to and writes the
respective q-cell IDs as a column to the vector table. After creating an appropriate
database index, this allows us to quickly find all vectors of a q-cell using a database
query.
3. Sorting of the vector table by q-cell ID improves locality, which is important, since
now vectors of the same q-cell can all be retrieved from a single or just a few disk
pages instead of having to read them from all over a large file.
4. At query time we search an input vector  for the top  q-cell centroids
closest to it using the Faiss search API. Thus,  controls the depth of the
similarity search trading of recall for speed.
5. Once the set of  relevant q-cells has been determined, we bring in all the
relevant vectors of these q-cells from disk and use a dynamically constructed Faiss
search index to find the top-  matches for input  .</p>
        <p>Importing large vector sets and creating nearest neighbor (NN) indexes for them is
expensive. K-Means clustering complexity is  ( ) where   -dimensional vectors are
clustered into  centroids in  iterations. Kypher-V allows the user to control some of
these parameters such as the number of centroids or iterations to keep run time in check.
Nevertheless, the size of the dataset and the dimensionality of the vectors are two of
the biggest determiners of overall run time which cannot easily be changed. In Table 1
we show data import and indexing times for diferent embedding sets using diferent
parameter settings. All imports and clusterings could be handled by a laptop, even
though some of the run times were quite large.</p>
        <p>Dataset
Wikidata KG
Wikidata KG
Wikidata KG
Short abstracts</p>
        <p>Type
ComplEx
TransE</p>
        <p>Text
Text</p>
        <p>N
53M
53M
39M
5.9M</p>
      </sec>
      <sec id="sec-4-5">
        <title>4.5. Vector join queries</title>
        <p>Vector joins generalize basic vector search by adding additional input and output
constraints to restrict the results generated by the search. In general, a similarity join
between sets  and  joins two elements   ∈ ,   ∈  if   is in the top- similars of
  , or if the similarity  (  ,   ) ≥  where  is some minimal threshold similarity. For
example, the query in Figure 5 joins Socrates with the set of all humans to find the top-5
similar matches. This query executes in less than a second now which is a greater than
250x speedup compared to the brute-force version of this query described in Section 4.</p>
        <p>The implementation described here is somewhat imperfect, since it requires a large
enough  to be specified to select enough matches that satisfy the join constraints and the
!$kypher -i short_abstracts_textemb -i labels -i claims
--match 'emb: (x:Q913)-[]-&gt;(xv),</p>
        <p>(xv)-[r:kvec_topk_cos_sim {k: 100, nprobe: 4}]-&gt;(y),
claims: (y)-[:P31]-&gt;(:Q5),
labels: (x)-[]-&gt;(xl), (y)-[]-&gt;(yl)'
--return 'xl as xlabel, yl as ylabel, r.similarity as sim'
--limit 5
sim
1.00000
0.93465
0.93445
0.93040
0.92928
limit clause. An experimental version of Kypher-V eliminating this limitation through
dynamic scaling of  is available on the KGTK repository. For certain types of queries
this can also be compensated through reformulation, such as using a similarity threshold
and/or aggregation functions to get the desired result. For example, Figure 6 shows how
we can find the top-1 human match for an input set of three philosophers.
!$kypher -i short_abstracts_textemb -i labels -i claims
--match 'emb: (x)-[]-&gt;(xv),</p>
        <p>(xv)-[r:kvec_topk_cos_sim {k: 100, nprobe: 4}]-&gt;(y),
claims: (y)-[:P31]-&gt;(:Q5),
labels: (x)-[]-&gt;(xl), (y)-[]-&gt;(yl)'
--where 'x in ["Q859", "Q868", "Q913"] and x != y and r.similarity &gt;= 0.9'
--return 'xl as xlabel, max(r.similarity) as sim, yl as ylabel'</p>
      </sec>
    </sec>
    <sec id="sec-5">
      <title>5. Related work</title>
      <p>
        There are only very few systems available today that support hybrid querying of vector
data combined with structured constraints. Alibaba AnalyticDB-V [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ], Milvus [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ],
Vearch [14] and ElasticSearch [15] all support some form of hybrid querying, but only
AnalyticDB-V (a very large scale proprietary system) integrates this capability in a
standard full-function query language such as SQL. To the best of our knowledge, there is
no system available today that supports this for a query language focused on knowledge
graphs as we do with Kypher-V. Systems such as Wembedder [16] and KGvec2go [17]
provide embedding vectors over Wikidata and other KGs through Web APIs, but do
not have a general hybrid query capability. However, the embedding vectors from
these systems could be used by Kypher-V. RDFSim [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ] is a system that supports
similarity-based browsing in KGs, but through precomputed similarity sets only.
      </p>
    </sec>
    <sec id="sec-6">
      <title>6. Discussion and Conclusions</title>
      <p>
        The main objective of KGTK, Kypher and Kypher-V is to democratize the exploitation
of Wikidata so that anyone with modest computing resources can take advantage of the
vast amounts of knowledge present in it. See [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ] for a set of use cases that use large
portions of Wikidata to distill new knowledge.
      </p>
      <p>Kypher-V extends these capabilities to store, index and query large sets of embedding
vectors in a fully integrated way, allowing users to eficiently combine large amounts of
structured and unstructured data while still preserving the ability to use the system on
modest resources such as a laptop.</p>
      <p>The examples presented in this paper are intended to give an initial overview of
the capabilities of the new system. Execution times are mostly anecdotal and a more
systematic evaluation of system performance has to be reserved for a future paper.
Kypher-V is still under active development and has not yet been merged with the main
branch of KGTK, but early snapshots are available on the KGTK GitHub repository.7</p>
      <p>We do not make any claims about the quality and usefulness of the diferent types
of embeddings used in the examples. They are all flawed in diferent ways, and in fact
combining multiple embedding types as supported by Kypher-V has the potential to
improve overall system performance. Kypher-V is completely agnostic to the types of
embeddings used, and any reasonable embeddings that can be linked to the nodes and
edges of a KG can be queried by it.</p>
      <p>The contribution of this paper is to show that KGTK, Kypher and Kypher-V are
efective tools for complex analytic use cases. The paper demonstrates that Kypher-V
supports a variety of hybrid queries that are dificult to implement with existing tooling.
Kypher-V allows researchers and developers to investigate use cases on their own laptop,
exploring extensions of Wikidata with large amounts of vector data, using minimal setup,
modest resources and a simple query language.</p>
    </sec>
    <sec id="sec-7">
      <title>Acknowledgments</title>
      <p>This material is based on research sponsored by Air Force Research Laboratory under
agreement number FA8750-20-2-10002. The U.S. Government is authorized to reproduce
and distribute reprints for Governmental purposes notwithstanding any copyright notation
thereon. The views and conclusions contained herein are those of the authors and should
not be interpreted as necessarily representing the oficial policies or endorsements, either
expressed or implied, of Air Force Research Laboratory or the U.S. Government.</p>
      <sec id="sec-7-1">
        <title>7https://github.com/usc-isi-i2/kgtk/tree/feature/kypher/embeddings</title>
        <p>of the 2021 International Conference on Management of Data, SIGMOD ’21, ACM,
2021, p. 2614–2627.
[14] J. Li, H.-F. Liu, C. Gui, J. Chen, Z. Ni, N. Wang, Y. Chen, The design and
implementation of a real time visual search system on JD E-commerce platform,
Proceedings of the 19th International Middleware Conference Industry (2018).
[15] ElasticSearch, Elasticsearch: Open source, distributed, restful search engine,
https://github.com/elastic/elasticsearch, 2020.
[16] F. Nielsen, Wembedder: Wikidata entity embedding web service, 2017. URL: https:
//arxiv.org/abs/1710.04099.
[17] J. Portisch, M. Hladik, H. Paulheim, KGvec2go – knowledge graph embeddings as a
service, 2020. URL: https://arxiv.org/abs/2003.05809.</p>
      </sec>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>D.</given-names>
            <surname>Vrandečić</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Krötzsch</surname>
          </string-name>
          ,
          <article-title>Wikidata: A free collaborative knowledgebase</article-title>
          ,
          <source>Communications of the ACM</source>
          <volume>57</volume>
          (
          <year>2014</year>
          )
          <fpage>78</fpage>
          -
          <lpage>85</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>A.</given-names>
            <surname>Bordes</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Usunier</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Garcia-Durán</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Weston</surname>
          </string-name>
          ,
          <string-name>
            <given-names>O.</given-names>
            <surname>Yakhnenko</surname>
          </string-name>
          ,
          <article-title>Translating embeddings for modeling multi-relational data</article-title>
          ,
          <source>in: Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, NIPS'13</source>
          ,
          <year>2013</year>
          , p.
          <fpage>2787</fpage>
          -
          <lpage>2795</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>T.</given-names>
            <surname>Trouillon</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Welbl</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Riedel</surname>
          </string-name>
          , E. Gaussier, G. Bouchard,
          <article-title>Complex embeddings for simple link prediction</article-title>
          ,
          <source>in: Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML'16</source>
          ,
          <year>2016</year>
          , p.
          <fpage>2071</fpage>
          -
          <lpage>2080</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>J.</given-names>
            <surname>Johnson</surname>
          </string-name>
          , M. Douze,
          <string-name>
            <given-names>H.</given-names>
            <surname>Jégou</surname>
          </string-name>
          ,
          <article-title>Billion-scale similarity search with GPUs</article-title>
          ,
          <source>IEEE Transactions on Big Data</source>
          <volume>7</volume>
          (
          <year>2021</year>
          )
          <fpage>535</fpage>
          -
          <lpage>547</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>F.</given-names>
            <surname>Ilievski</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P. A.</given-names>
            <surname>Szekely</surname>
          </string-name>
          ,
          <string-name>
            <given-names>G.</given-names>
            <surname>Satyukov</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Singh</surname>
          </string-name>
          ,
          <article-title>User-friendly comparison of similarity algorithms on Wikidata</article-title>
          ,
          <source>ArXiv abs/2108</source>
          .05410 (
          <year>2021</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>F.</given-names>
            <surname>Ilievski</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Garijo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Chalupsky</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N. T.</given-names>
            <surname>Divvala</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Yao</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Rogers</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Li</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Liu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Singh</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Schwabe</surname>
          </string-name>
          , P. Szekely,
          <article-title>KGTK: a toolkit for large knowledge graph manipulation and analysis</article-title>
          , in: International Semantic Web Conference, Springer,
          <year>2020</year>
          , pp.
          <fpage>278</fpage>
          -
          <lpage>293</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>H.</given-names>
            <surname>Chalupsky</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Szekely</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Ilievski</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Garijo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Shenoy</surname>
          </string-name>
          ,
          <article-title>Creating and querying personalized versions of Wikidata on a laptop</article-title>
          , in: L.
          <article-title>-</article-title>
          <string-name>
            <surname>A. Kafee</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          <string-name>
            <surname>Razniewski</surname>
            ,
            <given-names>A</given-names>
          </string-name>
          . Hogan (Eds.),
          <source>Proceedings of the 2nd Wikidata Workshop (Wikidata</source>
          <year>2021</year>
          )
          <article-title>co-located with the 20th International Semantic Web Conference (ISWC 2021), CEUR-WS</article-title>
          .org,
          <year>2021</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>S.</given-names>
            <surname>Auer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Bizer</surname>
          </string-name>
          , G. Kobilarov,
          <string-name>
            <given-names>J.</given-names>
            <surname>Lehmann</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Cyganiak</surname>
          </string-name>
          ,
          <string-name>
            <surname>Z. Ives,</surname>
          </string-name>
          <article-title>DBpedia: A nucleus for a web of open data</article-title>
          ,
          <source>in: The Semantic Web</source>
          , Springer,
          <year>2007</year>
          , pp.
          <fpage>722</fpage>
          -
          <lpage>735</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>N.</given-names>
            <surname>Francis</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Green</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Guagliardo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Libkin</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Lindaaker</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            <surname>Marsault</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Plantikow</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Rydberg</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Selmer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Taylor</surname>
          </string-name>
          , Cypher:
          <article-title>An evolving query language for property graphs</article-title>
          ,
          <source>in: Proceedings of the 2018 International Conference on Management of Data</source>
          ,
          <year>2018</year>
          , pp.
          <fpage>1433</fpage>
          -
          <lpage>1445</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>R.</given-names>
            <surname>Angles</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Hogan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>O.</given-names>
            <surname>Lassila</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Rojas</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Schwabe</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Szekely</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Vrgoč</surname>
          </string-name>
          ,
          <article-title>Multilayer graphs: A unified data model for graph databases</article-title>
          ,
          <source>in: Proceedings of the 5th ACM SIGMOD Joint International Workshop on Graph Data Management Experiences &amp; Systems (GRADES)</source>
          and
          <article-title>Network Data Analytics (NDA)</article-title>
          , ACM,
          <year>2022</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>M.</given-names>
            <surname>Chatzakis</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Mountantonakis</surname>
          </string-name>
          ,
          <string-name>
            <surname>Y. Tzitzikas,</surname>
          </string-name>
          <article-title>RDFsim: Similarity-based browsing over DBpedia using embeddings</article-title>
          ,
          <source>Information</source>
          <volume>12</volume>
          (
          <year>2021</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <given-names>C.</given-names>
            <surname>Wei</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Wu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Lou</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Zhan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Li</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Cai</surname>
          </string-name>
          , AnalyticDB-V:
          <article-title>A hybrid analytical engine towards query fusion for structured and unstructured data</article-title>
          ,
          <source>Proc. VLDB Endow</source>
          .
          <volume>13</volume>
          (
          <year>2020</year>
          )
          <fpage>3152</fpage>
          -
          <lpage>3165</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <given-names>J.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Yi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Guo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Jin</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Xu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Li</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Guo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Li</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Xu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Yu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Yuan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Zou</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Long</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Cai</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Z.</given-names>
            <surname>Li</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Z.</given-names>
            <surname>Zhang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Mo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Gu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Jiang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Wei</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Xie</surname>
          </string-name>
          ,
          <article-title>Milvus: A purpose-built vector data management system</article-title>
          ,
          <source>in: Proceedings</source>
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>