Frameworks for aligning embeddings of text and embeddings of knowledge graphs (KG) have been used for generating mappings for test-to-text alignment and KG-to-KG alignment, but little has been done for alignment between these two domains. In this dissertation proposal, I aim to create a framework for KG-to-text alignment that utilizes little to no training data to learn these correspondences. Additionally, motivated by the semantic geometries of these embedding spaces, I propose a new line of research into generating explicit embeddings of triples from a knowledge graph.
Knowledge graphs (KGs) and ontologies form the computational backbone of the modern Semantic Web, curated by taxonomists and ontologists in conjunction with domain subject matter experts. Collaboration between these parties is a bottleneck in large-scale organizations due to coordination of people and sourcing of relevant information and terminologies for ontologists to massage into an enterprise-wide standard. This bottleneck is felt both in developing ontological terminologies and populating the knowledge graph with facts and assertions about the domain being described.
Industrial terminologies are buried deep in policy documents or technical white papers, making the job of the ontologist one of synthesizing these documents into a conscise, machine-readable set of interlinked terminologies (T-Box). For large-scale knowledge graphs, such as those leveraged in popular search engines, the information scales past terminologies and additionally covers facts or assertions (A-Box) about the objects described in the graph. Validating the accuracy of assertions in the knowledge graph is critical for auditing the trustworthiness of those claims, which is especially relevant in highly regulated industries such as banking or pharmatecuticals. As facts (in the form of s, p, o triples) are added to a knowledge graph, either through automated methods such as link prediction techniques or human generated annotations, there is an additional opportunity to enrich the knowledge graph with metadata about these triples, such as a source of evidence from a text document. Developing methodologies for linking the T-Box and A-Box to textual evidence will provide a set of tools to allow ontologists and knowledge graph developers to expedite their work while ensuring the highest degree of accuracy and auditability, and thus is the focus of this work.
For example, an ontologist working in the financial services domain may wish to develop a standardized definition of a fixed-float interest rate swap. They may begin by inheriting the structure of previously defined terminologies, namely those related to interest rates and swaps, deriving the triples f ixed f loat interest rate swap, has type, swap contract , f ixed f loat interest rate swap, has leg, f ixed interest rate and f ixed f loat interest rate swap, has leg, f loating interest rate .
However, without a background in financial terminology, they may miss the fact that a fixed-float interest rate swap is more commonly refered to as a vanilla interest rate swap. This fact can be inferenced from a variety of textual sources, such as a sentence like 'A vanilla interest rate swap allows two counterparties to hedge against interest rate volatility by trading a floating rate for a fixed rate.', given the set of seed triples the ontologist has developed, helping to expand the coverage of the knowledge graph.
Of critical importance in the development of a text to KG methodology is its ability to rapidly adapt to new source ontologies and data domains. Additionally, such a system should not be bottlenecked by reliance on abundant training data, a point of failure for many ML projects. This motivates the use of unsupervised techniques for knowledge representation, and I propose an exploration of cross-domain optimal transport between embeddings of natural language and embeddings of a knowledge graph. Given this motivation, I present the following formalism.
Suppose S = {s 1 , s 2 , . . . , s n } is a set of sentences and T = {t 1 , t 2 , . . . , t n } is a set of knowledge graph triples, each triple of the form t i = s, p, o . Next, define two functions f and g that create low-dimensional (latent) representations of each sentence and triple, respectively, such that f (s i ) = e si ∈ R n (the source space) and g(t j ) = e tj ∈ R m (the target space) where n |S|, m |T |. In order to link triples to their most relevant sentences, both for validating terminology and assertions, we seek a mapping function Ψ : R n → R m to transport one set to another with minimal loss, i.e. Ψ (T ) ≈ S, such that for new triples t i / ∈ T we can find a supporting sentence representation Ψ (t i ) ≈ s j / ∈ S in a large-scale text corpus C that reflects the same semantic information. Learning such a Ψ should not be taxed by reliance on an abundance of paired labeled data points, i.e. a set of labels L = {(t k , s k ), . . . , (t l , s l )}. Instead, I assume no such set exists at training time, framing the learning of Ψ as an unsupervised task.
The desire to limit the reliance on paired labeled data points helps inform the potential choices of Ψ . Specifically, methods from optimal transport (OT) theory lend themselves well to this problem by exploiting the structure of each embedding space using pairwise distance metrics rather than relying on data with paired labels. Instead, couplings of objects from each respective space are inferred through probabilistic transport maps, shifting the focus from gathering labeled sentence-triple pairs for supervised learning to refining the representations of these objects in their respective latent spaces. Optimal transport also provides a probabilistic framework for mapping assignments; the Kantorivitch relaxation of Monge's original statement admits a solution where the mass at any point in the source space can be dispatched to several locations in the target space [18], fitting the problem setting as a single KG triple may admit infinitely innumerable sentence representations. The probabilistic and rigorous mathematical approach of OT add to the understandability of results in opposition to black-box models such as generative adversarial networks (GANs).
One drawback of OT techniques is the requirement of a cost matrix defined between the spaces X and Y. Defining such a cost matrix requires labeled data points beween the two spaces, although a weaker assumption can be used to avoid this by defining two inter-domain distance matrices D ∈ R n×n and D ∈ R m×m . Such matrices can then be aligned through the following formulation of the Gromov-Wasserstein problem
.
Using this formalism, the problem of interest can then be framed as follows:
Problem Statement: What are the optimal choices of embedding functions f and g to establish distance matrices D and D such that 1. for pairs of triples (t i , t i ) ∈ T that contain some notion of semantic similarity,
) is minimized 2. while simultaneously minimizing D j,j = d(g(s j ), g(s j )) for semantically similar sentences (s j , s j ) ∈ S for some distance metric d, thus allowing optimal couplings between t i and s i to be established via the Gromov-Wasserstein distance?
Prior research in this area shows that optimal transport of embedding spaces has been successful within a given data domain (i.e. unsupervised alignment of word embeddings [2], unsupervised alignment of knowledge graph entities [17]). My research questions seek to extend these methodologies to the cross-domain task in order to align embeddings of knowledge graph triples with embeddings of semantically related sentences.
Is there an accurate, unsupervised technique for aligning a set of knowledge graph triples to a set of semantically similar sentences?
H-I: Optimal transport methods provide a mathematical framework for unsupervised alignment based on intra-domain pairwise similarities. Successful application of OT is dependent on how similarities of like-objects are represented in each respective space.
To accomplish this, properties of each embedding space that make them useful for such alignment must first be established. In my prior work [9], these properties are explored for sentence embeddings while keeping the knowledge graph embeddings fixed as the concatenation of head, tail and relation vectors generated by TransE. It remains to be seen how changing the structure of knowledge graph embeddings, via changing the algorithm selection (such as using more expressive models like ComplEx [20] or ConvE [7]), incorporating additional information such as literals [11] or changing the way entity and relation vectors are combined to represent a triple, help or hurt the ability to generate high-quality alignments, motivating the next research question.
RQ-II: How can current knowledge graph embedding methods be extended past representing entities and relations as separate objects and instead focus on embedding triples as the target objects?
As the majority of current methods focus almost exclusively on the link prediction task, these methods may not be well-suited for establishing embeddings of triples, leading to the following hypothesis.
H-II: Triple embeddings built from aggregations of entity and relation embeddings do not sufficiently encode the underlying semantics of such triples.
Building upon the work of [8], treating triples as walks on the knowledge graph and weighting the strength of each relationship may help to create a semantic embedding space that will assist in alignment. The following section details how I will approach measuring the amount of semantic information captured by these methods.
Motivated by work on word embedding regularities [14], I wish to probe both sentence and KG embedding spaces generated by a variety of embedding algorithms and measure the degree to which they exhibit an underlying structure that can be leveraged for aligning these resources. Lurking beneath the above research questions is the fuzzily-defined notion of "semantic similarity," but metrics exist to make this quantification concrete. These metrics are used to define how well semantic similarity is encoded in the latent representations of both triples and sentences, and they are important to capture with the goal of defining optimal pair-wise distance matrices D and D in mind. To formalize the notion of structure, I introduce a definition of clusterability, following the work of [1]. For some dataset X ∈ R n , a description of the clusterability of X is a function c : X → v where v ∈ R is a real value. Here, v is a measure of how strong a clustering presence is in the underlying set X.
To test the clusterability hypothesis, I use the spatial histogram (SpatHist) approach to measure the clusterability of each space [1]. The SpatHist approach compares the data binned in all d-dimensions to samples randomly generated in the same d-dimensional space. As many of these bins may end up empty in high-dimensional embedding spaces, I perform principal component analysis (PCA) to project down to the two most informative dimensions, split the data into n equal-width bins, and compute the empirical joint probability mass function (EPMF). The same is then done for 500 sets of uniformly generated points with the same feature dimensionality, and the differences are compared using the Kullback-Leibler (KL) divergence -higher KL divergence indicates more clusterability. I report the mean and standard deviation of each of these experiments as my final estimates of clusterability. Additionally, I apply the Hopkins test of uniformity [12]. As the Hopkins test statistic tends to zero, the underlying data exhibits less uniformity, indicating that clustering may be a good way of exploring the data in an unsupervised way. As the test statistic increases, the data tends to be more uniformly distributed, exhibiting less of the structure I seek to exploit.
For evaluating the quality of learned alignments, I follow in the tradition of knowledge graph embedding literature [21] and evaluate these results for the Hits@5 and Hits@10 metrics. Analyzing the results of the top 5 and top 10 closest matches allows for a nearest neighborhood analysis of each aligned embbedding instance, helping to pinpoint areas for future improvement, such as the mitigation of the influence of hubs [13] .
To establish a baseline for this task, prior work [9] tested the ability for a lowcapacity linear model to learn a mapping between sentence and knowledge graph representations. The purpose of this work is in evaluating sentence representations, measuring the extent to which they are able to create structure in the low-dimensional embedding spaces by evaluating how well they cluster together around their semantic content, in this case the expression of a particular relationship. Findings on the clustering capacity of a selection of sentence embedding methods are reproduced below.
The results demonstrate the dramatic differences in the efficacy of sentence embedding methodologies. In particular, the geometrically motivated GEM algorithm vastly outperforms all others in terms of semantic clusterability, especially those using more complex deep neural models. In addition, the GEM algorithm outperforms all others in terms of Hits@5 and Hits@10 when performing a simple linear map for alignment (Linear@5,10), and all alignments show improvement when replacing the linear alignment with optimal transport techniques (OT@5,10). Utilizing these results gives a clue as to how to build knowledge graph triple embeddings: by focusing on the novelty of each predicate and the entities involved, they can be "pushed" into respective areas of the low-dimensional embedding space, leading to increased cluster cohesion and higher within cluster semantic similarity. Additional insights and recommended improvements are presented in [9].
Dim. Linear@5 Linear@10 OT@5 OT@
Regression models for word-to-word alignment were first proposed by [15] as a means of capturing geometric patterns between embeddings across embedding spaces. Inconsistencies in this approach were noted by [22] who in turn modified the regression process to add unit length normalization and constrain map to be orthogonal. Applications of pre-processing and orthogonal constraints spurred further research into ways to manipulate the source and target embedding spaces to further express their geometric structures in [3] and [4]. Building 'pseudodictionaries' as a means of reducing the amount of necessary training data is suggested in [19]. The work of [16] further explores iterative learning, alternating between supervised alignment and unsupervised distribution matching, as well as introducing novel metrics to assess the orthogonality assumptions used in supervised approaches. A key approach for unsupervised learning is described in [6] where the authors propose leveraging an adversarial learning paradigm. While the adversarial method directly leverages word frequencies, an alternative unsupervised method in [5] captures these patterns by analyzing the similarity distributions of the word vectors themselves. Using the Gromov-Wasserstein distance, [2] transform the alignment problem to one of finding an optimal transport from source X to target Z.
The majority of research in the area of knowledge graph embeddings focus on one specific task, namely knowledge graph completion, which seeks to make predictions of the following form: given s, p, ? , make the best prediction for an object o such that the triple is a valid one in the context of the greater graph. The state-of-the-art methods in this space leverage knowledge graph embeddings, low-dimensional representations of the entities and relations between them as vectors. The majority of research in this area focuses on representing the nodes, or entities, of the graph, with considerable less emphasis on how the relations are represented, and representations of the entire s, p, o triple are rarely considered.
Recent research into representing the entire triple is presented in [8], yet the results are limited to explorations in clustering and recommendation systems. This work can be extended to further use cases and eventually tied in with alignment methods to link knowledge graphs to text documents.
Based on the initial results presented herein, is clearly room for improvement in methodologies for representation learning of triples in a knowledge graph. Additionally, the alignment of cross-domain representations -those spanning both text and knowledge graph, is not currently well explored. Exploration in this area can provide a brigde between the two respective data realms and provide tooling for unsupervised automation of ontology and knowledge graph development. Future work will involve measuring the clusterability of multiple existing knowledge graph embedding algorithms, evaluating their efficacy in alignments with sentence embeddings, and proposing new, semantically grounded approaches to embedding triples as singular objects.