Finding alignments between ontologies is a challenging and time-consuming task. When the aligned ontologies change, these alignments need to be changed as well. A recent approach to this problem proposes using embeddings as a representation for classifying changes. In this work, we compare embedding-based approaches to a neural network architecture built for node classi cation in knowledge graphs, namely relational graph convolutional networks. In our evaluation on two datasets from the biomedical domain, the best-performing embedding-based methods are RDF2Vec and TransE. The Graph convolution approach achieves similar results as the best-performing embedding based methods on a smaller dataset but outperforms all other approaches in standard classication metrics on a bigger dataset.
Ontologies that cover overlapping topics are often connected by so-called
ontology alignments, that describe the relation of concepts in di erent ontologies.
Finding these alignments is challenging and requires some degree of manual work,
which can be supported by approaches from the area of ontology matching.
Ontology matching has been an important area of semantic web research for years
[
Rule-based approaches like [
In this work, we examine the application of relational graph convolutional
networks [
The remainder of this paper is structured as follows: Section 2 presents foundations and related work and identi es a research gap. Our approach is discussed in Section 3. In Section 4, an evaluation of di erent classi cation approaches is presented. Results of this evaluation are discussed in Section 5. Section 6 closes this paper with a conclusion and an outlook. 2
Ontology Alignments (sometimes referred to as Ontology Mappings) are used
to connect concepts in di erent ontologies. When these ontologies change, an
adaption of these alignments is usually done manually. The problem of adapting
these alignments is referred to as the mapping adaption problem [
The mentioned approaches either rely on ontology mapping approaches
between versions or a set of hand-crafted rules. Therefore, adaptions are either
dependent on the quality of automatic ontology matching techniques or rely
on manual work by an expert for creating rules. To automate incremental
approaches using machine learning, a representation of nodes in the ontologies is
required. An approach with manual feature engineering for predicting changes
in ontologies in general was proposed by [
Over the last ten years, several approaches for embedding knowledge graphs
have been proposed. Some of these approaches are inspired by language
modeling techniques such as Word2Vec [
The aforementioned idea focuses on rst creating embeddings and then
learning the actual task at hand, namely, predicting which part of an ontology
alignment should change as a consequence of an ontology change. [
To our knowledge, the state of the art on ontology mapping adaption lacks an evaluation of graph network approaches and a comparison of di erent knowledge base embedding methods as a foundation for change classi cation tasks. This aspect is at the core of the research presented in this paper. 3
The classi cation task we try to solve in this work is the same as in [
For the approach with intermediate representation learning, an embedding is
learned by creating a single graph out of both ontologies and the mapping and
applying a knowledge graph embedding approach to this graph. As embedding
approaches we compare the established knowledge base completion approaches
TransE [
For the graph-network-based approach we apply relational graph
convolutional networks (RGCNs) [
X r2R j2Nir ci;r 1 Wr(l)h(l) + W0(l)hi(l)) j where h(l) is the activation in node i at layer l of the neural network, R is i the set of all relations, Nir is the set of all nodes connected to i by relation r, is an activation function, ci;r is a normalization constant and Wrl is a weight matrix in layer l for relation r. Hence, neural network activation travels through the graph. The weight matrices determine what kind of relations at each layer transport what kind of information. The depth of the network determines how many steps in the graph the activation is propagated. To create the sets R and Nir for all relations, we use a graph constructed from both ontologies and the alignment prior to the changes we want to classify. 4
The main research question of our evaluation is how a graph-convolution-based approach compares to a two-step approach with separated representation learning and classi cation. To evaluate this question, the impact of the graph embedding choice in relation to the performance of the approach when separating classi cation and representation learning also needs to be examined. To address these issues, we conducted a series of experiments that are described in the following subsections. 4.1
The dataset we use has been extracted from biomedical ontologies by [
These ontology versions are used as a silver standard, since ontology and mapping versions are not necessarily perfect but contain errors. For each changed entity close to an alignment statement in a new version of the ontology, we examine, if an alignment statement has been changed. If this is the case, we 1 https://dbs.uni-leipzig.de/de/research/projects/evolution of ontologies and mappings/ontology mapping adaption assign the changed entity the class Cchange, else, we assign the class Cnochange. Table 1 gives an overview of the datasets we use. For each pair of ontologies, we use two sets of changes: one set consisting of changes from 2009-2011, which is always used as the training set, and one set of changes from 2011-2012, which is used as the test set. In general, Cchange is always smaller than Cnochange. This e ect is present to a larger extent in SNOMED-FMA than in FMA-NCI.
FMA-NCI
SNOMED-FMA
For comparing di erent embedding methods as a part of the representation
learning process, we rst train embeddings and then compare the performance of
classi ers that use these embeddings as features on our dataset. To train embeddings
from the knowledge graph completion area, we use the OpenKE-Framework [
The classi cation itself was implemented using scikit-learn[
As evaluation metrics we use standard classi cation metrics, namely
precision, recall, f1-measure, accuracy, roc-auc score and average precision. Precision,
recall and f1-score are measured regarding the class Cchange, since classi
cation performance regarding this aspect is the most important for this task. We
evaluated each classi cation method on each embedding on both datasets,
FMANCI and SNOMED-FMA. The only exception is that we did not use
RDF2Vecembeddings on SNOMED-FMA. RDF2Vec uses a two-step approach: rst,
random walks through the graph are created (typically around 200 random walks
per entity). With the SNOMED-FMA graph containing 4 million entities and
more than 15 million triples, creating random walks for all entities would have
taken too long to be feasible given the hardware we had available. The main
issue here was the size and number of random walks.
2 http://data.dws.informatik.uni-mannheim.de/rdf2vec/code/
For the graph network based approach, we used the RGCN implementation
written by the authors of [
Results of this evaluation procedure for the dataset FMA-NCI are shown in Table 3. Underlined entries represent the best values for each metric. For readability purposes, we only show the best and the second-best results for every metric of the combinations of embedding methods and classi cation approaches. For each embedding method, we also show the two best combinations of classi cation method and embedding regarding f-measure. The graph embedding is nondeterministic. To account for graph embedding stability, we repeated the embedding step and evaluation. Since the di erences between di erent runs of the embedding models were insigni cant, we only report results of the rst experiment.
Of the embedding-based methods, RDF2Vec with Naive Bayes achieves the best performance comparing all metrics except precision. At least one of the other classi er/embedding combination achieves a similar performance given one speci c metric. The RGCN approach achieves very similar results to the RDF2Vec-based classi er.
Results for SNOMED-FMA are shown in Table 4. When observing the metrics for this dataset, it is important to reconsider the distribution of classes in the test set: only 2.6% of changes in the test set are in class Cchange. As already mentioned, RDF2Vec is missing from this evaluation, as creating the embeddings would have taken to much time on this dataset. On this dataset, no embedding based method is better than the other methods in nearly all metrics. The RGCN approach clearly outperforms the embedding-based methods on this 3 https://github.com/tkipf/relational-gcn acc dataset. Applying RGCNs results in signi cantly higher values in nearly all metrics except for accuracy, where the combination of TransE and linear regression obtains similar results, which is not a remarkable score given the distribution of classes in the test set.
To adapt our approach to the unbalanced dataset, we conducted two further experiments: We repeated our experiments on the SNOMED/FMA-Dataset using (1) oversampling and (2) undersampling of training data so that the training set was balanced in both experiments. Table 5 shows an excerpt of the results. We only show the results with highest f-measure for each embedding method out of both experiments. All best results of embedding-based methods stem from the experiment with oversampling, whereas the best RGCN-result stems from the undersampling experiment. Using oversampling, the RGCN over tted despite regularization and dropout. While the f-measure values look similar to the best results without oversampling or undersampling, as expected, recall is higher, whereas precision is lower. 5
When comparing the approaches that combine embeddings with traditional classi cation methods, we can observe that RDF2Vec in combination with Naive Bayes seems to show the best results where the computational e ort allows it. However, since generating random walks is very costly, using this approach is not possible for large knowledge graphs. Another embedding method that seems TransE TransH Distmult Complex RGCN
LR NB KNN RF MLP RF MLP RF RF
MLP TransE TransH Distmult Complex RGCN
RF CART MLP MLP acc prec embedding very promising is TransE. TransE-based classi cation is present in all top two performers for every metric. Hence, the choice of embedding makes a di erence regarding classi cation performance. RDF2Vec and TransE as representations perform best on the presented datasets.
Given the performance of the RGCN approach on both datasets we can see that RGCN can achieve similar or better results than a combination of embedding and classi cation approaches. On the rst dataset, its performance was similar to the best combined approach, on the second dataset it was signi cantly better. Another advantage of this end-to-end learning approach is that it is significantly faster than rst training the embeddings, especially for large databases. Training embeddings using OpenKE-Embeddings or RDF2Vec at this scale was slower then the complete RGCN classi cation by a factor of 20-50. On the larger dataset, embedding the graph using RDF2Vec even took to long to be actually usable. Therefore the answer to the research question is that a graph-networkbased approach can achieve similar or superior performance than a separated representation learning and classi cation approach.
In this paper, we presented two approaches to predicting, whether alignment statements need to change after an ontology update: a two-step approach that consists of representation learning as a rst step and established classi cation methods as a second step and an end-to-end approach that uses a neural network architecture specialized on node classi cation in relational graphs. In our evaluation, we could show that on the dataset we used, the best-performing representation learning approaches where RDF2Vec and TransE. The end-to-end learning approach was able to achieve similar results on one dataset and outperform the other approaches on a second, much larger dataset while in both cases needing much less computing time.
As future work, the integration of node and change features seems promising, since the current approach only uses the graph structure to reason about possible changes. Naturally, this can not be su cient information to determine everything about the nature of a change and how to react to it. To evaluate the capabilities of the presented approach in other domains besides biomedical ontologies, another dataset is required. To our knowledge, the dataset used in this paper is the only dataset currently available for ontology mapping adaption. Hence, a new dataset needs to be built that contains knowledge from a di erent domain.