One of the strongest signals for automated matching of ontologies and knowledge graphs are the textual descriptions of the concepts. The methods that are typically applied (such as character- or token-based comparisons) are relatively simple, and therefore do not capture the actual meaning of the texts. With the rise of transformer-based language models, text comparison based on meaning (rather than lexical features) is possible. In this paper, we model the ontology matching task as classi cation problem and present approaches based on transformer models. We further provide an easy to use implementation in the MELT framework which is suited for ontology and knowledge graph matching. We show that a transformer-based lter helps to choose the correct correspondences given a high-recall alignment and already achieves a good result with simple alignment post-processing methods.3
Ontology Matching is the non-trivial task of nding correspondences between classes, properties, and instances of two or more ontologies. The match operation can be seen as a function f which returns an alignment A given two ontologies O1 and O2: f (O1; O2) = A. The alignment is a set of correspondences in the form he1; e2; ri where e1 2 O1, e2 2 O2, and r is some relation which holds between the two concepts; in this paper r is always equivalence ( ).
Multiple techniques exist to perform the matching operation in an automated
manner [
Commons License Attribution 4.0 International (CC BY 4.0).
NLP area and transformer models achieved remarkable results on tasks such as
machine translation [
In this paper, we bring transformers to the ontology matching task. Our
contributions are twofold: Firstly, we present a transformer extension to the
Matching and EvaLuation Toolkit (MELT), which allows users to easily exploit
state-of-the-art pre-trained transformer models like BERT [
Transformers are deep learning architectures which combine stacked encoder
layers with a self-attention [
Applications of transformers for the pure ontology matching task are less
frequent compared to the entity matching domain. Wu et al. [
Since transformer models are language models, it is a hard requirement that the elements in the ontology have labels or descriptions. We propose to model the match operation as an unbalanced binary classi cation problem where the classi er receives a correspondence and predicts whether this correspondence is correct or not. Eventually, only correct correspondences are kept. The match operation can be (i) complete or (ii) partial. In a complete matching setting, each element e1i 2 O1 respectively e2i 2 O2 needs a textual representation. The latter can be obtained, for instance, by concatenating the URI fragment and all annotation properties. The transformer model then classi es each element in the Cartesian product of the ontologies to be matched. Since the set of comparisons grows quadratically for the complete matching case, and matching with transformers can be computationally intensive, it is also possible to use a candidate generator which reduces the total number of comparisons. This candidate generator can be regarded as matching system which returns an alignment AC . In the partial case, we generate textual representations only for candidates in the alignment (c 2 AC ) and perform a classi cation operation only for the correspondences c 2 AC . Therefore, focus of the candidate generator should be recall since the generator determines the theoretically largest attainable recall score of the system, i.e., for the nal alignment A, A AC holds. This approach can also be seen as an matching repair technique.
MELT 4 [
In this work, we extend the ML component of MELT so that transformer
operations can be called directly from the Java code. Therefore, we use the
Hugging Face transformers library [
In order to serialize textual descriptions, MELT o ers various classes extending the TextExtractor interface. For any given resource, those return extracted 4 https://github.com/dwslab/melt/ 5 https://dwslab.github.io/melt/matcher-packaging/web text as a set of strings. They do not normalize the text because this is a post processing step. They merely select speci c literals, URI fragments, etc. In our experiments, we use three of those extractors. They are ordered by the number of strings which are returned (most strings to fewest strings)6:
TextExtractorSet returns the highest amount of literals because it retrieves all literals where the URI fragement of the property is either a label, name, comment, description, or abstract. This includes also rdfs:label and rdfs:comment. Furthermore, the properties prefLabel, altLabel, and hiddenLabel from the skos vocabulary are included, as well as the longest literal (based on the lexical representation of it). Additionally, all properties which are de ned as owl:AnnotationProperty are followed in a recursive manner in case the object is not a label but a resource. In such a case, all annotation properties of this resource are added. The extractor reduces the potentially large set of literals by comparing the normalized texts and only returns the ones which are not identical (note here that the original literals are returned, not the normalized ones).
The TextExtractorShortAndLongTexts reduces the set of literals further by checking if a normalized literal is fully contained in another literal. In this case, the literal is not returned. This is only applied within the two groups of long and short texts to extract not only a long abstract but also a short label. Label-like properties are regarded as short texts, while comment/description properties are regarded as long texts.
The TextExtractorForTransformers extracts the smallest number of literals (out of the text extractors presented here) by returning exclusively labels that are not contained in other labels (without distinguishing in long and short texts). This results in reducing the set of strings even more because labels which appear in a comment are also not returned. 4.3
In order to allow for re-usable matching code, MELT allows to chain matchers to build a dedicated matching pipeline for various problems. In such a pipeline, each matcher receives the alignment of the previous component together with the ontologies that are to be matched (and optionally con guration parameters).
MELT di erentiates between matchers and lters. A lter is a component which does not add new correspondences to the alignment but instead further processes the given alignment by (1) removing correspondences and/or (2) adding new con dence / feature weights to existing correspondences.
Since the transformer evaluation of the Cartesian product of descriptions is not a scalable option for most test cases, MELT o ers the usage of transformers as a lter through class TransformersFilter. The training process is implemented using TensorFlow and PyTorch, the user can decide which implementation shall be used. Therefore, we recommend a transformer-based matching 6 A more detailed overview can be found in the user guide: https://dwslab.github.io/melt/matcher-development/matching-with-jena# textextractors
rdfs:label :C12666 rdfs:label
rdfs:label
rdfs:label :00116 Adrenal Glands Adrenal Glands Suprarenal gland Suprarenal gland
Adrenal Gland Suprarenal gland Adrenal Gland Suprarenal gland ? ? ? ? max Adrenal Glands Suprarenal gland
Adrenal Gland Suprarenal gland ? pipeline as shown in Figure 1: In a rst step, we use a matcher that generates a recall-oriented alignment. The transformer lter will then use the correspondences in the latter alignment to calculate the estimated similarity. The similarity is calculated by rst serializing the textual descriptions of each correspondence to a CSV le. Textual descriptions are obtained by a TextExtractor. In case there are multiple textual descriptions available, two modes are implemented: (1) A multi-text option (depicted in Figure 2), which serializes all combinations of the individual texts; eventually, the maximum similarity will be used. (2) A single-text option which concatenates all textual elements.
After serializing the texts to be compared to a le, the ML Python server is started in the background and predicts the likelihood of a match given the textual description of each correspondence. It is optionally also possible to lter the alignment, for instance, by using a threshold or by reducing the alignment to a one-to-one alignment if applicable.
The MELT extension presented in this paper is publicly available in the main branch7 together with a reference implementation8 that was used to run the experiments. The new features are documented in the MELT user guide9. 4.4
In order to run a training process, such as ne-tuning a transformer, data is required for the training step. Positive correspondences can be obtained either from the reference10 or from a high-precision matching system. However, neg7 https://github.com/dwslab/melt/ 8 https://github.com/dwslab/melt/tree/master/examples/transformers 9 https://dwslab.github.io/melt/ 10 Note that convenience methods to do so exist in MELT such as generateTrackWithSampledReferenceAlignment(Track track, double fraction) of class TrackRepository. ative examples are also required. Multiple strategies can be applied here. For example, negatives can be generated randomly using an absolute number of negatives (class AddNegativesRandomlyAbsolute) or a relative share of negatives to be generated (class AddNegativesRandomlyShare). If the gold standard is not known, it is also possible to exploit the one-to-one assumption and add random correspondences involving elements that already appear in the positive set of correspondences (class AddNegativesRandomlyOneOneAssumption). The new extension to the MELT ML module contains multiple out-of-the box strategies that are already implemented as matching components which can be used within a matching pipeline. All of them implement the new interface AddNegatives. Since multiple avors can be thought of (e.g. generating type homogeneous or type heterogeneous correspondences), a negatives generator can be easily written from scratch or customized for speci c purposes. MELT o ers some helper classes to do so such as RandomSampleOntModel which can be used to sample elements from ontologies.
Since the (partial) reference alignments of OAEI tasks are known and the one-to-one assumption holds, we propose to generate negatives using the same high-recall matcher that is also used in the matching pipeline and to apply the one-to-one sampling strategy: Given the reference and the alignment produced by some high-recall matcher, we determine the wrong correspondences as correspondences where only one element is found in the reference (but not the complete correspondence) and add them to the training set. This is implemented in class AddNegativesViaMatcher. Note that for this approach, the reference alignment does not have to be complete. One advantage here is that the characteristics of training and test set are very similar (such as the share of positives and negatives). This process is visualized in Figure 3. 4.5
A transformer model can be used as is (particularly, if the application is equal or very similar to its training objective) or be ne-tuned for a speci c task. The default transformer training objectives are not suitable for the task of ontology matching.
Therefore, a pre-trained model needs to be ne-tuned. Once a training alignment is available, class TransformersFineTuner can be used to train and persist a model. Like the TransformersFilter, the TransformersFineTuner is a matching component that can be used in a matching pipeline.11 Such a training pipeline is visualized in the orange (upper) part of Figure 3: A high-recall matcher can be used to generate candidates and negatives can be generated using a sampled reference (or a reference-like alignment). Repeated calls of the match method will extend the number of training candidates, the actual training is performed when calling method finetuneModel. This setup allows to train 11 Note that this pipeline can only be used for training and model serialization. For the application of the model within a matching pipeline, TransformersFilter must be used. one model given multiple test cases. The implementation allows, for instance, to train a ne-tuned model per test case, per track, or a global model for multiple tracks. In this paper, we ne-tune the model per track to cover their individual characteristics. 4.6
By default, the ne-tuning of the transformer models is executed with the
standard training parameters such as a xed number of epochs (3), a learning rate
of 5 10 5 etc. (those default values originate from the transformers library12).
In hyperparameter optimization, a simple grid search is often applied. But such
a tuning method has some disadvantages: (1) each run (parameter combination)
needs to be executed until the end to analyze the performance (2) all
combinations need to be executed (no information about previous runs are taken into
account). Bayesian Optimization [
Due to the fact that training of transformer based models is rather slow, even
more sophisticated methods need to be applied. One of them is population based
12 https://huggingface.co/transformers/main_classes/trainer.html#
trainingarguments
training [
The seed is also optimized because di erent initializations of the classi cation head of the model can also improve the nal metric. The reason behind this is that most models are trained on the masked language modeling task and need a classi cation layer (usually a linear layer on top of the pooled output) to create the nal prediction. This linear layer is initialized with di erent random weights.
As described above, the hyperparameters can also be changed during training. The following parameters are mutated: weight decay: uniform distribution between 0.0 and 0.3; learning rate, and batch size as de ned above.
The metric which is optimized can be chosen from the following KPIs: loss (of the model), accuracy, F1, recall, precision, or AUC. The last one is the default because in a later step in the matching pipeline, the con dence of a correspondence is important for ltering or selection. AUC optimizes this con dence such that all negatives have a low value and all positives a high one. Furthermore, it allows to decide which model is better even if they have the same F-measure. The hyperparameter tuning can be easily performed in MELT with class TransformersFineTunerHpSearch. It has the same interface as the netuning class but when calling the finetuneModel method, the hyperparameter search is started. 5 5.1
In order to show the e ectiveness of transformers for matching in MELT, we
performed multiple experiments { each focuses on a di erent aspect: (1) We
evaluate an o -the-shelf transformer model in a zero-shot setting for three OAEI
tracks: Anatomy, Conference, and Knowledge Graph(KG) [
We use the following transformer models from the huggingface repository:
bert-base-cased [
The matching pipeline consists of 4 components: (1) high-recall matcher, (2) transformer lter, (3) con dence threshold cut-o lter, and (4) max weight bipartite partitioning lter.
The high-recall matcher adds candidates with overlapping tokens, the transformer lter assigns a con dence to each candidate found in the previous step. An optimal threshold is determined to lter out non-matches. The threshold is calculated not with the complete gold standard but merely with the correspondences that were sampled for the training step. Therefore, the ConfidenceFinder class has been extended to work also with incomplete gold standards. Lastly, the max weight bipartite partitioning lter enforces a one-to-one alignment. 5.2
In the following, the results to all experiments are presented. The rst part covers
the zero-shot approach as well as the ne-tuning. Afterwards, we report on the
impact of di erent sampling sizes and the results of the hyperparameter search.
Zero-shot and Fine-tuning The results of the zero-shot and ne-tuning
experiments are depicted in Table 1. The SimpleString baseline is a simple matcher
which we use as a baseline. The high-recall matcher is the one which is used as a
rst step in the pipeline in the zero-shot as well as in the ne-tuning setup. This
also means that the recall value of this matcher is automatically an upper bound
for the recall because the transformer-based ltering will not add any new
correspondences. For the zero-shot case where an already ne-tuned model is applied
directly (in this case no reference sampling is necessary), we selected a dataset
which is rather close to our setup. Due to the fact that paraphrasing is very
similar to the task of nding same concepts, the Microsoft Research Paraphrase
Corpus [
Baseline SHiimghplReSetcrailnlg Zero-Shot (bmerrtp-bc-atsuen-ceads)ed 0.650 0.548 0.594 0.531 0.817 0.644 0.739 0.714 0.726 (Fpinere-TTruancekd) rabolebbreter-rbtt-aabs-abes-aecs-aevs2ed 000...867164278 000...343996781 000...554378307 000...877512456 000...876248599 000...778303279 000...649804701 000...637688589 000...638795639 Table 1. Results of non- ne-tuned and ne-tuned transformer models (multi-text) with 20% sampling from the reference alignment. As per OAEI customs, we report micro average scores for the conference and macro average scores for the KG track. the KG track, bert performed much better. One reason why di erent models perform better is the di erent characteristics of the labels and comments.
For Conference and Anatomy, the TextExtractorSet is used with the multitext setup to generate many classi cation examples whereas for the KG track the TextExtractorForTransformers is used to extract less literals which are then concatenated together to create only one classi cation example for each correspondence.
Sampling Rates We analyzed the performance of the best model on anatomy (albert) using varying sampling rates s 2 [0:1; 0:2; 0:3; 0:4; 0:5; 0:6] from the reference. The results are presented in Figure 4. Interestingly, fairly good performance can be achieved with very low sampling rates (10% and 20% respectively). Intuitively, the overall performance tends to increase with an increasing share of samples from the reference. Hyperparameter Tuning The hyperparameter tuning was executed for the anatomy track and the albert-base-v2 model. The given search space in Section 4.6 is used and overall 12 trials are sampled from it which is also the amount of the model population. The search needs 45 minutes to run in parallel on 4 GPUs (NVIDIA GeForce GTX 1080 Ti). All other settings are the same as in the normal ne-tuning setup (thus, the numbers are comparable). With PBT, the precision could be improved by 0.02 to 0.874 whereas the recall is only a bit higher (0.832). In terms of F-Measure, the hyperparameter tuning additionally gives an improvement of 0.013 (eventually leading to an F1 of 0.852). 6
In this paper, we introduced a new matching component to the MELT framework which is based on transformer models. It allows to extract a textual description of the resource with so called text extractors and provides an easy option to apply and ne-tune transformer based models. We propose and evaluate an exemplary matching pipeline for transformer training and application. We hope that our implementation bene ts the ontology matching community and enables other researchers to further explore this topic.
In addition, we performed four experiments which demonstrate the capabilities of the newly implemented component. We showed that a transformer-based lter can improve a given alignment by providing a con dence for each correspondence based on its textual description. Moreover, we presented a sophisticated approach for hyperparameter tuning and showed that improvements can be achieved when optimizing the model hyperparameters.
Since the ne-tuning obviously has a large impact on the results, we will conduct further experiments on that step in the future. Examples include netuning with text corpora from the domain of matching (e.g., biomedical texts for the anatomy track), or transfer learning setups where ne-tuning is conducted based on matching gold standards from other domains.
Moreover, we plan to extend the implementation to also cover components
that do not require any input alignment. These would also include matches
which would not be possible with string comparison based systems. The library
Sentence Transformers [
Acknowledgements The authors acknowledge support by the state of BadenWurttemberg through bwHPC.