In this paper, the Fine-Tuned Transformes for Ontology matching (Fine-TOM) matching system is presented along with the results it achieved during its rst participation in the Ontology Alingment Evaluation Initiative (OAEI) campaign (2021). The system uses the publicly available albert-base-v2 model, which has been ne-tuned with a training dataset that includes 20% of each reference alignment from the Anatomy, Conference, and Knowledge Graph track, as well as a wide variety of generated false examples. The model is then used by a separate matching pipeline which calculates a con dence score for each correspondence. In the submitted docker container, only the matching pipeline with an already ne-tuned model is included.3
1.1
Fine-Tuned Transformers for Ontology Matching (Fine-TOM) is a
transformerbased matching system. It consists of two separate pipelines, a pipeline for
generating training data and model training, and a matching pipeline which performs
the actual matching task. Both can be executed individually or in a row. Each
pipeline uses prede ned components, which are included in the Matching
Evaluation Toolkit (MELT) [
Commons License Attribution 4.0 International (CC BY 4.0). 4 https://dwslab.github.io/melt/matcher-packaging/web#
web-interface-http-matching-interface
problems is the use of transformers. The initial transformer was introduced by
Google in 2017 and uses a, so called, Self-Attention Architecture [
Fine-Tuning In order to achieve good results, a transformer needs to be
initially trained on a large amount of training data. This process is also called
pre-training. As it requires a vast amount of data as well as processing power to
pre-train a transformer model, most models are pre-trained on a speci c task,
like next sentence prediction and then uploaded to huggingface5 [
During the development of Fine-TOM, di erent BERT models were ne-tuned
and evaluated on the Anatomy [
The Fine-TOM matching system consists of two individual pipelines, as shown
in Figure 1:
{ A trainging pipeline, which handles the Fine-Tuning process of a transformer
and saves it to the disk
{ a matching pipeline, which will perform the actual matching task and is
based on the architecture presented in the TOM paper [
This architecture can also be used to run transformers with a zero shot approach, by only executing the matching pipeline with a pre-trained model. 5 https://huggingface.co
Training Pipeline The Training Pipeline, shown in Figure 2, consists of
several prede ned components of the MELT [
Matching Pipeline The Matching Pipeline, shown in Figure 3, also consists
of several prede ned components of the MELT Framework. As in the Training
Pipeline, a recall matcher is used as a starting point, thus, marking the
theoretically highest recall that can be achieved with this matching system. The resulting
alignment will then be processed by a con dence splitter, which will delete all
correspondences that are simple string matches and have a con dence level of
1.0, as well as their entities from the alignment returned by the recall matcher.
These correspondences are then saved temporarily into a separate alignment,
so they will not get reclassi ed by the transformer model. Then the cleaned up
alignment is passed on to a transformer lter, which will load the previously
ne-tuned transformer model from the disk and add another con dence score to
each correspondence in the alignment. In order to make use of this newly added
con dence level, and to eliminate correspondences the transformer classi ed as a
bad match by a low con dence score, a con dence lter is used. It will \cut o "
the alignment by a certain threshold which can be con gured. Fine-TOM uses
the same threshold of 0.8 as proposed by the TOM paper [
Con dence Splitter As described earlier, the con dence splitter takes an alignment as input and removes every correspondence with a con dence score of 1.0, as well as every other correspondence of the entities included in the removed correspondence. This is done in order to prevent a reclassi cation of these rather \save" matches by another component in the pipeline. Therefore, the con dence splitter is also able to add the alignment, which was saved during the splitting process, to an alignment that has been passed on to it as an input. Transformer Filter The transformer lter iterates over the alignment, which has been passed on to it as an input, and processes each correspondence individually by calling a separate Python server which is running locally in the background. This is needed because the transformer models themselves are implemented in Python, where as the matching components and pipeline is implemented in Java. Each pair of textual representations received by the Python server is processed by the selected model, which can either be loaded from the disk or it can be sourced from the huggingface library. This transformer model will then provide a con dence level, which is send back to the transformer lter class and added to the actual correspondence in the alignment, therefore classifying each correspondence.
Con dence Filter The con dence lter will exclude every correspondence with
a con dence score lower than a con gurable threshold. This is needed since the
transformer lter itself does not remove any correspondences from the alignment,
it just reclassi es them. Therefore, in order to exclude matches that have been
marked as a bad match by a low con dence, the con dence lter is needed.
Max Weight Bipartite Extractor The alignment generated by matching
components can include multiple correspondences for an ontology element.
However, the assumption was made earlier that the solution for the posed ontology
matching problem is of a one-to-one arity. Therefore, the alignment provided as
an input to the max weight bipartite extractor needs to be converted into an
alignment with a one-to-one arity. In order to do that, an e cient
implementation of the Hungarian method, known as Maximum Weight Bipartite Matching
(MWBM) [
This section discusses the results of Fine-TOM during the OAEI 2021 campaign.
Only the Anatomy [
The results6 of Fine-TOM on the Anatomy track are depicted in Table 1.
As shown, Fine-TOM was able to outpeform the OAEI StringEquiv matcher
in terms of recall and the f-measure, although its precision was lower. This
proves that the Fine-TOM matching system is able to nd matches that can
not be found by checking for string equivalence. However, if compared to the
TOM matching system, which is strongly related to Fine-TOM as they share a
similar architecture with regards to the matching pipeline, Fine-Tom achieved
slightly lower scores (~1-1.5%) for all measures shown. That is a rather
interesting result, as the transformers used in the TOM paper are not re-trained with
domain speci c data, nor were they pre-trained with data of an ontology
matching task. Nevertheless, TOM has one advantage: it uses the Sentence-BERT
transformer model paraphrase-TinyBERT-L6-v2 [
Precision Recall F-Measure StringEquiv 0.76 0.41 0.53 TOM 0.69 0.48 0.57
Fine-TOM 0.64 0.53 0.58
Table 2. Results on the Conference track according to the OAEI 2021 campaign 2.3
On Knowledge Graph, Fine-TOM was able to achieve slightly better results as the OAEI baseline, as shown in Table 3. In this paper, the Fine-TOM matching system has been presented. First, a new pipeline architecture that includes a dedicated training pipeline and a matching pipeline was introduced. This training pipeline rst generates a training set based on reference alignments and a high recall matcher, which is then used to re-train a selected model. The model is then injected in a so called matching pipeline. It then performs the actual matching process by using di erent lters. The results showed that transformers can improve the overall performance of matching systems in terms of recall and the f-measure. Besides that, the similar results of TOM and Fine-TOM proved that ne-tuning has a great impact on the performance of transformer models, since the model used by Fine-TOM has not been pre-trained for ontology matching or to nd semantic similarities between input sequences. Therefore, the presented approach promises a lot of potential for further increases in performance in the future, by using a di erent model, e.g. a Sentence-BERT model, or by improving or changing di erent pipeline components like the high recall matcher. In addition to that, this year's submission marks the rst participation for the Fine-TOM matching system in an OAEI campaign and the results reported are promising and motivate further research in the area of transformer-based ontology and instance matching.