In this paper, we propose TEXTO, a TEXT-based Ontology matching system. This matcher leverages the rich semantic information of classes available in most ontologies by a combination of a pre-trained word embedding model and a pre-trained language model. Its performance is evaluated on the datasets of the OAEI Common Knowledge Graphs Track, augmented with the description of each class, and a new dataset based on the refreshed alignment of Schema.org and Wikidata. Our results demonstrate that TEXTO outperforms all state-of-art matchers in terms of precision, recall and F1 score. In particular, we show that almost perfect class alignment can be achieved using textual content only, excluding any structural information like the graph of classes or the instances of each class.
CEUR ceur-ws.org
Ontology matching [
String Matching
No
Text Embedding
Candidate Selection
MaxWeight Matching
Confidence Threshold
Yes Similarity matrix = 1 < 1 Class pairs 1 ↔ 1 2 ↔ 2 …
Step 1: String Matching. In the first step, the label of each class is preprocessed through
tokenization (white space and camel case tokenization, splitting of multi-word expressions such
as televisionnetwork) and stop word removal. After text preprocessing, any class pair , sharing
exactly the same labels is assigned a similarity score of 1; the other classes are passed to Step 2.
Step 2: Text Embedding. In Step 2, a similarity score in [
each word of the label (mean pooling). Second, a vector representation of the class description is obtained through Sentence-BERT (SBERT), a language model trained by contrastive learning for Semantic Textual Similarity tasks. Here, the pre-trained model all-MPNet-base-v24 is used, which provides the best quality among all SBERT models available on Hugging Face5.
Denoting by label, label the vector representations of the labels of classes , , and desc, desc
the vector representations of the descriptions of classes , , the similarity score between and is
computed as the weighted sum:
= cos( label, label) + (1 − ) cos( desc, desc)
(1)
where ∈ [
Step 3: Candidate Selection. In this step, top- candidates are selected in terms of similarity.
Specifically, a class pair , is selected as a candidate either if ranks among the top- classes
for matching or if ranks among the top- classes for matching . The similarity scores of the
selected candidates remain unchanged, while the others are set to 0 (see Figure 1).
Step 4: Max-Weight Matching. The new similarity matrix, after Step 3, is used to match the
class pairs by Max-Weight Matching (MWM). Specifically, MWM finds a one-to-one mapping
that maximizes the total similarity [
Step 5: Confidence Threshold. After Step 4, some class pairs might have low similarity due to the one-to-one mapping constraint. To solve this, a confidence threshold is applied so that any matched class pair , with a similarity score less than is removed from the prediction.
Datasets. TEXTO is evaluated on two benchmarks of the OAEI Common Knowledge Graphs
Track6. The first benchmark NELL-DBpedia has been annotated by humans and verified by
experts. Note that the alignment is partial, not every class of an ontology has an equivalent
class in the other. The second benchmark YAGO-Wikidata was originally created from the
mapping between Schema.org and Wikidata7 used in YAGO [
The textual descriptions of the classes are absent in both datasets. For the NELL-DBpedia dataset, the missing descriptions of DBpedia classes are completed with those of the equivalent Wikidata classes. For the YAGO-Wikidata dataset, the descriptions are directly extracted from the corresponding databases, Schema.org and Wikidata.
In addition to these two reference datasets, a new dataset is proposed called SchemaWikidata, based on the refreshed alignment of Schema.org and Wikidata. Note that the alignment proposed in the benchmark YAGO-Wikidata dates back to 2017. Some class URIs are deprecated. Our dataset is based on the SPARQL query on Wikidata using the owl:equivalentClass predicate (property P1706). The new dataset consists of 343 class pairs, 39 more than those available in the YAGO-Wikidata dataset. The new dataset is available in the OAEI
5https://www.sbert.net/docs/pretrained_models.html 6http://oaei.ontologymatching.org/2022/commonKG/index.html 7https://github.com/okfn-brasil/schemaOrg-Wikidata-Map standard8 on the GitHub repository of the paper. The statistics of the considered datasets are shown in Table 1. Results. TEXTO is implemented in Python using Pytorch and Owlready29. Our evaluation is completed in under 5 minutes on a Linux machine with 62 GB of RAM and 8 CPUs (3.47GHz) processors. For both the NELL-DBpedia and Schema-Wikidata datasets, the hyper-parameter weighting label and description similarities is set to 0.5; for the YAGO-Wikidata dataset, it is set to 0.4. For all datasets, the parameter for candidate selection is set to 5, and the confidence threshold is set to 0.4. Table 2 reports the results of the experiments in terms of precision, recall and F1-score, for the two benchmarks of the OAEI, for which results of the state-of-the-art methods are available. For the dataset NELL-DBpedia, where the alignment is partial, any prediction involving both classes that are not mapped in the gold standard is ignored. Ablation study. We now present an ablation study of TEXTO on the three datasets, focusing on the processing of textual information and the impact of MWM. The results are shown in Table 3. For methods without MWM, the top-1 candidate with positive similarity is selected. The methods with MWM component also include candidate selection (Step 3) and confidence threshold (Step 5).
The ablation study reveals that leveraging the textual information contained in class descriptions increases both precision and recall, a trend prominently observed in the YAGO-Wikidata and Schema-Wikidata datasets. In the case of NELL-DBpedia, the addition of class descriptions does not improve recall. A potential explanation is the limited number of classes in this dataset, making pure label information suficient for ontology matching. Finally, replacing GloVe by SBERT for embedding the labels tends to decrease performance, suggesting that a word embedding model is more appropriate than a language model for encoding labels. Discussion. In the benchmarks used in the literature for common KGs, the set of available classes is restricted to the set of classes for which the gold mapping is available (or slightly larger, cf. Table 1). In reality, the number of classes to match is much higher (there are more than 2M classes in Wikidata, for instance) and these classes have a complex, hierarchical structure that makes matching much more dificult. This raises the following question:
What is the efectiveness of existing methods beyond the benchmarks of the literature? To this end, further experiments have been conducted on expanded versions of the YAGOWikidata and Schema-Wikidata datasets. Specifically, we have added all 1,360 classes of Schema.org instead of only those for which the gold standard is known. These additional classes could be sub-classes, super-classes, or siblings of the original classes. The results of TEXTO on these extended datasets are presented in Table 4.
We observe that TEXTO still achieves good performance in these more realistic scenarios, although the alignment is no longer perfect. This suggests that richer benchmarks should be considered for ontology matching, to make the task more challenging.
In this paper, we have introduced an ontology matching system called TEXTO, which leverages both the label and the textual description of each class. This matcher combines the vector representations obtained from the GloVe word embedding model for the label and from the SBERT language model for the description. The resulting similarity scores are then used to derive the best mapping.
Our system has been tested on the OAEI benchmarks, enriched with the description of each class, and on a new dataset matching Schema.org and Wikidata. The results show that TEXTO achieves almost perfect alignment, using textual information only. This calls for the publication of new benchmarks with more classes, most likely without complete alignment, to make the task more challenging. In such more realistic scenarios, the question of the respective importance of textual content and structural information for ontology matching remains open.