Top-level ontologies provide a set of foundational concepts that have a well-founded philosophical meaning, being a useful tool in ontology engineering. However, in practice, few domain ontologies integrate top-level concepts. One of the dificulties refers to the selection of appropriate top-level concepts. This paper presents an analysis of top-level categories of a set of well-known domain ontologies from ontology matching benchmarks. Our main hypothesis is that training classification models using only concept comments (i.e., rdfs:comment) from top-level concepts can improve reported results in the literature. We then consider the best classifiers to estimate the distribution of concepts from Ontology Alignment Evaluation Initiative (OAEI) ontologies aligned to DOLCE top-level concepts.
Top-level ontologies provide a set of foundational concepts that have a well-founded
philosophical meaning, being a useful tool in ontology engineering [
A recent efort in such direction has been done in [
The remainder of this paper is structured as follows: in Section 2 related work is presented. In Section 3 the multi-inheritance is discussed along with the approaches adopted. Section 4 presents an evaluation of the performance of the models. Section 5 presents the analysis of OAEI ontologies and, finally, in Section 6 the conclusion and future work are discussed.
Previous work has shown the importance of associating concepts from top-level to domain
ontologies. In [
Analyzing the impact of using top ontologies as semantic bridges in ontology matching
have been done in [
In order to automate this process, in [16], word sense disambiguation and word embedding
models have been used to automatically align top and domain concepts. The evaluation has been
conducted with the task of associating DOLCE and SUMO top-level concepts to ontologies from
three diferent domains. Automatisation has been also addressed in [
This section describes the characteristics of the datasets from [
Dataset Lopes22-5c This is the original dataset from [
Dataset Sousa23-5c The Sousa23-5c rebuilds the Lopes22-5c dataset while taking into account the problem of multi-inheritance, which is further detailed in Section 3.1.1. Hence, the resulting dataset Sousa23-5c difers from Lopes22-5c since the strategy of dealing with multi-inheritance filters ambiguous entities, while in Lopes23-5c, the entity is inserted multiple times with each possible top-level concept.
1The source code used in dataset generation and experiments can be found at https://gitlab.irit.fr/melodi/ ontology-matching/top-level. For the rebuild of the dataset, the version of OntoWordNet used was downloaded from http://www.loa.istc.cnr.it/ontologies/OWN/OWN.owl(on 01/04/23) having 66065 entities
Dataset Sousa23-6c Another characteristic of Lopes22-5c dataset is its highly imbalance concept distribution (Endurant 76%, Perdurant 10%, Quality 4%, Situation 6%, Abstract 3%). Our proposal to deal with this problem is to break Endurant into two groups of concepts, generating a more balanced dataset with 6 concepts. This dataset was built from Sousa23-5c by following the hierarchy of entities until reaching one of the 5 top-level concepts (Endurant, Perdurant, Quality, Situation, Abstract), and in the case that the type Endurant is found, it is replaced by the immediate child in the path (Physical-endurant or Non-physical-endurant).
The concept distribution of the 3 datasets is presented in Table 2.
In order to deal with the case of multiple paths to the top-level concepts, we consider two distinct scenarios: one when the multi-inheritance occurs in the Wordnet part of OntoWordNet, and the other when it is present in the DOLCE hierarchy. If an entity in WordNet has multi-inheritance, many paths are traversed, and if they all lead to the same type, the entity is added to the dataset. If the paths diverge, the entity is ignored.
In DOLCE, one example of a concept without a direct path to the proposed top-level concepts is Physical-realization that is a sub-concept of Spatio-temporal-particular and which in turn is the super concept of Endurant, Perdurant, and Quality causing ambiguity. To deal with these cases, when multi-inheritance occurs in the DOLCE hierarchy, a breadth-first search is performed until one of the defined top-level concepts is found. If the concept found is Spatio-temporal-particular, then the WordNet entity is not added to the dataset.
From the total of 66065 entities present in OntoWordNet, 889 entities in the WordNet hierarchy have multi-inheritance. However, 5023 entities in the DOLCE hierarchy remain ambiguous even after applying the strategy mentioned above. To solve this problem, for DOLCE concepts that do not have a direct superclass, a breadth-first search is performed by traversing the predicates RDFS.subClassOf, OWL.equivalentClass, OWL.intersectionOf, OWL.unionOf, RDF.first , RDF.rest in decreasing order priority, adding the resulting objects to the priority queue used for the search. Using this approach, the distribution of concepts remained deterministic over several runs.
After this disambiguation process, entities are post-processed, in which labels are selected from the entity name in the WordNet part and rdfs:comment as comments. comments are then converted to lowercase. Both newline characters and quotes are removed, and semicolons are replaced with periods. Labels having synonyms separated by two underscores (‘__‘) are split and generate new rows in the dataset. For example, SOFTHEARTEDNESS__TENDERNESS is split and generates two entries in the dataset SOFTHEARTEDNESS and TENDERNESS.
For the purpose of evaluating the performance of the classification models, 2 testing datasets based on OAEI Conference track ontologies2 were created. The top-level concepts are assigned using an existing reference alignment provided in [18] that aligns the highest concepts in Conference to DOLCE. The Conference dataset contains 70 correspondences between the concepts in Conference with concepts in DOLCE-Lite-Plus (DLP). The sub-concepts of topconcepts in Conference are aligned, by transitivity, to the top-level concepts in DOLCE. From the 70 alignments present in the initial reference alignment, 1 has multiple paths leading to the same top-level concept, whereas 34 were ambiguous and were manually assigned with the concept Endurant. The resulting datasets have 5 concepts (Conference-5c) and 6 concepts (Conference-6c) and their respective distributions are shown in Table 3.
In [
However, using more robust architectures like BERT [21] may achieve improved results in this task as also reported in [17]. One of the reasons is that BERT can generate better natural language text representations due to its capacity of managing context. Another point is that some entities have the same label while being assigned to diferent top concepts in the dataset Lopes22-5c. This can hamper the model’s ability to distinguish among the diferent concepts 2https://oaei.ontologymatching.org/2022/conference/index.html (on 01/07/23) while giving less importance to the label part of the input. Another issue is that, while comments can impact the training step, ontologies often contain a low amount of comments. Since the model makes a distinction between labels and comments, its capacity for generalization in the test phase can be reduced.
In this way, unifying the model’s input between labels and comments can improve the model’s performance since it will be able to take advantage of the information from comments during training while being able to work only with labels when comments are not present. Based on those assumptions, for better generalization, we used BERT with a classification head to predict the top-level concept that accepts a single text input that can be both labels or comments. The architecture using BERT is present in Figure1 b).
In order to verify the hypothesis that the use of comment improves the baseline model results in
BERT and in the model proposed in [
In [
One can notice that all classifiers achieved higher results using only comment as input, except the Gaussian Naive Bayes in the datasets Lopes22-5c and Sousa23-5c. The BERT model achieved the highest performance in all categories. And, in some cases, the BERT model obtained the same results even using label+comment input. One possible reason for that result is that the model can make better use of label information when appropriate due to the attention mechanism.
The confusion matrix for the BERT model in the 3 datasets can be seen in Tables 5, 6, and
7. One can see that, in the dataset Lopes22-5c, the model tends to misclassify a considerable
amount of Perdurants into Situation (16%), and Situations into Perdurant (17%). In the dataset
Sousa23-5c, this misclassification is between Perdurant and Quality 21%. In the Sousa23-6c
dataset, the model also misclassifies 13% of Perdurants into Situation and 23% of situations into
Perdurant. Similar misclassification in Situation concept is found in [
The models trained on Sousa23-5c and Sousa23-6c were tested on Conference-5c and Conference6c, respectively. Since the downsampling technique used for balancing the training datasets generates diferent dataset partitions, the results are evaluated in 150 steps to account for possible variations. In this evaluation phase, all the models receive as input only the labels of the entities, while the BERT model was trained only with comments. Model-Lopes is trained and tested in diferent cases. Case 1: the model was trained only with labels and the test input is fed into the label input of the model. Case 2: the model was trained only with comments and the test input is fed into the comment input of the model. Case 3: the model was trained with both labels and comments and the test input is fed into the label input of the model. And Case 4: the model was trained only with both labels and comments and the test input is fed into the comment input of the model. The results are present in Table 8.
In the results, the BERT model trained with the adopted hyperparameter settings is unstable and has the highest standard deviation. This model collapses in some cases, giving the same output for every input, causing it to have values ranging between 0 and 0.78 F-measure. The model achieves 0.00 F-1 when it outputs Quality for every input, as the test dataset does not have any element labeled as Quality. On the other hand, it achieves the highest F-1 (0.78) when the model predicts Endurant for every input as, in the test dataset, 0.78% of the entities are Endurant. The BERT model had the highest scores in Conference-5c when considering the 75% percentile results, excluding instances of collapse. In Conference-6c, the BERT model and the model trained and tested with comments had the best and nearly equal results. Since the BERT model achieves the highest scores on Conference test datasets, it was selected to analyze the OAEI datasets described in Section 5.
This section presents an analysis of the top-level concepts in diferent OAEI tracks (Conference, Anatomy, Complex, Food, BioML, BioDiv, MSE, and KG3 along with the characteristics of 3These tracks are described on https://oaei.ontologymatching.org/2022/ (on 01/07/23) comments present in ontology entities. In the first subsection, an analysis of the distribution of top concepts in the ontologies is provided. The second subsection evaluates the consistency of the reference alignments in terms of their top-level concepts. The third subsection evaluates the distribution of label length compared to the comments present in the training datasets.
Using the best model (BERT), the distribution of the top-level types of the concepts in the ontologies from schema alignment tracks is estimated. The distribution concerns the entities of each ontology present in the tracks, excluding blank nodes, and properties. For each entity, one label is collected searching for label predicates rdfs.label, skos.prefLabel, skos.altLabel, or if no label is found the label is retrieved from the resource identifier. As can be seen in Figure 2, the distribution of concepts for each track is distinct. The estimation by the model trained in Sousa23-5c shows that Complex, Food, and BioDiv have a high concentration of Endurants while the others are more distributed.
The distribution by the model trained on Sousa23-6c is presented in Figure 3 as well as its estimation. The Anatomy, Food, BioML, and KG ontologies have a high concentration of entities in one concept. For BioDiv, the majority of entities concentrate on Physical-endurant and Non-physical-endurant. The two models disagree with the distribution of Quality entities between tracks. The model trained on Sousa23-5c tends to classify some Endurant as Quality compared to the model trained on Sousa23-6c. The two models achieve similar distributions of Perdurant and Abstract types.
We analyzed the number of correspondences having entities associated with the same toplevel types (a correspondence is composed of a source and a target ontology entities). The proportion can be seen in Table 9 for the estimations given by the model trained on Sousa23-5c and Sousa23-6c. The distribution of correspondences that have the same type is similar for the two models in Conference, MSE, CommonKG, BioML, and KG. The diference in scores for the Endurant
Anatomy track is related to the distribution given by each model. For the model Sousa23-5c, the majority of entities are distributed as Endurant and Quality. In contrast, as the model Sousa23-6c yielded a high concentration of entities in the same concept, the reference alignments also follow the same tendency. In BioDiv, the model trained on Sousa23-6c achieves only 9.65% of the alignments with the same type for the alignment between Agrovoc and Nat ontologies as the majority of the diference between them is that the model gives physical-endurant for one entity, and non-physical-endurant for the other. This problem does not happen with the model trained on Sousa23-5c since both will be classified as Endurant and so, the alignments will have the same type. Both models trained in 5 and 6 classes yielded a few correspondences with the same types
Perdurant in the MSE track. The main reason resides in the lack of information in the entities, such as acronyms for chemical elements, in which the model has enough information to give the correct classification. A similar problem appears in the Conference datasets, for instance, between the alignments of PaperAbstract and Abstract or Country and State. Without further information, the label State is ambiguous, and the model can yield an incorrect classification based on it. As can be seen in these examples, a dataset containing entities along with its contained subgraphs could lead to better results for this task, as the model should have more information to deal with the ambiguous labels.
As entity comments are equivalent to ontology comments, the models are expected to have the best results when evaluated on them. However, comments are rare in the ontologies present in all tracks, and in this case, the labels need to be used to predict the top level. The distribution of labels and comments in all ontologies for schema matching is analyzed to verify the relation to the distribution of the proposed datasets. The number of labels and comments in all tracks are present in Table 11. Among all tracks, Anatomy, Food, and BioDiv have no comments. BioML (MONDO), BioML (UMLS) and BioDiv have less than 1% of comments. Conference and Complex have less than 5% and KGh while MSE have respectively 13.49% and 36.17%. Since the number of comments is low, the labels need to be used to predict the top-level types. However, as most machine learning models sufer from the well-known Out-of-Distribution Generalization (OOD) [26] problem, they are hampered by both the labels that do not have a similar syntactic structure of their comments and their length distribution difers.
The frequency of the lengths of labels and comments for each entity in all ontologies of all tracks is compared to the distribution of the comment lengths in the Sousa23-5c dataset. As can be seen in Figure 4, the average length of the comments in the training datasets is 50 characters, however, it was noticed that the majority of the labels are shorter. Furthermore, comments, which are relatively rare, have a high standard deviation. Also, the diferences in the distribution of text length between labels and comments hinder the capacity for generalization of the models.
In this work, we investigated the task of top-level concept prediction. We generated one dataset with 5 top-level concepts (Sousa23-5c) and one with 6 concepts (Sousa23-6c) based on OntoWordNet. In this generation, we discussed the multi-inheritance problem and proposed procedures to obtain top-level concepts for entities in unambiguous cases. In addition, classifier models were then trained and tested varying between the use of only labels, comments, and labels+comments. The yielded results show that the use of rdfs:comment improves the prediction performance of classification models. These results show the importance of rdfs:comment for automated system understanding of concepts. We selected the best-generated model to estimate the distribution of concepts in ontologies from well-known ontology matching benchmarks (OAEI).
The results show that tracks have diferent distributions among top-level types. The performed analysis of the reference alignments showed that a high number of correspondences, as expected, are of the same type. We consider that this gives us an estimation of the accuracy of the trained classifiers.
In future work, we intend to conduct experiments with new deep-learning architectures that should improve the results reported in this paper. Dynamically predicting the top-level types for each concept of an ontology should help in downstream tasks such as ontology matching. Therefore, as diferent ontologies have distinct top-level distributions, we expect that our present analysis could be used for generating better classification models in the near future. Since the labels of entities are ambiguous in some cases, including the ontology structure as contextual information for the classification models may improve the prediction of top-level concept types. Also, the high number of correspondences that have the same type in some OAEI tracks shows that using these tools could help increase matching systems performance by increasing the similarity of entities with the same top-level type. A systematic evaluation, IEEE Trans. Knowl. Data Eng. 22 (2010) 609–623. URL: https: //doi.org/10.1109/TKDE.2009.154. doi:10.1109/TKDE.2009.154. [13] N. F. Padilha, F. Baião, K. Revoredo, Alignment Patterns based on Unified Foundational
Ontology, in: Proc. of the Brazilian Ontology Research Seminar, 2012, pp. 48–59. [14] P. Mika, D. Oberle, A. Gangemi, M. Sabou, Foundations for Service Ontologies: Aligning
OWL-S to Dolce, in: Proc. of the 13th Conf. on World Wide Web, 2004, pp. 563–572. [15] R. Stevens, P. Lord, J. Malone, N. Matentzoglu, Measuring expert performance at manually classifying domain entities under upper ontology classes, J. Web Semant. 57 (2019). URL: https://doi.org/10.1016/j.websem.2018.08.004. doi:10.1016/j.websem.2018.08.004. [16] D. Schmidt, R. Basso, C. Trojahn, R. Vieira, Matching domain and top-level ontologies exploring word sense disambiguation and word embedding, in: E. Demidova, A. Zaveri, E. Simperl (Eds.), Emerging Topics in Semantic Technologies - ISWC 2018 Satellite Events [best papers from 13 of the workshops co-located with the ISWC 2018 conference], volume 36 of Studies on the Semantic Web, IOS Press, 2018, pp. 27–38. URL: https://doi.org/10.3233/978-1-61499-894-5-27. doi:10.3233/978-1-61499-894-5-27. [17] A. Lopes, J. L. Carbonera, D. Schmidt, L. F. Garcia, F. H. Rodrigues, M. Abel, Using terms and informal definitions to classify domain entities into top-level ontology concepts: An approach based on language models, Knowl. Based Syst. 265 (2023) 110385. URL: https://doi.org/10.1016/j.knosys.2023.110385. doi:10.1016/j.knosys.2023.110385. [18] D. Schmidt, C. Trojahn, R. Vieira, M. Kamel, Validating top-level and domain ontology alignments using wordnet, in: Proceedings of the IX ONTOBRAS Brazilian Ontology Research Seminar, Curitiba, Brazil, October 3rd, 2016, volume 1862 of CEUR Workshop Proceedings, CEUR-WS.org, 2016, pp. 119–130. [19] A. Graves, N. Jaitly, A. Mohamed, Hybrid speech recognition with deep bidirectional LSTM, in: 2013 IEEE Workshop on Automatic Speech Recognition and Understanding, Olomouc, Czech Republic, December 8-12, 2013, IEEE, 2013, pp. 273–278. URL: https: //doi.org/10.1109/ASRU.2013.6707742. doi:10.1109/ASRU.2013.6707742. [20] M. E. Peters, W. Ammar, C. Bhagavatula, R. Power, Semi-supervised sequence tagging with bidirectional language models, in: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, Association for Computational Linguistics, 2017, pp. 1756–1765.
URL: https://doi.org/10.18653/v1/P17-1161. doi:10.18653/v1/P17-1161. [21] J. Devlin, M. Chang, K. Lee, K. Toutanova, BERT: pre-training of deep bidirectional transformers for language understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), Association for Computational Linguistics, 2019, pp. 4171–4186.
URL: https://doi.org/10.18653/v1/n19-1423. doi:10.18653/v1/n19-1423. [22] Y. Sun, A. K. Wong, M. S. Kamel, Classification of imbalanced data: A review, International journal of pattern recognition and artificial intelligence 23 (2009) 687–719. [23] J. Pennington, R. Socher, C. D. Manning, Glove: Global vectors for word representation, in: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014, pp. 1532–1543. [24] P. Bojanowski, E. Grave, A. Joulin, T. Mikolov, Enriching word vectors with subword information, Trans. Assoc. Comput. Linguistics 5 (2017) 135–146. URL: https://doi.org/10. 1162/tacl_a_00051. doi:10.1162/tacl\_a\_00051. [25] Z. Zhang, Improved adam optimizer for deep neural networks, in: 26th IEEE/ACM International Symposium on Quality of Service, IWQoS 2018, Banf, AB, Canada, June 4-6, 2018, IEEE, 2018, pp. 1–2. URL: https://doi.org/10.1109/IWQoS.2018.8624183. doi:10. 1109/IWQoS.2018.8624183. [26] Z. Shen, J. Liu, Y. He, X. Zhang, R. Xu, H. Yu, P. Cui, Towards out-of-distribution generalization: A survey, CoRR abs/2108.13624 (2021). URL: https://arxiv.org/abs/2108.13624. arXiv:2108.13624.