In the realm of ontology engineering, accurately classifying domain entities into top-level ontology concepts is a critical task, with significant implications in the time and efort required to build ontologies from scratch. This paper delves into the influential factors afecting the performance of using informal definitions to represent domain entities textually, Language Models to represent these definitions as embedding vectors, and the KNearest Neighbors (KNN) algorithm to classify these embeddings into top-level ontology concepts. Also, we particularly focused on the Gene Ontology (GO) and Chemical Entities of Biological Interest (ChEBI) ontologies. We hypothesize that the embedding representation of informal definitions of highly specialized domains may present diferent behaviors regarding their proximity with other informal definitions of other domains, influencing the predicted top-level ontology concept. To test our hypothesis, we conducted a series of experiments using variations on the number of GO and ChEBI domain entities in the training sample of our classifier. Our results indicate that the relation between the proximity of domain entities in the embedding space and the top-level ontology concept of these domain entities varies according to the domain specificity. Also, this result is strongly influenced by how ontology developers write the informal definitions in each domain. The findings underscore the potential of informal definitions in reflecting top-level ontology concepts and point toward using consolidated domain entities in a domain ontology during the training stage of the classifier.
In ontology engineering, the accurate classification of domain entities into top-level ontology concepts
is a crucial task with significant implications for the time and efort required to build ontologies from
scratch. This task involves not only recognizing and extracting entities but also understanding the
theoretical foundations and implications of aligning these entities with appropriate top-level concepts.
Top-level ontologies, such as the Basic Formal Ontology (BFO) [
This paper takes advantage of the classification pipelines proposed in [
To test our hypothesis, we conducted a series of experiments using variations in the number of GO and ChEBI domain entities in the training sample of our classifier. Our results indicate that the relationship between the proximity of domain entities in the embedding space and their top-level ontology concepts varies according to domain specificity. Furthermore, this relationship is strongly influenced by how ontology developers write the informal definitions in each domain. The findings underscore the potential of informal definitions in reflecting top-level ontology concepts and point toward the use of consolidated domain entities in a domain ontology during the classifier’s training stage.
The remainder of this paper is organized as follows: Section 2 reviews related work in ontology, language models, and the use of informal definitions. Section 3 outlines the research questions and objectives guiding our study. Section 4 describes the experimental setup and presents the results of our experiments. Finally, Section 5 discusses the implications of our findings and concludes the paper with suggestions for future research.
In this section, we discuss the key aspects of ontologies, top-level ontologies, and informal definitions in enhancing semantic interoperability and knowledge representation. We review the evolution from the Distributional Hypothesis to modern transformer models like BERT and GPT in NLP. Finally, we examine methods for automating domain entity classification into top-level concepts, including using external resources, combining terms with informal definitions, and cross-domain classification scenarios.
The Basic Formal Ontology (BFO) is a top-level ontology designed to support data integration in scientific domains by providing general concepts that can be reused across multiple domain ontologies, aiding in the unification and categorization of information from diferent domains. The main subdivision in the BFO structure regards Continuants and Occurrents. Continuants represent entities that persist through time while retaining their identity, such as physical objects or substances. Occurrents encompass entities that unfolds over time, encompassing processes and events that have temporal duration. Occurrents include phenomena such as a biological process (e.g., cell division) or a historical event (e.g., a volcanic eruption). Furthermore, Continuants can also subdivided into Independent Continuants, Specifically Dependent Continuants, and Generically Dependent Continuants. Independent Continuants are entities that exist independently of other entities, such as organisms, artifacts, or specific substances, e.g., a human or a rock. Specifically Dependent Continuants are entities that depends on one or more Independent Continuants to exist, e.g., a biological function (like digestion) or qualities (like color). Generically Dependent Continuants are entities that depends on Independent Continuants for their existence, but unlike Specifically Dependent Continuants, they can exist in multiple instances or locations. For example, a software program can be copied across diferent systems but still remains the same entity. In addition, Occurrent entities can be subdivided into Processes, which are entities with temporal parts that, at some time, involve a material entity as a participant.
The BFO ontology serves as a backbone to several domain ontologies, such as The Gene Ontology
(GO) and The Chemical Entities of Biological Interest (ChEBI) ontologies. The GO ontology describes
knowledge in the biological domain through three subdomains: molecular functions, cellular
components, and biological processes. The ChEBI ontology, maintained by the European Bioinformatics
Institute, provides a controlled vocabulary for biochemical terminology across four subdomains:
Molecular Structure, Biological Role, Application, and Subatomic Particle, all mapped to conform with BFO.
A key aspect of GO and ChEBI ontologies is the use of clear and precise definitions. In ontologies,
definitions are designed to align terms with cognitive and linguistic requirements, enhancing inferential
competence and ensuring efective communication [
Harris formulated the Distributional Hypothesis in 1954 [
Transformer-based models like BERT (Bidirectional Encoder Representations from Transformers)
[
In recent years, significant strides have been made in automating the classification of domain entities
into top-level ontological concepts. For instance, [
Building on these advancements, [20] proposed a systematic Foundational Ontology (FO) probing methodology. As opposed in using informal definitions, this methodology uses pairs of words and their example sentences to test language models’ ability to classify words into FO categories, achieving around 90% accuracy in FO classification tasks with Transformer-based models like BERT and RoBERTa. This high accuracy demonstrates that these models can naturally encode fundamental ontological concepts, improving semantic understanding and reasoning in natural language processing applications.
Further expanding the domain, [
The classification pipeline we previously proposed in [
Objective: The objective of this research question is to determine whether the approach proposed
in [
Scope and limitations: The scope of this research includes the assessment of the classification
approach proposed in [
Objective: The objective of this research question is to assess the overall performance of the
KNN classification approach proposed by [
Scope and limitations: In this research question, we performed a quantitative assessment of
the classifier’s accuracy and error rates when applied to GO and ChEBI entities. From that, the
study focuses on only these two domain ontologies, which, while representative, do not cover
the full diversity of domain ontologies. Results may vary with other ontologies not included in
this analysis. Additionally, the study focuses on quantitative analysis and may not delve deeply
into the underlying semantic and contextual nuances of the domain entities.
3. How does the approach perform with a small ratio of entities of a domain ontology
inside the training sample?
Motivation: The motivation behind this research question arises from the necessity to understand
the classifier’s performance proposed by [
Objectives: The objective of this research is to evaluate the classifier’s performance when faced with a small ratio of entities from specific domain ontologies, such as GO and ChEBI, within the training sample. Based on this, we examine the classification results to assess the accuracy and errors in categorizing GO and ChEBI entities. We diferentiate between correctly and incorrectly classified instances and identify the 5-nearest neighbors for each instance. By analyzing these neighbors, we determine which domain ontologies within the OBO Foundry have the most significant influence on both the correct and incorrect classifications.
Scope and limitations: This research is scoped to evaluate the performance of the classifier
specifically with GO and ChEBI entities when they are sparsely represented in the training sample.
The study includes a detailed quantitative analysis of classification metrics and an investigation
into the influence of other domain ontologies. Also, the limitations of this study are the same as
the Research Question 2.
4. How does the approach perform with highly specialized informal definitions?
Motivation: The motivation for this research question stems from the need to understand how the
classification approach proposed by [
Scope and limitations: The scope of this research includes evaluating the performance of
the classification approach proposed by [
The baseline setup1 for our experiments to answer the research questions presented in Section 3
follows the BERT+KNN classifier proposed in [
The BFO data used in our experiments are provided in [
We evaluated the results of all experiments in terms of accuracy (Equation 1):
+ Accuracy = (1)
+ + + where is the number of true positives, is the number of true negatives, is the number of false positives, and is the number of false negatives.
(a) Confusion matrix for the DLP dataset. (b) Confusion matrix for the BFO dataset.
In this experiment, we compared the results achieved using datasets from diferent top-level ontologies.
In this context, we used the DOLCE-Lite-Plus datasets from [
Figure 1 presents the confusion matrices of the experiment according to each top-level ontology dataset. The confusion matrix for the BFO dataset (Figure 1b) demonstrates that the classifier performs exceptionally well with the “Independent continuant" and “Process" to-level concepts, achieving 98% and 96% of accuracy, respectively. Although the accuracy results for “Specifically Dependent Continuant" 2The source code, datasets, and all experiments performed are available at https://github.com/BDI-UFRGS/ Alcides-ONTOBRAS-2024
(a) Correct classifications using ChEBI dataset. (b) Incorrect classifications using ChEBI dataset. (c) Correct classifications using GO dataset. (d) Incorrect classifications using GO dataset. and “Generically Dependent Continuant" are slightly worse, with, respectively, 87% and 89% accuracy, the overall BERT+KNN classifier performance using the BFO dataset is equally comparable with the results achieved using the DLP dataset (Figure 1a), indicating a promising capability to accurately classify domain entities into top-level ontology concepts in diferent top-level ontologies.
In addition to the BFO dataset, we investigated how the GO and ChEBI ontologies performed in this experiment. Figure 2 presents the confusion matrices of the experiment according to ChEBI and GO datasets. The confusion matrix for ChEBI reveals that the model performs excellently with “Independent continuant" entities, achieving a remarkable 99% accuracy. However, the “Specifically dependent continuant" concept exhibits significant challenges, with an accuracy rate of only 62%. On the other hand, the confusion matrix for the GO dataset (Figure 2b) shows high accuracy for “Independent Continuant" and “Process" top-level concepts, with 90% and 98% of accuracy, respectively.
Figure 3 presents the other domain ontologies inside the BFO dataset which most contribute to the right and wrong classifications for both GO and ChEBI datasets. By analyzing Figure 3a and Figure 3c, we can see that the main domain entities that contributed to the BERT+KNN classifier making correct classifications were entities of the same domain ontology. Although this result is interesting to analyze, it can be a disadvantage if no ontology domain entities exist in the classifier training set.
(a) Confusion matrix for the ChEBI dataset. (b) Confusion matrix for the GO dataset.
In this experiment, we investigate the performance of the BERT+KNN classifier for classifying domain entities, particularly focusing on entities from domain ontologies that were not part of the training sample. In this context, the results are presented through confusion matrices (Figure 4) and bar charts (Figure 5), comparing the classifications for two domain ontologies: ChEBI (Chemical Entities of Biological Interest) and GO (Gene Ontology). These visualizations provide insights into how well the model generalizes to unseen data and the efectiveness of the classification approach.
Figure 4a shows the confusion matrix for the ChEBI dataset, indicating that the majority of entities, specifically “Independent Continuants", were misclassified as “Processes", suggesting a high rate of misclassification in this domain. Also, for the “Specifically Dependent Continuant" top-level concept, the accuracy rate is low, with a significant number of false positives for “Processes". These results indicate that the model struggles to accurately classify entities from the ChEBI ontology, particularly due to “Process" entities in other domain ontologies. In contrast, the GO confusion matrix (Figure 4b) shows a diferent pattern. Here, “Independent Continuant" have a higher accuracy rate, with a substantial number of entities correctly classified. However, there are still significant misclassifications, with “Process" entities often being confused with “Independent Continuant" entities. This suggests that while the model performs better with GO entities compared to ChEBI, there are still challenges in correctly classifying processes.
Figure 5 presents the bar charts representing the percentage of examples in the 5-nearest neighbors of the classified domain entities. For ChEBI, the wrong prediction chart indicates a dominant presence of GO ontology, suggesting this ontology has a significant influence on incorrect classifications. This result fits exactly with the confusion matrix in Figure 4a, where most of the misclassifications are due to (c) Correct classifications using GO dataset. (d) Incorrect classifications using GO dataset. “Process" entities and GO is the domain ontology with most of the “Process" entities in the BFO dataset. In contrast, for GO, the wrong prediction chart shows a strong influence of ChEBI entities. This suggests that although ChEBI and GO are diferent ontologies in diferent domains, the top-level concepts of their entities overlap in the embedding space because they have similar embedding representations of their informal definitions but the informal definition represents entities with diferent top-level concepts.
As opposed to the previous experiments, here we investigate the performance of the BERT+KNN classifier for classifying domain entities into top-level concepts by considering a small ratio of the domain ontology entities in the training sample. As in the previous experiment, the focus was on two domain ontologies, ChEBI and GO. For each of these ontologies, we used 10% of the entities in the training sample. The results are summarized in confusion matrices (Figure 6) and bar charts showing the distribution of examples in the 5-Nearest Neighbors (5-NN) (Figure 7) for each ontology. These visualizations provide insights into how well the model generalizes with a small number of data of the domain ontology dataset used to test the classifier.
The confusion matrix for ChEBI (Figure 6a) demonstrates high accuracy for “Independent Continuant," with 96% of accuracy, but lower performance for “Specifically Dependent Continuant," where only 48% of accuracy. For the GO ontology, the confusion matrix (Figure 6b) reveals a diferent pattern. The model shows 85% accuracy for “Independent Continuant" and 71% for “Process," indicating reasonably good performance in these categories. These accuracy ratios that although the model uses a small amount of the entities of the domain ontology evaluated, the BERT+KNN classifier performs well in comparison with the results of the previous experiments.
The bar chart for the right classified examples for the ChEBI dataset (Figure 7a) shows that almost 100% of this classification result is due to domain entities of own ChEBI. This same result is also reflected
(a) Correct classifications using ChEBI dataset. (b) Incorrect classifications using ChEBI dataset. (c) Correct classifications using GO dataset. (d) Incorrect classifications using GO dataset. for GO (Figure 7c), where GO was the domain ontology that most influenced the correct classifications. In terms of incorrect classifications, we can see the same pattern as the previous experiment, where GO influences ChEBI entities to be misclassified, and vice versa.
The experiments conducted in this section answer the Research Questions 1, 2, and 3. These experiments also reveal several critical insights regarding the performance of the approach with highly specialized informal definitions (Research Question 4). The role of definitions in the classification process is to provide textual representations of domain entities, which are then encoded into embeddings used by the classifier. According to the experiments, the quality, structure, and specificity of these definitions significantly impacted classification accuracy, with specialized definitions leading to challenges in correctly mapping entities to top-level ontology concepts. From that, the BERT+KNN classifier demonstrated variable performance, significantly influenced by the specificity and domain of the informal definitions used. For instance, while the classifier performed well in top-level concepts like “Independent Continuant" and “Process" for the Gene Ontology (GO) and Chemical Entities of Biological Interest (ChEBI) ontologies, it showed lower accuracy in categories such as “Specifically Dependent Continuant." This variability underscores the challenge of accurately classifying specialized entities. In this context, compared to the performance with more general informal definitions from sources like WordNet and Wikipedia, the classifier faced new challenges with specialized definitions. General definitions tend to be more uniformly distributed in the embedding space, leading to higher classification accuracy. Also, this study highlighted that the way ontology developers write informal definitions in specialized domains has a substantial efect on classification performance. Definitions that are clear, precise, and consistent within a domain tend to result in better classification accuracy. Specifically, informal definitions following a structured format, such as “X is a Y that Z,” were more likely to be accurately classified. This structure helps maintain consistency and clarity, crucial for embedding models to capture the correct semantic meaning. However, specialized denfiitions, though rich in domain-specific context, introduce complexity requiring more sophisticated handling by the classifier.
The research presented in this study delves into the intricate task of classifying domain entities into top-level ontology concepts using informal definitions, language models for embedding vectors, and the K-Nearest Neighbors (KNN) algorithm as a classifier. The focus on the Gene Ontology (GO) and Chemical Entities of Biological Interest (ChEBI) ontologies has provided valuable insights into the influential factors afecting this classification process. Our experiments revealed that the relationship between the proximity of domain entities in the embedding space and their corresponding top-level ontology concepts varies significantly according to domain specificity. This variation underscores the critical influence of how ontology developers write informal definitions in each domain. This highlights the importance of maintaining consistency and clarity in definitions to enhance the semantic representation captured by language models. Furthermore, the results demonstrated the classifier’s robustness and versatility when applied to diferent top-level ontologies, such as the Basic Formal Ontology (BFO) and DOLCE-Lite-Plus (DLP). The analysis also highlighted the classifier’s performance when faced with entities from domain ontologies not included in the training sample, as well as scenarios with a small ratio of entities from a specific domain ontology. These findings revealed that while the classifier can generalize reasonably well, the presence of diverse and representative training data is crucial for optimal performance. In conclusion, this study emphasizes the potential of using informal definitions and language models to classify domain entities into top-level ontology concepts. The results advocate for the incorporation of consolidated domain entities during the training stage of classifiers to improve accuracy and robustness. Future research should explore the integration of more sophisticated techniques to handle highly specialized definitions and further validate the approach across a broader range of ontologies and domains.
Research supported by Higher Education Personnel Improvement Coordination (CAPES), code 0001, Brazilian National Council for Scientific and Technological Development (CNPq), and Petrobras. S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, T. Scialom, Llama 2: Open foundation and fine-tuned chat models, 2023. arXiv:2307.09288. [18] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, W. E. Sayed, Mistral 7b, 2023. arXiv:2310.06825. [19] G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S.
Kale, J. Love, et al., Gemma: Open models based on gemini research and technology, arXiv preprint arXiv:2403.08295 (2024). [20] M. Jullien, M. Valentino, A. Freitas, Do transformers encode a foundational ontology? probing abstract classes in natural language, 2022. URL: https://arxiv.org/abs/2201.10262. arXiv:2201.10262. [21] F. H. Rodrigues, A. G. Lopes, N. O. dos Santos, L. F. Garcia, J. L. Carbonera, M. Abel, On the use of chatgpt for classifying domain terms according to upper ontologies, in: International Conference on Conceptual Modeling, Springer, 2023, pp. 249–258.