Current AI systems, including smart search engines and recommendation systems tools for streamlining literature reviews, and interactive question-answering platforms, are becoming indispensable for researchers to navigate and understand the vast landscape of scientific knowledge. Taxonomies and ontologies of research topics are key to this process, but manually creating them is costly and often leads to outdated results. This poster paper shows the use of SciBERT model to automatically generate research topic ontologies. Our model excels at identifying semantic relationships between research topics, outperforming traditional methods. This approach promises to streamline the creation of accurate and up-to-date ontologies, enhancing the efectiveness of AI tools for researchers.
The current generation of AI technologies, such as smart search engines, recommendation
systems, and question-answering applications, significantly aids researchers in exploring and
interpreting scientific literature [
To address this, scientific knowledge graphs (SKGs) [
Research topics are essential for describing research concepts within SKGs, making ontologies
of research topics (e.g., MeSH, UMLS, CSO, NLM) crucial for organising and querying academic
information [
However, manually creating ontologies of research topics is costly and time-consuming,
often resulting in outdated representations. To address this challenge, several approaches have
been proposed, including the integration of ontology learning with crowdsourcing methods,
combining statistical analysis with user feedback [
In the same direction, this poster paper explores the use of SciBERT for generating research topic ontologies. Our goal is to develop a method that incorporates language model technology to update CSO and construct large-scale ontologies across scientific disciplines. We developed a model to identify four semantic relationships (supertopic, subtopic, same-as, and other ) between research topics and compared its performance to traditional feature-based solutions. Preliminary results show that the transformer-based model significantly outperforms traditional models. The gold standard and code are available on a GitHub repository5.
In this section, at first we describe the addressed task and the used datasets. Then, we illustrate a traditional feature-based approach, and our transformer-based technique.
In this work, we address a single-label multi-class classification problem. The task is to classify the relationship between a pair of research topics ( , ) according to four categories which are essential for ontology generation: • supertopic: is a parent topic of . E.g., ontological languages is a broader area than owl • subtopic: is a child topic of . E.g., nosql is a specific area within databases • same-as: and are diferent labels for the same concept. E.g., haptic interface and haptic device • other : and do not relate according to the above categories. E.g., blockchain and user interfaces
In this context, other can refer to either negative samples or alternative semantic relationships not currently considered by our method, such as partOf, or contributesTo.
For our gold standard, we selected portions of the Computer Science Ontology [
CSO comprises four primary semantic relationships. Among them, superTopicOf and
relatedEquivalent essentially correspond to our superTopic and same-as relationships, respectively. To
5Gold standard and code - https://github.com/aleessiap/LeveragingLMforGeneratingOntologies.git
construct the gold standard, we selected 4,713 superTopicOf triples from the CSO and
designated them as superTopic instances. Additionally, we chose 3,034 relatedEquivalent triples to
represent equivalence using the same-as relation. We also derived 4,713 subTopic relationships
by reversing the superTopic relationships. Lastly, we randomly paired topics to create 5,151
other relationships, ensuring that none of these pairs shared any of the previously identified
relationships within the CSO. The resulting gold standard dataset consists of 17,611 triples,
divided into 15,154 triples (86%) for the training set, 2,166 triples (12.3%) for the validation set,
and 291 triples (1.7%) for the test set. To prevent bias, we ensured that topic pairs in one set do
not appear in another. Moreover, each test set triple includes at least one topic not present in
the training set. These measures make the test set more challenging compared to those used
for Klink-2 [
Our classification task is commonly approached exploiting numerical features, usually measuring
the frequency and common usage of the two topics [
The first two features indicate the popularity of a topic. The third feature quantifies the relatedness of two topics. The fourth feature assesses the hierarchical relationship between the topics. After normalising the features, we trained two ensemble machine learning models: Gradient Boosting (GB) and Random Forest (RF); varying the number of estimators from 10 to 3000 to determine the optimal configuration.
Our method leveraging language models relies on SciBERT [
To address our classification task we fine-tuned SciBERT using the training set described
in Section 2.1. Specifically, we used the scibert-scivocab-uncased model from Huggingface.
As optimiser, we selected AdamW [
Using the test set described in Section 2.1, we evaluated the three methods outlined in the previous section: Gradient Boosting and Random Forest (both feature-based), and SciBERT (language model-based). We compared their performance using accuracy, precision, recall, and F-score, which are standard metrics for text classification.
Table 1 reports the experimental results. The language model-based method was far superior to the feature-based methods in all areas, achieving an impressive F1 score of 0.9129. This was over 27% higher than the other methods. Among the feature-based approaches, Random Forest performed better. The language model-based method was particularly efective in recognising superTopic and subTopic relations, where feature-based methods struggled, likely due to the presence of unfamiliar topics in the test set.
The language model-based method generally priorities precision over recall, particularly for the relations superTopic, subTopic, and same-as. However, for the other relation, it tends to miss some semantic connections, resulting in lower precision compared to recall. This suggests the model may incorrectly classify some related topics as other, an issue we intend to explore further in future research.
In this poster paper, we introduced a new method based on SciBERT to identify the relationship between research topics and conducted a comparative analysis against feature-based solutions. We fine-tuned a SciBERT model using a gold standard of triples derived from CSO. The model achieved an F1 score of 0.9129, a 27% improvement over methods using numerical features. These findings are significant given the growing demand for detailed ontologies to enhance content characterization in scientific KGs
In our future work, we aim to develop an innovative method for creating taxonomies of research topics to improve CSO and create large-scale ontologies across diferent scientific ifelds. We plan to combine language models and numerical features using knowledge injection techniques and experiment with recent large language models. We also intend to explore potential challenges when applying these techniques to other research domains and assess the impact of cross-disciplinary applications.