Introduction

The current generation of artificial intelligence technologies, such as smart search engines, recommendation systems, tools for systematic reviews, and question-answering applications, plays a crucial role in helping researchers explore and interpret scientific literature [1]. However, managing the vast amount of scientific literature, which increases by approximately 2.5 million papers each year [2], still presents a significant challenge. Large language models have revolutionised the field of natural language processing [3,4], but still struggle to process a large quantity of text. While they can answer questions about specific papers, they struggle to understand the broader context of a research area covering millions of papers.

To tackle this issue, it was proposed to develop structured and formal representations of the content of research publications, which could be more easily ingested by AI systems [5,6]. We thus saw the release of several knowledge graphs (KG) [7] that describe the metadata of research publications (e.g., SemOpenAlex [8], AIDA-KG [9]) as well as KGs that focus on the content of

Taxonomies in Computer Science

In the field of Computer Science, the ACM Computing Classification System2 is a well-known taxonomy of research topics. It is developed and maintained by the Association for Computing Machinery (ACM), the world's largest educational and scientific computing society, and covers about 2K research topics. It is manually curated, which makes its update process laborious and costly. Consequently, this taxonomy undergoes infrequent updates, with the latest one occurring in 2012, and becomes quickly outdated.

The Computer Science Ontology (CSO), discussed in the introduction, is one of the largest topic classifications, covering 14K research areas [31]. It has been automatically generated using the Klink-2 algorithm [26] on a dataset of 16 million scientific articles. Different from alternative solutions, CSO offers two main advantages over alternative solutions: i) it provides a very fine-grained representation of the field, rendering all the nuances of the area, and ii) it can be easily updated by executing Klink-2 on recent corpora of publications. CSO serves as the backbone for several tools utilised by the editorial team at Springer Nature, contributing to diverse applications such as research publication classification, identification of research communities, and forecasting research trends [16].

The IEEE Taxonomy mainly covers the field of Engineering but also contains different concepts relevant to computer science. It was developed and maintained by the Institute of Electrical and Electronics Engineers3 (IEEE). It supports the organisation of the Electrical and Electronics Engineering field, providing a standardised framework for classifying academic publications, research topics, and technical content within the IEEE's publications and databases. It contains around 5.6K topics and 24K relationships. The IEEE Taxonomy is also manually curated with minor updates released yearly.

In this paper, we will focus on CSO, as it represents the most extensive taxonomy in the field of computer science. Additionally, it includes sections that have undergone manual verification, making them suitable for use as a gold standard.

Ontology Generation

The review of existing literature reveals a variety of both semi-automatic and fully automatic approaches for the generation of ontologies and taxonomies. The initial step in formulating an ontology involves the identification of its underlying topics. In order to expedite this process, research is currently underway to develop automatic methods. For example, BERT [32] was used in [33] to solve the topic extraction task. Ontology extraction methods were traditionally based on natural language processing, clustering techniques, or statistical methods [34,35]. For example, Text2Onto [34] is a framework designed to learn ontologies from a collection of documents. This method identifies synonyms, sub-/superclass hierarchies, and more through the application of natural language processing techniques on sentence structures, leveraging phrases such as "such as... " and "and other... " to imply hierarchies between terms.

Shan et al. [36] applied a variation of this technique to generate Fields of Study (FoS) for Microsoft Academic [36], incorporating both hand-crafted concepts (first two levels) and topics automatically derived from Wikidata. However, this taxonomy learning approach focuses on Wikidata and does not leverage metadata associated with research papers. The OpenAlex team adopted a similar strategy [37], by employing the ASJC structure in Scopus and augmenting it with topics drawn from the papers using citation analysis.

Other approaches included the combination of ontology learning and crowdsourcing strategies, integrating statistical measures and user opinions [38,39]. For instance, Wohlgenannt et al. [38] merged human effort and machine computation by crowdsourcing the evaluation of an automatically generated ontology, aiming to dynamically validate the extracted relations.

Lately, the community has started to work towards leveraging LLMs for the creation of taxonomies, ontologies, and KGs [40]. For instance, Chen et al. [41] proposed an approach for taxonomy generation that consists of two modules: the first predicts parenthood relations and the other reconciles these predictions into trees. The parenthood prediction module generates likelihood scores for potential parent-child pairs, forming a graph of parent-child relation scores. The tree reconciliation module approaches the task as a graph optimisation problem, yielding the maximum spanning tree of this graph. The model is trained on subtrees sampled from Wordnet and tested on non-overlapping Wordnet subtrees.

To the best of our knowledge, specific methodologies employing language models for generating ontologies of research topics have not yet been established.

Methodology

This section outlines two main approaches for identifying the relationship between two research topics. As discussed in the introduction, this is the key component of a system for generating ontologies of research topics [26]. First, we describe and formalize the task (Section 3.1) and the dataset (Section 3.2). Then, we present a feature-based approach that uses a variety of traditional features adopted by the state-of-the-art methods (Section 3.3) and a transformer-based approach that employs the SciBERT model (Section 3.4).

Task Definition

The addressed task is the identification of the relationship between two research topics. More formally, given a pair of topics (𝑡 𝐴 , 𝑡 𝐵 ), we employ a single-label multi-class classification model to determine the specific semantic relationship between them. Naturally, various categories can be defined based on the specific predicates that need representation. For this paper, we have chosen three essential predicates from the CSO schema.

Therefore, we aim to classify the relationship between two topics according to four classes:

• supertopic: 𝑡 𝐴 is an ancestor of 𝑡 𝐵 , e.g., semantic web is a super area of rdf ;

• subtopic: 𝑡 𝐴 is a descendant of 𝑡 𝐵 , e.g., neural networks is a sub-topic of machine learning;

• same-as: 𝑡 𝐴 and 𝑡 𝐵 are two alternative labels for the same topic, e.g., haptic interface and haptic device; • other: 𝑡 𝐴 and 𝑡 𝐵 do not fit into any of the aforementioned relationships, e.g., cryptocurrency

and particle swarm optimizer.

Datasets

To conduct the experiments, we relied on two datasets: the Computer Science Ontology (CSO) [16] (introduced in Section 2.1) and the AIDA Knowledge Graph (AIDA-KG) [9]. We used CSO to derive a gold standard and AIDA-KG to compute a set of features that require linking topics to relevant papers (e.g., co-occurrence between two topics). CSO is made available on a website that allows domain experts to verify and modify the ontology. Therefore, different portions of the ontologies were manually verified and refined over time, often when conducting a specific analysis on certain topics (e.g., Software Engineering [42]). We thus take advantage of these manually verified portions to build a gold standard to train and evaluate the approaches. The CSO data model includes four main semantic relationships: superTopicOf : indicating that one topic is a sub-area of another (e.g., Artificial Intelligence is a super-area of Machine Learning); relatedEquivalent: denoting that two topics can be considered equivalent for the sake of exploring research data (e.g., Ontology Mapping and Ontology Matching); contributesTo: indicates that the research output of one topic contributes to another; owl:sameAs: it lists entities from other KGs (e.g., DBpedia, Wikidata) referring to the same concepts.

In order to build the gold standard, we selected 4,713 superTopicOf triples and mapped them as superTopic. We also selected 3,034 relatedEquivalent triples to represent equivalence through the same-as relation. Then, we derived 4,713 subTopic relationships by reversing the superTopic relationships. Finally, we randomly coupled topics to generate 5,151 other relationships, ensuring that none of these pairs shared any of the previously mentioned relationships according to the CSO framework.

The resulting gold standard counts 17,611 triples, which have been partitioned into 15,154 triples (∼86%) for the training set, 2,166 triples (∼12.3%) for the validation set, and 291 triples (∼1.7%) for the test set. The test set is intentionally small for two main reasons. First, to prevent data leakage bias, we ensured that none of the couples of topics appearing in a triplet of one set appeared in a triple of another set. For instance, we avoided that a triple <𝑡 𝐴 , superTopic, 𝑡 𝐵 > in the training set could appear as <𝑡 𝐵 , subTopic, 𝑡 𝐴 > in the test set. Second, we generated the test set so that each triple contains at least one topic that is completely absent from the training set. It is important to note that these adjustments make this test set more challenging than the ones previously used to test Klink [25] and Klink-2 [26].

AIDA-KG [9] is a KG integrating 25 million publications linked to research topics in CSO, researcher profiles, and 66 industrial sectors. We employ this resource to derive the occurrence of the relevant topics across the paper abstracts as well as their co-occurrences. These metrics will be used for our feature-based methods.

Feature-based Method

The task defined in Section 3.1 has been usually tackled by leveraging a variety of numerical features, typically derived from the two topics frequency and common usage [26,43]. These approaches typically involve combining these features in a mathematical function or with a classifier [26].

We implemented a feature-based classification method that, for each pair of topics (𝑡 𝐴 , 𝑡 𝐵 ), leverages four features:

• occA: number of times topic A appears in paper abstracts;

• occB: number of times topic B appears in paper abstracts;

• cooccurrenceAB: number of times both topic A and B simultaneously appear in abstracts;

• subsumption: it indicates the degree of overlap between the co-occurring topics, calculated using subsumption

= 𝑐𝑜𝑜𝑐𝑐𝑢𝑟𝑟𝑒𝑛𝑐𝑒𝐴𝐵 𝑜𝑐𝑐𝐴 − 𝑐𝑜𝑜𝑐𝑐𝑢𝑟𝑟𝑒𝑛𝑐𝑒𝐴𝐵 𝑜𝑐𝑐𝐵 .

The initial two features reflect the popularity of a topic. The third feature quantifies how related two topics are, based on their frequency of co-occurrence in research papers. The fourth feature evaluates the presence of a hierarchical relationship between the two topics.

For each triple, we extracted these features by querying the AIDA KG. We normalised these features and then we trained two machine learning models: Gradient Boosting (GB) and Random Forest (RF). These approaches are widely employed and renowned for their strong performance across various domains [44], making them excellent candidates for our task. They are both ensemble models, combining multiple weak learners. We conducted several experiments with both models, varying the number of estimators, ranging from 10 to 3000.

Language Model-based Method

To devise a method leveraging language models we employed SciBERT [30], a model based on BERT [32]. BERT is a widely acclaimed model in natural language processing, renowned for its proficiency in understanding and processing human language. SciBERT extends BERT's capabilities by specializing in scientific texts, making it an ideal choice for our objectives. Specifically, SciBERT was trained on a large corpus of scientific text, primarily from SemanticScholar. BERT and SciBERT excel in comprehending context and disambiguating polysemous words, demonstrating a human-like common sense in language parsing [32].

To adapt SciBERT for our specific classification task, we undertook a fine-tuning process using the training set described in Section 3.2. To this purpose, we leverage the scibert-scivocabuncased with Huggingface [45]. We chose AdamW [46] as the optimiser, which is a weighted version of Adam [47] that helps prevent overfitting in large models.

The fine-tuning process involved providing the model with the surface forms of the two topics, separated by a semicolon, as well as the correct relationship class from the training set.

In our experiments, we varied the number of epochs (from 1 to 10), while keeping 50 warm-up steps. Our best-performing model was obtained after training for five epochs. more than a 27% increase compared to the alternatives. Among the feature-based approaches, the Random Forest classifier yields better results across all metrics. The superiority of the language model-based method is especially marked when considering the superTopic and subTopic relations. Feature-based methods achieve rather poor results in recognizing these relations (i.e., F-score close to 0.5). This underperformance might stem from the presence of at least one unfamiliar topic in each pair within the test set.

Examining the precision/recall tradeoff, the language model-based approach obtains higher precision than recall for three relations, namely superTopic, subTopic, and same-as. On the other hand, in the case of the other relationship, the precision is considerably lower than the recall (i.e., 0.8286 vs 0.9831). This discrepancy suggests that the method is prone to overlooking some semantic connections between topic pairs, mistakenly classifying them as unrelated. We plan to further investigate this issue in future work.

Conclusions

In this paper, we presented a novel SciBERT-based method for identifying the relationship between research topics and conducted a comparative analysis against feature-based solutions. For this purpose, we fine-tuned a SciBERT model using a gold standard of triples derived from CSO. The SciBERT-based model attained an F1 score of 0.9129, marking an improvement of more than 27% compared to methods that utilize numerical features. These findings are significant considering the growing demand from the scholarly community for developing more fine-grained ontologies of research topics that can enhance the characterisation of content within scientific KGs.

In future work, we aim to develop an innovative method for generating taxonomies of research topics to enhance CSO and generate large-scale ontologies across various scientific fields. To this end, we plan to integrate language models and numerical features by employing knowledge injection techniques [48]. We also intend to conduct experiments with recent large language models, such as Mistral [49] and LLaMa 2 [50]. This evaluation will take into account factors such as cost and environmental impact. Additionally, we intend to study the potential challenges that could arise when extending these techniques to other research domains, including fields like Engineering, Material Science, and Mathematics. Finally, we aim to explore whether a model trained in one discipline, such as Computer Science, can be effectively adapted and applied to a different field and assess the impact of such a cross-disciplinary application.