This paper presents our approach for the FoRC4CL shared task of NSLP 2025, which aims to classify computational linguistics publications into a predefined three-tiered taxonomy of 181 categories. We explore various strategies, including -Nearest Neighbors (-NN) on sentence embeddings, graph neural networks, and incorporating Large Language Models (LLMs) for label refinement in a zero-shot scenario. Our best-performing approach involves using a -NN model to fetch candidate labels, followed by an LLM model for prediction. We achieve a macro F1-score of 0.661 on the public test leaderboard, surpassing the nearest competitor by 20%. Our results demonstrate that a retrieval-enhanced LLM-based zero-shot approach, with a posteriori enforced hierarchical consistency, is a feasible solution for this task.
1. -Nearest neighbors (-NN) with Sentence Embeddings: Retrieving candidate labels based on semantic similarity of texts. 2. Graph Neural Networks for Text Classification: Representing papers as nodes in a -NN distance graph and modeling the task as multi-label node classification. 3. Zero-Shot Classification using Large Language Models (LLMs): Using an LLM to select the labels while enforcing hierarchy constraints based on Nearest Neighbors models.
Our best-performing approach involves a two-stage pipeline. First, candidate labels are retrieved using a -NN index based on sentence embeddings, and second, an LLM refines the predictions while ensuring hierarchical validity. Compared to traditional approaches, this method improves classification performance by leveraging both local similarity-based retrieval and the global reasoning capabilities of LLMs. 2nd International Workshop on Natural Scientific Language Processing and Research Knowledge Graphs (NSLP 2025), co-located with ESWC 2025, June 01–02, 2025, Portorož, Slovenia * Corresponding author. $ kbogas@di.uoa.gr (K. Bougiatiotis) 0000-0002-1910-2758 (K. Bougiatiotis); 0000-0001-9629-2367 (G. Paliouras)
© 2025 Copyright © 2025 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
The rest of this paper is structured as follows: Section 2 discusses related work, and Section 3 describes the dataset used in the shared task. Section 4 outlines the models created for this task, followed by the evaluation results and discussion in Section 5. Finally, we conclude with future directions in Section 61.
Hierarchical multi-label text classification (HMTC) assigns documents to multiple categories arranged
in a hierarchy, posing unique challenges due to label dependencies and the hierarchical structure.
Traditional methods, such as tree-based approaches like BONSAI [
Other recent advancements have explored the integration of Graph Neural Networks (GNNs) to
capture complex relationships in HMTC tasks. For instance, the Hierarchical Graph-Based Label
Propagation (HGBL) model constructs a heterogeneous graph of the entire corpus and employs Graph
Convolutional Networks (GCNs) to learn document representations, efectively capturing word
order and semantic relationships [
Large Language Models (LLMs) have also been applied to text classification tasks. The Clue And
Reasoning Prompting (CARP) method, for example, enhances LLMs’ reasoning abilities to address
complex linguistic phenomena in text classification [
The dataset used in this task consists of computational linguistics publications, structured across three
hierarchical levels of subject labels. This year’s shared task data comprised 1,050 training, 250 evaluation,
and 250 test samples. Each record contains multiple metadata fields, including the ACL Anthology ID,
title, abstract, author(s), URL to the full text, publisher, publication year and month, proceedings title,
DOI, and venue. This year, each sample was also enriched with the corresponding publication’s full
text.
1The code for reproduction and further experimentation is available at https://github.com/kbogas/DICE-FoRC25.
Hierarchy Level
1
2
3
ranging from general to fine-grained categories, following a predefined taxonomy 2. These annotations
were curated based on subject extraction techniques and aligned with topics derived from multiple
paper sources. The reliability of these annotations was evaluated using inter-annotator agreement
scores [
ImageandVideColaPsrsoicfiecsastiinognApApulitchaotrisohnispVerifiTceaxttioPnreprocSeosfstiwngareDDoecvuemloepnmteLnatyoutAnalAyscitsiv(eDLLeAa)rninIgntentDetectiSotnanceDetectiVonisualQAN(LVPQfAo)rtheLegalDoBmiaasinDetectioSntoryGeneraIntifoonrmationFCilotererifnegrenceResMoluulttiiohnopReasonBiniogmediNcaarlrNaLtiPvePlotinStOonryttoelollginygExtension Figure 1: Label counts, across hierarchy levels, in the training set. The further down the taxonomy tree, the sparser the label occurrences become.
A significant challenge of the dataset is the severe imbalance in label distribution, as illustrated in Figure 1. Certain labels appear infrequently, and some labels in the validation and test sets are absent from the training data. In addition to the primary dataset, this year’s frozen splits are accompanied by weakly supervised data of approximately 41,000 ACL publications. In our current experiments, these weakly supervised data were not utilized.
In this section, we provide details on the three main lines of research we worked on.
Building upon prior research using BERT-based and other transformer-based models for this task [
Our first approach combined Sentence Transformer embeddings with a -Nearest neighbors (NN) classifier on top (denoted ST + -NN). Specifically, we computed embeddings for all training samples and, at inference time, embedded the query document and retrieved its closest training samples based on cosine similarity. The final labels of the query sample were determined using a distanceweighted voting scheme, where labels from the nearest neighbors were aggregated proportionally to the samples’ similarity scores. This baseline served as a rapid prototyping framework, enabling eficient experimentation and metadata selection. Specifically, after a small-scale grid search, we adopted a setup where each training sample was represented by its title, abstract, and concatenated labels. The query samples were embedded using only the title and abstract. 2See the full taxonomy tree in https://huggingface.co/spaces/DFKI-SLT/Taxonomy4CL.
It is interesting to note that we can achieve high recall levels despite this baseline’s simplicity. This can be observed in Figure 2, which illustrates the recall across diferent hierarchy levels as the number of nearest neighbors increases. Even with a relatively small neighborhood size (e.g., = 20), we can achieve approximately 95% recall for Level 1 labels and around 90% on Levels 2 and 3. 1.0 0.9
In terms of the hyperparameters for this method, we use a fixed top- = 19, based on the elbow of the chart in Figure 2, cosine similarity as the embedding distance metric, and the default all-MiniLM-L6-v2 model for encoding each text. Regarding the final classification, we also have a voting threshold fixed at 0.3, denoting that if approximately3 19 × 0.3 ≈ 6 of the neighbors exhibit a specific label, we mark it as a valid prediction for the query sample at hand.
Given the strong performance of the baseline model, we sought to further exploit the structure of
the nearest-neighbor relationships by creating an explicit graph structure. Inspired by prior work
demonstrating the efectiveness of graph-based approaches for text classification [
We first construct a -NN graph in which each document represents a node, and edges are formed based on their nearest-neighbor connections as explained before. The sentence transformer embeddings serve as node features, and label prediction is formulated as a multi-label node classification problem.
We experimented with various GNN architectures, including Graph Convolutional Networks
(GCNs) [
Our final line of experimentation was motivated by the observation that transformer-based retrieval methods, while achieving high recall, sufered from reduced precision and low macro-F1 scores, denoting low performance on infrequent labels. To address this, we introduced a secondary filtering step using Large Language Models (LLMs) in a zero-shot setting without further fine-tuning.
In this setup, given a query sample, we first retrieve the nearest neighbors using our transformerbased approach and extract their corresponding labels. We then construct a natural language prompt containing the title and abstract of the query document, along with the label set of its nearest neighbors. 3Because the voting is distance-weighted, train samples near the query sample hold more “voting power” than the ones further away.
The LLM was tasked with refining these candidate labels and creating the final label set. Figure 3 contains a schematic representation of this workflow.
We experimented with various open-source LLMs, including the commonly used LLaMA3.1:8B [
The prompt structure for each of these scenarios is presented in AppendixA. For these models, the underlying -NN model is the same as before, with the voting threshold set to 0.01 to ensure high-recall in the label set of the nearest neighbors. Moreover, to make sure the LLMs generate valid labels, we validate their output using Pydantic4 templates, and we fuzzy-match the predicted label strings to the actual labels of the taxonomy. All LLMs are locally hosted using an ollama5 server.
The baseline model performs very well, given that it operates in a fully unsupervised manner with ifxed hyperparameters. Despite its simplicity, this approach already provides a strong micro F1-score, showcasing its potential on more common labels. In the GNN model, utilizing a 2-layer Graph Attention Network (GAT) layer, we observe a more pronounced drop in macro F1-score compared to micro F1-score, indicating poorer performance on infrequent labels. This aligns with expectations, as the -NN-based graph construction inherently biases predictions toward more common/homogeneous labels, leading to reduced performance on rare classes.
On the other hand, the first Gemma3-based model, ZS-Flat, significantly improves macro-F1, suggesting better handling of rare labels compared to purely neighborhood-based methods. However, its micro F1-score lags behind the sentence transformer baseline, indicating a lack of sample-wide consistency in label assignments. The hierarchical variant ZS-Hier performs worse than the flat version, suggesting that introducing hierarchical constraints in a zero-shot setting may increase task complexity, thus deteriorating results.
The NL variants show substantial improvement over the zero-shot variants in the micro F1-score. This aligns with expectations, as providing label context from similar examples helps refine predictions without extensive fine-tuning. The NL-Count variant further boosts the micro F1-score, being the ifrst LLM variant to surpass the baseline in this measure. Finally, the best-performing model is the variant which explicitly integrates hierarchical constraints. This yields a significant improvement in the micro F1-score, confirming that previous variants successfully identified fine-grained labels but frequently omitted their higher-level categories. Regarding the other tested LLMS, Llama 3.1:8B delivers results similar to Gemma but slightly worse due to its smaller number of parameters and older architecture. Surprisingly, the DeepSeek-R1:70B model underperforms both Gemma and Llama despite having approximately 9 times the number of parameters of Llama.
Overall, our findings highlight the efectiveness of our initial unsupervised baseline, which performs competitively against many zero-shot models in terms of micro F1-score. This is consistent with our intuition about the dataset’s “homophily” where similar samples tend to share labels, making nearestneighbor-based approaches particularly efective. However, the substantial gains in macro F1-score when using LLMs, even in a zero-shot setting, emphasize the need for better handling of “heterophilous” samples and rare labels.
In this study, we explored multiple approaches for multi-label classification of Computational Linguistics scholarly papers as part of the shared task challenge of FoRC4CL, organized by the NSLP 2025 workshop. Our experiments spanned unsupervised retrieval-based methods, graph-based learning, and Large Language Model zero-shot classification. The best-performing model combined Sentence Transformer embeddings for retrieval, with Gemma3:12B as a zero-shot predictor, leveraging related labels for final predictions. This approach achieved the highest performance among publicly ranked submissions on the task leaderboard. For future work, an interesting direction would be to investigate the complementarity of the diferent models explored in this study, potentially improving predictions through ensemble methods. Additionally, fine-tuning LLMs specifically for this task could further enhance classification accuracy.
This work was funded by the SIMPATHIC project in the context of European Union’s Horizon 2020 research and innovation programme under the agreement No 101080249.
During the preparation of this work, the authors used GPT-4 and Grammarly in order to: Drafting content, Grammar and spelling check, Improve writing style. After using these tools/services, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content.
The following table contains the prompts used for the diferent scenarios as explained in the Methodology Section 4.3.