This work proposes a novel approach to text categorization - for unknown categories - in the context of scientific literature, using Natural Language Processing techniques. The study leverages the power of pre-trained language models, specifically SciBERT, to extract meaningful representations of abstracts from the ArXiv dataset. Text categorization is performed using the K-Means algorithm, and the optimal number of clusters is determined based on the Silhouette score. The results demonstrate that the proposed approach captures subject information more efectively than the traditional arXiv labeling system, leading to improved text categorization. The approach ofers potential for better navigation and recommendation systems in the rapidly growing landscape of scientific research literature.
In the past decade there has been a significant increase in the number of published research
papers, creating a need for better tools to navigate through the vast literature. ArXiv[
Recent advancements in Natural Language Processing (NLP) and the success of pre-trained https://rosanna-turrisi.webnode.it/ (R. Turrisi)
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
models [
This work leverages NLP to process abstracts of ArXiv papers and classify them into more
meaningful and flexible subject categories that go beyond the original labeling. The aim is to
create categories providing information about the original subject categories but less restrictive
and sector-based. The ultimate goal is to enhance literature search and recommendation systems
by providing more accurate and relevant categorization. The proposed approach difers from
most studies on text categorization [
The automatic classification of research publications is commonly achieved by assigning papers
to existing categories within hierarchically-structured vocabularies, such as Medical Subject
Headings (MeSH) [
Top 6 Categories
Bottom 10 Categories 350 300 250 200 150 100 50 0 hep-ph hep-th
gr-qc quant-ph ncuocnld-t-hmat.mtrl-sci Subject Category mphaytship.cGhsTy.fcslouicn-sdd.yc-mnheamt.q-puhant-gamsath.NmTath.pDhSyscicss.L.paOthoymsi-cpsh.gen-pmhath.DG
Subject Category learning algorithm is here proposed to extract complex information from paper abstracts and identify research categories that are related to, but distinct from, the pre-existing categories, ofering a more refined representation of the research articles.
The ArXiv Dataset [26] is a rich corpus of approximately two million articles, including author information, title, journal references, ArXiv categories, and abstract. For computational reasons, the subset of articles published in 2023 was selected. To ensure data quality, duplicated and withdrawn papers were removed from the dataset. Categories with a small number of papers, specifically those containing less than 250 articles, were also excluded. Furthermore, abstracts with fewer than 31 words were not considered in the analysis. This decision was made due to the observation that such short abstracts often contain meaningless or misleading texts, such as sentences indicating revisions or the absence of an abstract for comments. This resulted in a ifnal corpus of 43853 samples and reduced the memory usage from 240.3MB to 2.3MB.
ArXiv categories. The selected ArXiv subset comprises 40 unique subject categories (e.g., Algebric Geometry) from which 15 macro categories (e.g., Mathematics) can be retrieved.
Fig. 1 (left) shows the number of papers associated with the 6 most frequent categories. Specifically, it displays the number of papers associated to 1. High Energy Physics - Phenomenology (hep-ph); 2. High Energy Physics - Theory (hep-th); 3. General Relativity and Quantum Cosmology (gr-qc);
4. Quantum Physics (quant-ph); 5. Nuclear Theory (nucl-th); 6. Materials Science (cond-mat.mtrl-sci).
The most frequent category contains more than 9000 papers while the 6th most frequent category includes about 2000 papers. Fig. 1 (right) reports instead the 10 less frequent categories: math.GT (Geometric Topology), physics.flu-dyn (Fluid Dynamics), physics.chem-ph( Chemical Physics), cond-mat.quant-gas (Quantum Gases), math.NT (Number Theory), math.DS (Dynamical Systems), cs.LO (Logic in Computer Science), physics.atom-ph (Atomic Physics), physics.gen-ph (General Physics), math.DG (Diferential Geometry ). As we can see, the number of papers is between 250 and 350 for all of them.
Fig. 2 (a) provides a visual representation of the number of papers within each category. It is evident from the graph that the distribution of category labels is highly unbalanced and that the large majority of categories include less than 1000 articles. Since some papers may be associated with multiple labels, a histogram illustrating the number of categories associated with each paper is presented in Figure 2 (b). This histogram provides insights into the distribution of multiple category assignments for individual papers.
Abstract length. Fig. 3 reports an histogram of abstract length in terms of number of words. As we can see, the abstracts’ length typically ranges from 50 to 300 with few exceptions. Table 1 shows average, standard deviation (Std), minimum (Min), maximum (Max) and first, second and third quartiles (25%, 55%, 75%) of the distribution of abstracts’ length. This analysis is relevant for the study as the third quartile was used to fix a maximum text length in the tokenization process.
Text processing was performed using the spaCy [27] package, specifically using the en_core_ sci_lg component. This component is trained on scientific papers and ofers a large vocabulary and 600,000 word vectors. Each abstract in the dataset was first syntactically parsed in order to ifnd linguistic units and their grammar dependencies. Then, the following processing steps were applied: i) the text was converted to lowercase; ii) lemmatization was performed on linguistic units, excluding personal pronouns, to transform them into their base form; iii) punctuation marks and stop words were removed from the text. Finally, processed data was tokenized using the AutoTokenizer class from the Hugging-face/transformers package [28].
Text embedding was performed by relying on the SciBERT pre-trained model [
Fig. 4 and 5 provide a qualitative visual assessment of the resulting embeddings, highlighting the 6 most frequent subject categories. The t-SNE method [29] is employed to project the samples into a 2D space. In both figures, samples belonging to the selected ArXiv category are represented as green points, while the remaining categories are displayed in grey. The categories are presented in descending order of frequency.
The observed clustering of the green points in the extracted embeddings indicates a tendency
for points with similar category associations to group together. However, it is important to
note that the 3 most frequent categories exhibit overlapping clusters, particularly in the
PCASciBERT-CLS representation. This observation should be interpreted considering two factors: i)
a paper abstract can have multiple labels, as illustrated in Figure 2 (b); and ii) the representation
of distant points in a high-dimensional space may not be accurately captured in a 2D projection.
FT-SciBERT. As reported in [
The dataset was split into learning (90%) and testing (10%) set. The learning set was further
split into training (80%) and validation (20%) set. Training was performed by using 32 as batch
size, 2e-5 as learning rate, and dropout with a dropping probability of 0.1. These parameter
values were reported in [
Fig. 6 and 7 illustrate the embeddings obtained from the learning set using FT-SciBERT-T and FT-SciBERT-CLS, respectively. In all figures, green points represent the 2D projections of the embedding points associated with the 6 most frequent subject categories, while grey points represent other categories. The t-SNE algorithm was used to obtain the 2D projection. As anticipated, fine-tuning the model improved the embedding representation, resulting in more distinct clustering of abstracts with the same arXiv category.
Unsupervised text classification was performed using K-Means algorithm for each text representation. The goal was to determine the optimal number of categories, denoted by , for the clustering task. The range of was set from 2 to 50. To evaluate the clustering performance for diferent values of , the algorithm was trained on the training set and then evaluated on the validation set using the Silhouette metric. The Silhouette score ranges from -1 to 1, where a score of -1 indicates that points are wrongly assigned to clusters, a score of 0 suggests overlapping clusters, and a score of 1 indicates that points are perfectly assigned to well-separated clusters. The Silhouette metric was used as a criterion to determine the optimal number of categories. The value of that yielded the highest Silhouette score on the validation set was taken as the best choice for the number of categories. Alternatively, one may adopt the Within-Cluster Sum of Square (WCSS) curve. The WCSS curve measures the sum of squared distances between each point and its assigned cluster centroid. Typically, the curve exhibits an ‘elbow’ shape, and the optimal corresponds to the point where the curve starts to flatten out significantly. However, in this study, the elbow method was not utilized as the WCSS curve was found to be too smooth, making it challenging to identify a clear elbow point.
When using PCA-SciBERT representations, the Silhouette scores on the validation set was found to be close to 0 for any value of . This suggests that these embedding points are dificult to associate with non-overlapping clusters, indicating a lack of clear separation between categories. On the other hand, when using FT embeddings, the K-Means clustering algorithm achieved better results. The Silhouette score for FT-SciBERT-T was 0.36, while FT-SciBERT-CLS achieved a score of 0.41. As the higher Silhouette score for FT-SciBERT-CLS indicated better cluster separation and cohesion, further observations and the evaluation on the testing set were conducted only for the FT-SciBERT-CLS embedding.
Fig. 8 presents the evaluation results on the validation set using the Silhouette metric for (a) Green points visualizes articles belonging to same arXiv subject category, while the others are plotted in grey. (b) K-Means clusters for = 32 . diferent numbers of clusters ( ) ranging from 2 to 50. As expected, the Silhouette score initially increases as increases, reaching a peak value, and then starts to decline rapidly for > 38 . The best clustering performance is observed at = 32 , which falls between the number of macro subject categories (15) and the number of subject categories (40) of ArXiv. This finding suggests that the arXiv categories do not fully capture the underlying structure of the abstract classes.
Unsupervised text categorization A qualitative evaluation of FT-SciBERT-CLS embedding can be observed in 9 (a). The 2D t-SNE projection of the testing set shows samples colored based on the 6 most frequent arXiv categories (green), while the grey points represent other categories. The clustering of green points indicates that the embedding captures arXiv category information. However, there is some overlap between embeddings of hep-th (High Energy Physics - Theory) and gr-qc (General Relativity and Quantum Cosmology) – two related scientific ifeld –, suggesting that the FT-SciBERT-CLS representation goes beyond arXiv categories.
For the final evaluation of the categorization task, = 32 was chosen based on the results obtained on the validation set. The classification model achieved a Silhouette score of 0.4 on the testing set. Figure 9 (b) displays a t-SNE projection of the K-Means results, where the testing samples are color-coded based on their assigned K-Means class. A more detailed analysis of the results on the testing set and the relationship between the identified and the arXiv categories can be seen in Figure 10. The figure presents four bar charts representing the four largest clusters. Each chart displays the distribution of papers across diferent arXiv categories within the cluster. Interestingly, most clusters exhibit one or a few dominant peaks, indicating a strong association with the arXiv category tags. This pattern is observed consistently across all K-means clusters.
A further examination was conducted to evaluate the top three most frequent subject categories within each cluster. Table 2 presents the list of the clusters sorted by decreasing cardinality, excluding arXiv categories with fewer than 10 samples. Results indicate that 66% of clusters – i.e. the identified categories – correspond to a predominant arXiv category or multiple subject categories within the same macro-category. For example, cluster 8 contains samples associated with Astrophysics of Galaxies, Astrophysics, and Cosmology and Non-galactic Astrophysics, all belonging to the macro-category of Astrophysics. According to the results, 21% of clusters have two main macro-categories, while only 7% have three. Notably, the K-Means algorithm successfully classifies texts that share the same subject area but are labeled with different category tags in arXiv. For instance, the stat.Th and math.ST papers, which correspond to the same subject of Statistic Theory are correctly clustered together in cluster 29. Similarly, cluster 2 and cluster 6 group together abstracts labeled as math-ph and mat.MP respectively, both representing the subject of Mathematical Physics. This demonstrates that the proposed approach efectively identifies and clusters abstracts based on their subject matter, overcoming the limitations of the original category labels and providing a more meaningful classification.
This study introduces a novel approach for text categorization in scientific literature using pretrained language models. By surpassing the limitations of traditional arXiv subject categories, this method enables more meaningful and accurate categorization of abstracts. The results demonstrate the efectiveness of the proposed approach in capturing the underlying subject matter, improving literature search and recommendation systems. This research contributes to the advancement of natural language processing techniques and addresses the need for more eficient tools in navigating scientific literature. Further studies will extend the proposed approach on other datasets from various scientific disciplines. to reach competitive performance to full-text, in: Proceedings of the 18th ACM/IEEE on joint conference on digital libraries, 2018, pp. 169–178. [17] L. Galke, F. Mai, A. Schelten, D. Brunsch, A. Scherp, Using titles vs. full-text as source for automated semantic document annotation, in: Proceedings of the Knowledge Capture Conference, 2017, pp. 1–4. [18] C. Nishioka, A. Scherp, Profiling vs. time vs. content: What does matter for top-k publication recommendation based on twitter profiles?, in: Proceedings of the 16th ACM/IEEE-CS on joint conference on digital libraries, 2016, pp. 171–180. [19] B. Kandimalla, S. Rohatgi, J. Wu, C. L. Giles, Large scale subject category classification of scholarly papers with deep attentive neural networks, Frontiers in research metrics and analytics 5 (2021) 600382. [20] F. Osborne, E. Motta, Mining semantic relations between research areas, in: The Semantic Web–ISWC 2012: 11th International Semantic Web Conference, Boston, MA, USA, November 11-15, 2012, Proceedings, Part I 11, Springer, 2012, pp. 410–426. [21] G. Erétéo, F. Gandon, M. Bufa, Semtagp: semantic community detection in folksonomies, in: 2011 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology, volume 1, IEEE, 2011, pp. 324–331. [22] A. A. Salatino, F. Osborne, T. Thanapalasingam, E. Motta, The cso classifier: Ontologydriven detection of research topics in scholarly articles, in: Digital Libraries for Open Knowledge: 23rd International Conference on Theory and Practice of Digital Libraries, TPDL 2019, Oslo, Norway, September 9-12, 2019, Proceedings 23, Springer, 2019, pp. 296–311. [23] K. Hitha, V. Kiran, Topic recognition and correlation analysis of articles in computer science, in: 2021 Fifth International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud)(I-SMAC), IEEE, 2021, pp. 1115–1118. [24] S. Al-Shareef, R. Alharbi, R. Alharbi, R. Almfarriji, M. Alsharif, R. Alharthi, L. Althaqafi, Investigating community detection in arabic scholarly network using ontology-based semantic expansion, in: 2022 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), IEEE, 2022, pp. 96–103. [25] A. Sharma, S. Kumar, Machine learning and ontology-based novel semantic document indexing for information retrieval, Computers & Industrial Engineering 176 (2023) 108940. [26] C. U. Library, 2019. URL: https://www.kaggle.com/datasets/Cornell-University/arxiv. [27] M. Honnibal, I. Montani, spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing, 2017. To appear. [28] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, A. M. Rush, Transformers: State-of-the-art natural language processing, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, Online, 2020, pp. 38–45. URL: https://www.aclweb.org/anthology/2020.emnlp-demos.6. [29] L. van der Maaten, G. Hinton, Visualizing data using t-sne, Journal of Machine Learning Research 9 (2008) 2579–2605. URL: http://jmlr.org/papers/v9/vandermaaten08a.html.