Text classification has become increasingly important with the exponential growth of digital text data, finding applications in sentiment analysis, spam detection, topic categorization, and content moderation across various domains. Our research introduced a novel approach that integrates reinforcement learning with a specialized reasoning path. This methodology enabled smaller 7B parameter language models to increase performance significantly to the level comparable to larger models e.g. Claude 3.7, on an open source Pubmed multilabel text classification task. We experimented with 1) Claude 3.7 and DeepSeek-R1-Distill-Qwen-7B (Qwen-7B) zero shot, 2) Supervised Fine-Tuned (SFT) Qwen-7B, 3) Reinforcement Learning (RL) Qwen-7B and 4) SFT + RL Qwen-7B. We also experimented with diferent reasoning paths: 1) no reasoning, and 2) Socratic reasoning, as well as diferent evaluation metrics as reward: 1) F1 score as reward, 2) Trustworthiness (or reasoning process accuracy) as reward. The training data are composed of ~11,000 pubmed publication abstracts. We evaluated the performance in another ~1,000 abstract. SFT + RL Qwen-7B with Socratic reasoning and F1 score as reward achieved the highest F1 score of 0.8348. In summary, we proposed an innovative post-training paradigm integrating SFT, RL, Socratic reasoning path, and Trustworthiness-as-Reward. With this paradigm, we were able to double the F1 score compared to the base 7B model and achieved a ~ 0.15 lift in F1 score compared to using SFT alone without reasoning. Our pipeline demonstrates that strategic optimization of smaller models can achieve superior results compared to simply scaling up the model size.
Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of natural language processing tasks, including text classification. Their performance can often be further enhanced through supervised fine-tuning (SFT). However, in our previous experiments on various datasets, it was found that using SFT alone cannot increase the performance of 7B models to the level of Claude 3.7 on various tasks. This finding motivates us to explore opportunities to further improve the 7B models through reinforcement learning and reasoning path design. Moreover, we propose increasing the trustworthiness of LLM reasoning process to increase the LLM’s performance on the final task.
This paper introduces a novel approach that integrates SFT, reinforcement learning (RL) with "Trustworthiness-as-Reward" and a specialized reasoning path to improve LLM performance specifically on text classification tasks. Using the evaluation metrics (F1-score or Trustworthiness) of classification tasks as reward signals, we create a feedback loop that allows the model to learn from its own predictions and gradually improve its performance. This method ofers a promising direction for optimizing LLMs through RL in a task-specific, metric-specific manner. Our approach not only aims to boost classification performance but also explores the potential of reasoning path for self-improvement in language models, paving the way for more adaptive and eficient AI systems in the field of natural language processing.
The outline of the paper includes: Background and related work survey (Section 2), the Proposed method (Section 3), Experiments (Section 4), and Conclusions (Section 5).
Recent advances in text classification using LLMs have demonstrated diverse approaches to enhance
classification performance while addressing computational and resource constraints. Several innovative
methods have emerged, including self-training techniques where LLMs generate augmented training
data and assist smaller models [
Researchers have also explored adaptive boosting frameworks, such as RGPT, which creates
specialized classification LLMs through recurrent ensemble of base learners [
Reinforcement Learning (RL) is a learning approach where an AI agent learns through trial and error by interacting with its environment. Instead of being directly taught, it discovers optimal actions by receiving feedback (rewards or penalties) based on its choices. Natural Language Processing (NLP), particularly in modern LLMs, shares some fundamental connections with RL.
The transformer architecture used in modern LLMs can be viewed as implementing a sophisticated form of RL’s state-action mapping, where the attention mechanism helps determine the most relevant context (state) for generating the next token (action). Recent trends to combine RL with LLMs target 1) model performance improvement through fine-tuning and 2) prompt optimization. Fine-tuning methods modify the LLM’s parameters, while prompt optimization methods focus on improving how we interact with unchanged models.
Human input plays an important role in RL-Fine tuning. Human input can be incorporated into
finetuning through two main channels: policy model training (where humans demonstrate desired LLM
behavior) and reward model training (where humans rank LLM outputs). In one study [
Prompt optimization can often align LLM behavior with human preferences without the computational
burden of fine-tuning. Most studies focus on tuning soft prompts (e.g., embeddings), which are dificult
to interpret and non-transferable across diferent LLMs [
Improvements in LLM reasoning are closely tied to advancements in a variety of techniques in inference scaling at test time and learning-to-reason at training time. On the other hand, the release of Reasoning Language Models (RLMs) such as OpenAI’s o1 and DeepSeek’s R1, marked a significant increase in research dedicated to learning-to-reason approaches.
While Chain-of-Thought (CoT) laid the groundwork, researchers have developed more complex
frameworks such as Tree-of-Thought (ToT) and Forest-of-Thought (FoT), with the latter introducing sparse
activation and dynamic self-correction strategies for improved eficiency [
Recent advances in LLM reasoning studies have seen a surge in RLMs that simulate inference, generating trajectories that capture potential reasoning paths using supervised and/or reinforcement learning. Training innovations have included the development of preference trees for comprehensive reasoning alignment [23] and rule-based reinforcement learning approaches using synthetic logic puzzles [24]. Process reward models (PRMs) have emerged as a promising direction, with innovations like steplevel advantages and process advantage verifiers (PAVs) showing improvements in both accuracy and compute eficiency [ 25]. While traditional RLHF methods remain influential, [ 26] reveals that diferent algorithms like Expert Iteration, PPO, and Return-Conditioned RL perform comparably well for improving reasoning capabilities. Novel approaches include ofline RL methods, with [ 27] introducing OREO, which jointly optimizes a policy model and value function using the soft Bellman Equation, showing superior performance on mathematical reasoning tasks. Several papers explore domainspecific applications, such as [ 28]’s SWE-RL, which employs a lightweight rule-based reward system for software engineering tasks, and [29]’s Rank-R1, which enhances document reranking through RL-based reasoning. More recent developments include[30]’s ReSearch framework, which integrates search operations into the reasoning chain without supervised data on reasoning steps, and [31]’s DAPO algorithm, which introduces decoupled clip and dynamic sampling policy optimization for large-scale RL training. These advancements are characterized by diverse reward mechanisms, from simple rule-based approaches to more sophisticated joint optimization strategies, all contributing to enhanced reasoning capabilities in LLMs.
The purpose of the paper is to compare and identify the best LLM setup for Pubmed publication category classification. Specifically, we introduced novel approaches leveraging ‘Socratic reasoning’ and ‘Trustworthiness-as-Reward’ to improve classification performance. ‘Socratic reasoning’ refers to the prompt instructions to guide LLM to examine several key aspects for the classification task through Question and Answer (QA). ’Trustworthiness’ score was calculated by evaluating the answer accuracy of LLMs to the fifteen Socratic questions provided in the prompt. Our definition of ’Trustworthiness’ is not focused on LLM’s internal mechanism behind the generation process. It aims to measure how well LLM collects all the useful information from the input and how accurate LLM understand key aspects of the input. The Socratic QA reasoning path + ‘Trustworthiness-as-Reward’ helps LLM to examine all useful information from the input and corrects incorrect understanding of the input during RL. With our hypothesis, this setup could provide users with the most accurate and trustworthy final answer that is based a correct and thorough synthesis of the input.
We experimented with several post-training methods: 1) Claude 3.7/DeepSeek-R1-Distill-Qwen-7B (Qwen-7B) zero-shot, 2) Supervised Fine-Tuned (SFT) Only, 3) Reinforcement Learning (RL) Only and 4) SFT + RL for Qwen-7B. We also experimented with diferent reasoning path prompt for inference optimization: 1) no reasoning, and 2) Socratic reasoning. Finally, we compared using F1-score as reward vs using F1-score + Trustworthiness’ as reward during RL training.
We leveraged an open source Kaggle dataset [32] to select training and test data for our experiments. Our training data includes ~11,000 randomly selected Pubmed publication abstracts from the original Kaggle dataset. Our test data is another randomly selected sample of ~1,100 Pubmed publication abstracts. The output used as training material was generated by Claude 3.7. Claude 3.7 was given the gold standard publication category and was instructed to provide reasoning (when applicable) on why the publication should belong to those categories. On average, each publication has 5.7 MeSH tags associated with it.
We compared three diferent base LLMs: 1) Claude 3.7 from Anthropic, released on 02/19/2025 and 2) DeepSeek-R1-Distill Qwen-7B [33] to be the base model for the task.
We experimented with four diferent post-training setups of LLM: 1) Zero-shot, 2) SFT-only, 3) RL-only, 4) SFT + RL. For SFT + RL, we performed SFT first, followed by RL as a second step. For RL, we adopted Group Relative Policy Optimization (GRPO) algorithm [34], a reinforcement learning algorithm that extends the concept of Proximal Policy Optimization (PPO) to handle group fairness constraints in decision-making systems. It was introduced as a method to address fairness concerns in reinforcement learning while maintaining good performance. For both SFT and RL, we employed QLoRA (Quantized Low-Rank Adaptation) [35] for both SFT and RL phases. QLoRA enables eficient model tuning by utilizing 4-bit quantization and low-rank adapters while maintaining model quality. This approach significantly reduces memory requirements compared to full-parameter fine-tuning, allowing the training of large language models on consumer-grade hardware. For RL, we tried three diferent reward functions, including: 1) F1-score as reward, 2) Trustworthiness (or reasoning process accuracy) as reward. ’Trustworthiness’ score was calculated by evaluating the answer accuracy of LLMs to the iffteen Socratic questions provided in the prompt.
We experimented three diferent prompt instructions (see Figure 1): 1) No reasoning: provide LLM with direct instruction of the task, which is to assign fifteen categories to each publication, 2) Socratic reasoning: asking LLM to answer fifteen binary (Yes/No) questions, which were designed based on domain knowledge (also provided by Claude 3.7) of the definitions of the fifteen categories.
We used two metrics for evaluation of model performance: F1-score and Trustworthiness. F1-score was calculated based on Precision and Recall calculated based on gold standard topic list vs model output topic list. Trustworthiness was calculated based on the accuracy of answer fifteen questions during reasoning process. During RL training, we used F1 or Trustworthiness as reward, and applied the same logic each generated output and feedback to the model.
In this section, we will share and compare performance with diferent reasoning paths, diferent training methods, and diferent reward function for RL.
Table 1 shows the F1 score of Claude 3.7 performance on the task of Pubmed publication classification, using diferent reasoning paths. It shows with a Socratic reasoning path, Claude 3.7 can achieve a ~ 2% lift in F1 score compared not instructed to reason. F1 lift mostly comes from improving on Recall (~ 8% lift), which is aligned with our hypothesis that using Socratic reasoning can induce more thorough examination of details in the abstract and reduce information skipped or missed during summarization.
Table 2 shows F1 score of fine-tuned Qwen-7B’s performance, using diferent reasoning paths. With only SFT, providing no reasoning instructions achieves a comparable F1 score compare to using Socratic reasoning. However, when combining with R1, training using Socratic reasoning path provide most notable improvement on model performance (~ 4% lift for SFT+RL vs SFT-only). Although guiding language models with Socratic reasoning path improved Recall, it does not lead to better precision than models without such reasoning instructions. The addition of complex reasoning prompts may actually increase the likelihood of errors and fabricated responses. This highlights the critical need to enhance the reliability and accuracy of model outputs to achieve better precision and overall performance (F1-score).
In summary, we proposed an innovative LLM training paradigm combining SFT, RL, Socratic reasoning path and Trustworthiness reward. By leveraging RL with Socratic reasoning path and Trustworthiness as reward, our paradigm efectively enhances the LLM’s capacity for learning and reasoning, while optimizing key performance metrics: we were able to double the F1 score compared to the base 7B model and achieved a ~ 0.15 lift in F1 score compared to using SFT alone without reasoning. This innovative combination proposed a new direction to maximize the potential of LLMs in various applications.
The author(s) has not employed any Generative AI tools to write this manuscript. [20] S. Hao, Y. Gu, H. Luo, T. Liu, X. Shao, X. Wang, S. Xie, H. Ma, A. Samavedhi, Q. Gao, et al., Llm reasoners: New evaluation, library, and analysis of step-by-step reasoning with large language models, arXiv preprint arXiv:2404.05221 (2024). [21] T. Wu, C. Xiang, J. T. Wang, P. Mittal, Efectively controlling reasoning models through thinking intervention, arXiv preprint arXiv:2503.24370 (2025). [22] V. Cahlik, R. Alves, P. Kordik, Reasoning-grounded natural language explanations for language models, arXiv preprint arXiv:2503.11248 (2025). [23] L. Yuan, G. Cui, H. Wang, N. Ding, X. Wang, J. Deng, B. Shan, H. Chen, R. Xie, Y. Lin, et al.,
Advancing llm reasoning generalists with preference trees, arXiv preprint arXiv:2404.02078 (2024). [24] T. Xie, Z. Gao, Q. Ren, H. Luo, Y. Hong, B. Dai, J. Zhou, K. Qiu, Z. Wu, C. Luo, Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning, arXiv preprint arXiv:2502.14768 (2025). [25] A. Setlur, C. Nagpal, A. Fisch, X. Geng, J. Eisenstein, R. Agarwal, A. Agarwal, J. Berant, A. Kumar, Rewarding progress: Scaling automated process verifiers for llm reasoning, arXiv preprint arXiv:2410.08146 (2024). [26] A. Havrilla, Y. Du, S. C. Raparthy, C. Nalmpantis, J. Dwivedi-Yu, M. Zhuravinskyi, E. Hambro, S. Sukhbaatar, R. Raileanu, Teaching large language models to reason with reinforcement learning, arXiv preprint arXiv:2403.04642 (2024). [27] H. Wang, S. Hao, H. Dong, S. Zhang, Y. Bao, Z. Yang, Y. Wu, Ofline reinforcement learning for llm multi-step reasoning, arXiv preprint arXiv:2412.16145 (2024). [28] Y. Wei, O. Duchenne, J. Copet, Q. Carbonneaux, L. Zhang, D. Fried, G. Synnaeve, R. Singh, S. I.
Wang, Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution, arXiv preprint arXiv:2502.18449 (2025). [29] S. Zhuang, X. Ma, B. Koopman, J. Lin, G. Zuccon, Rank-r1: Enhancing reasoning in llm-based document rerankers via reinforcement learning, arXiv preprint arXiv:2503.06034 (2025). [30] M. Chen, T. Li, H. Sun, Y. Zhou, C. Zhu, F. Yang, Z. Zhou, W. Chen, H. Wang, J. Z. Pan, et al., Learning to reason with search for llms via reinforcement learning, arXiv preprint arXiv:2503.19470 (2025). [31] Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, T. Fan, G. Liu, L. Liu, X. Liu, et al., Dapo: An open-source llm reinforcement learning system at scale, arXiv preprint arXiv:2503.14476 (2025). [32] O. Ahmad, Pubmed multilabel text classification dataset mesh, 2022. URL: https://www.kaggle.
com/datasets/owaiskhan9654/pubmed-multilabel-text-classification. [33] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al., Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, arXiv preprint arXiv:2501.12948 (2025). [34] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al., Deepseekmath: Pushing the limits of mathematical reasoning in open language models, arXiv preprint arXiv:2402.03300 (2024). [35] T. Dettmers, A. Pagnoni, A. Holtzman, L. Zettlemoyer, Qlora: Eficient finetuning of quantized llms, Advances in neural information processing systems 36 (2023) 10088–10115.