In this PAN-CLEF 2025 Subtask 2: Human-AI Collaborative Text Classification challenge, our objective is to categorize documents co-authored by humans and Large Language Models (LLMs). Specifically, we aim to classify texts into six distinct categories based on the nature of human and machine contributions. Utilizing the pre-trained language model DeBERTa-v3-large, we fine-tuned it for this specific classification task. Our experimental results demonstrate that this approach efectively distinguishes between diferent types of texts, contributing significantly to the understanding of human-AI collaboration and mitigating risks associated with synthetic text.
In recent years, research on human-AI collaborative writing has gradually increased. Early work mainly
focused on determining whether a text was generated by AI, such as the GPT-2 Detector developed
by OpenAI [
We utilize the oficially provided training, validation, and test sets, which collectively contain six types of text samples corresponding to diferent human-AI collaborative writing styles. Each sample includes the original text content (text) and its associated class label (label). Data preprocessing mainly involves the following steps: • Uniform Formatting: The raw text is directly fed into the model to preserve potentially styledistinguishing linguistic features. • Text Encoding and Alignment: The text is tokenized using the tokenizer corresponding to DeBERTa-v3-large, and all sequences are truncated or padded to a maximum length of 512 tokens to meet the model’s input requirements. • Label Mapping: A bidirectional mapping between category names and integer IDs (id2label / label2id) is established to support multi-class classification.
Recently, advances in Natural Language Processing (NLP) have benefited significantly from progress
in pre-trained language models, with DeBERTa standing out among them [
DeBERTa-v3-Large is the third-generation DeBERTa model proposed by Microsoft, with 178 million
parameters [
During actual training, we employed the standard full-parameter fine-tuning strategy. Specifically, we loaded a pre-trained language model and added a classification head on top of it. Then, all parameters of the entire model were updated without freezing any layers. This strategy is suitable when the target task dataset is moderately sized and suficient computing resources are available. In this case, the dataset meets these conditions.
Throughout the training process, the AdamW optimizer and a relatively small learning rate were used to ensure that the language representations already learned by the pre-trained model would not be disrupted during fine-tuning. Meanwhile, weight decay was implemented to prevent overfitting. The model was trained for three epochs in total. Additionally, an early stopping mechanism was utilized to both prevent overfitting, verifying promptly at per epoch and stopping early on the dev set, which improve model performance. Micro F1 score was used as the primary evaluation metric to guide the optimization process. To enhance training eficiency and save GPU memory, all input texts were uniformly truncated or padded to a maximum length of 512 tokens.
We firstly conducted tests on the oficial dev set. Evaluation metrics included macro-averaged recall, macro-averaged F1 score and macro-averaged precision.
Finally, we conducted tests on the oficial test set. Evaluation metrics included accuracy, macroaveraged recall (Macro-Recall), and macro-averaged F1 score (Macro-F1).
Recall
In the PAN-CLEF 2025 Subtask 2, the Recall performance of the DeBERTa-v3-large model became a key evaluation criterion. From the results on the dev set (Table 1), it is evident that the model performs exceptionally well in classifying "human-written, then machine-polished" and "deeply-mixed text" categories but faces challenges in identifying "machine-written, then machine-humanized" texts. On the test set, experimental results (Table 2) showed that compared to the baseline model RoBERTabase, DeBERTa-v3-large achieved a Macro Recall value of 56.74% on the same task, representing an improvement of 8.42%, indicating its superiority in complex multi-category classification tasks. Nevertheless, despite overall performance improvements, the significance of the Recall metric and potential optimization paths warrant deeper exploration.
This study leverages the DeBERTa-v3-Large model to achieve eficient classification of six types of Human-AI collaborative texts. Experimental results demonstrate that this model outperforms baseline pre-trained models in terms of Recall and accuracy, especially excelling in handling texts with complex semantic structures and ambiguous category boundaries.
Future research directions include: • Exploring multimodal feature fusion, such as integrating syntactic, semantic, and emotional information; • Introducing contrastive learning or self-supervised learning strategies to enhance the model’s sensitivity to subtle diferences; • Constructing more representative datasets covering a broader range of real-world applications; • Developing lightweight versions of the model suitable for deployment on low-resource devices.
This work is supported by the Research Projects of OrdinaryUniversities in Guangdong Province under Grant2023KTSCX133, the Guangdong Basic and Applied Basic Research Foundation under Grant 2022A1515140103.
The authors declare that the Qwen3 large language model was used during the preparation of this paper for text translation and language polishing. The final responsibility for the content, accuracy, and scientific integrity of the paper lies solely with the authors. [7] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized BERT pretraining approach, CoRR abs/1907.11692 (2019). URL: http://arxiv.org/abs/1907.11692. arXiv:1907.11692.