In recent years, large language models (LLMs) have developed rapidly and have been widely used in the field of text generation. The increasingly realistic generated content has led to some security risks in information dissemination. Human-machine collaborative text classification has become a critical task and is extremely challenging. This paper proposes a human-machine collaborative text classification model, namely DeBERTa-FPN, which combines DeBERTa-V3 and feature fusion pyramid (FPN), aiming to use the powerful text processing capabilities of DeBERTa-V3-Large and the multi-scale feature extraction capabilities of FPN to improve the model performance of human-machine collaborative text classification. The introduction of FPN can enhance the model's function in feature extraction, and also more efectively combine global features to help complete the classification task. Experimental results show that our method significantly outperforms the baseline model, with 12% increase in Recall, 13% increase in F1, and 10% increase in Accuracy. Thus, we have verified the efectiveness of this method.
The goal of the human-machine collaborative text classification task in PAN@CLEF2025 [
In order to solve the problem of text writing classification and distinguish diferent writing types, this paper proposes DeBERTa-FPN, a new method that combines the DeBERTa-V3-Large pre-trained model and feature fusion pyramid to meet these challenges. As a pre-trained model, DeBERTa-V3-Large has strong context understanding ability and can capture detailed features in text. Feature Fusion Pyramid (FPN) is a multi-scale feature fusion module. This method was originally proposed in image classification tasks to link global and local features of images to improve classification eficiency. In this task, we use its multi-scale characteristics to associate features of texts of diferent lengths. By combining text features of diferent lengths, we efectively utilize global and local feature information, enhance the model’s perception of features, and ensure eficient and accurate classification. By combining the two, our model can efectively complete the classification task in the human-machine collaborative text classification task.
To evaluate the efectiveness of DeBERTa-FPN, we tested it through the submission platform speciifed by PAN. The platform provides a strictly controlled testing environment to ensure fair and transparent benchmarking based on established benchmarks. This initial submission is crucial to evaluating the practicality of the model in real-world scenarios and improving its performance based on objective feedback. After this evaluation, DeBERTa-FPN performed well in multiple key indicators, with the Recall index of 54.49%, F1 index of 54.4%, and Accuracy of 62.89%. These results are significantly better than baseline models such as RoBERTa-base and DetectGPT, which shows the efect of DeBERTa-FPN in distinguishing human-machine collaborative text and the efectiveness of our method in solving human-machine collaborative text classification tasks.
In recent years, text generation artificial intelligence tools have developed rapidly, and the generation
efect has been significantly improved in logic and vividness. As the quality of text generated by LLMs
(such as GPT-4, BERT, etc.) is close to human level, its wide application in education, medical care, news,
law and other fields has also brought ethical issues, such as the spread of false information, academic
plagiarism and public opinion manipulation. Therefore, it is urgent to develop eficient AI-generated text
detection technology[
This study used the “Human-machine Collaborative Text Classification Task” dataset provided by PAN@CLEF. This dataset is a publicly available dataset specifically designed to verify human-machine collaborative text classification. The PAN@CLEF dataset contains Multi-domain documents such as academic, news, and social media, Human-written and machine-generated samples, Collaborative texts with annotation layers for human/machine contributions, and these texts are in multiple languages. The PAN@CLEF dataset is usually organized into the following format, which contains text content, language type, label, data source, model type, and label type information: {"text":"...","language":"...","label":0,"source_dataset":"...","model":"...","label_text":"fully human-written"} {"text":"...","language":"...","label":1,"source_dataset":"...","model":"...","label_text":"human-written, then machine-polished"}
The validation set provided by the ”Human-Computer Collaborative Text Classification Task” in PAN@CLEF is a key component for testing and optimizing the author’s validation model. Its organization is consistent with the above. It should be mentioned that the label and label type are linked in the dataset, and 0-5 is used to represent diferent text types when predicting. Its comparison is as follows: { 0: "fully human-written", 1: "human-written, then machine-polished", 2: "machine-written, then machine-humanized", 3: "human-initiated, then machine-continued", 4: "deeply-mixed text (human + machine parts)", 5: "machine-written, then human-edited"}
The sample sizes of the training set and development set are 288,918 and 72,661 respectively. The specific number of categories is shown in Table 1. This study uses the PAN@CLEF dataset to train and evaluate a hybrid model that combines DeBERTa-V3-Large and FPN, aiming to improve the accuracy of human-machine collaborative text classification.
Specifically, this dataset helps us systematically understand and identify the characteristics of diferent types of human-machine collaborative text. Through a series of experiments on the PAN@CLEF dataset, we evaluate the performance of the model in distinguishing six types of human-machine collaborative text classification. We use multiple evaluation metrics such as precision, recall, and F1 score to comprehensively analyze the efectiveness of the model. The results show that our model can achieve satisfactory performance when dealing with this specific task.
In our study, we designed and implemented a hybrid neural network model named DeBERTa-FPN that combines DeBERTa-V3-Large and FPN to perform complex human-machine collaborative text classiifcation, aiming to distinguish six types of text with diferent machine participation. The structure of the model is shown in Figure 1. The model architecture aims to make full use of the deep semantic processing capabilities of DeBERTa-V3-Large and the global information integration and extraction capabilities of the FPN module to enhance the model’s performance in handling fine-grained text analysis tasks. We use the pre-trained DeBERTa-V3-Large model provided by Hugging Face as the pre-processing module for text information. This pre-trained model has a 24-layer Transformer, which can process long texts, more accurately model inter-word dependencies, capture long-distance dependencies in text, and obtain efective features for classification. In order to better combine global and detailed features, we use FPN as a feature enhancement module. FPN can fuse feature information at diferent levels in DeBERTa-V3-Large, enhance the model’s ability to understand global information, improve the utilization of features, and provide more efective features for the classifier. For our model, we used the features of the 12th and 18th layers for fusion. These two levels are chosen because they not only ensure the adequacy of feature extraction, but also combine feature information at diferent levels to provide efective information for further fusion. They were processed by Conv1d for deep feature extraction and concatenated before being passed to the Max Pooling layer to enhance key features. Finally, the features were passed to the classifier, where a dropout layer was added to prevent overfitting. Finally, the features were mapped to the classification results through a fully connected layer to achieve classification of six types of human-machine collaborative texts.
In this study, we used the pre-trained DeBERTa-V3-Large model as the basis for text feature extraction. Its large resource consumption also provides very impressive text feature extraction capabilities, so it is necessary to select the most efective training scheme in combination with time and computing resources. During the training process, we selected a batch size of 2, a learning rate of 1e-5, and a total of 3 training cycles to ensure that the model can efectively learn text features and avoid overfitting. In addition, we used the AdamW optimizer, which was selected for its optimization efect in deep learning model training, especially in dealing with gradient sparsity and weight decay. To ensure the repeatability of the experiment, we set a fixed random seed, and all experiments were conducted in a computing environment equipped with NVIDIA GeForce RTX 3090 and Intel(R) Core(TM) i9-10900K.
The main experimental process includes three stages: data preparation, model training, and performance evaluation. First, in the data preparation stage, the dataset is preprocessed, including text cleaning, label correspondence, etc. In addition, the dataset is divided into a training set and a validation set. During the model training phase, the model is iteratively learned on the training set. At the end of each cycle, we evaluate the performance of the model on the validation set to monitor whether there is overfitting during the training process. Finally, in the performance evaluation phase, we use standard classification metrics such as accuracy, recall, and F1 to evaluate the model. We pay special attention to the performance of the model on an independent test set to verify its generalization ability in practical applications. Through this series of meticulous and rigorous experimental processes, we ensure the accuracy and practicality of the research results.
In order to comprehensively evaluate the performance of our proposed DeBERTa-FPN, we selected a series of indicators, including F1, recall, and Accuracy. These indicators not only reflect the overall performance of the model, but also provide diferent performance evaluation perspectives to help us understand the performance of the model in specific aspects.
The specific performance metrics are as follows:
Accuracy represents the proportion of samples correctly predicted by the model to the total samples, which can intuitively reflect the overall prediction accuracy. It is calculated as:
Accuracy =
+ + + + (1)
Among them, TP stands for True Positive, that is, the number of correctly judged positive examples, TN stands for True Negative, that is, the number of correctly judged negative examples, FN stands for False Negative, that is, the number of positive examples that are not judged correctly, and FP stands for False Positive, that is, the number of negative examples that are mistaken for positive examples.
Recall represents the proportion of samples that are actually positive and are correctly predicted. It measures the ability of the model to predict correctly. It is calculated as:
F1 score is the harmonic mean of precision and recall.It is calculated as:
Recall =
+
Where
Precision = (4)
+
Through these evaluation indicators, we can learn to evaluate the overall performance of the model and conduct in-depth analysis of the specific capabilities of the model from multiple dimensions. These evaluation indicators can also objectively measure the performance diference between our proposed model and other models, ensure the comprehensiveness and reliability of the evaluation results, and lay a solid foundation for future model optimization and application. Through the above indicators, we compare with the baseline model. Compared with the baseline model, our model achieved 54.49%, 54.40%, and 62.89% in Recall, F1 Score, and Accuracy, respectively, which are 12%, 13%, and 10% higher than the baseline model. The evaluation results under the evaluation set are shown in Table 2.
Approach Baseline DeBERTa-FPN
Accuracy (%) F1 Score (%) Recall (%)
In this study, we proposed and implemented a hybrid neural network model that combines DeBERTaV3-Large and FPN, aiming to improve the accuracy of human-machine collaborative text classification. The results show that the Deberta-FPN hybrid model proposed in this study has achieved satisfactory results in text classification tasks. It has achieved impressive results in multiple indicators. Compared with the baseline model, the Recall indicator is improved by 12%, the F1 indicator is improved by 13%, and the Accuracy is improved by 10%, which shows the efectiveness of the model structure and can achieve excellent performance in related tasks. At the same time, we also know that there are still many aspects of this model that can be improved. In future work, we will continue to improve the method and strive to obtain better results in the task of human-machine collaborative text classification.
This work was supported by grants from the Guangdong-Foshan Joint Fund Project (No. 2022A1515140096) and Open Fund for Key Laboratory of Food Intelligent Manufacturing in Guangdong Province (No. GPKLIFM-KF-202305).
During the preparation of this work, the authors used DeepSeek-R1 in order to: Grammar and spelling check. After using this tool, the authors reviewed and edited the content as needed. full responsibility for the publication’s content.