With the rapid development and wide application of Large Language Models (LLMs), recognizing the diferences between human and machine-generated content has become increasingly complex. Although several classification methods have been proposed for distinguishing text sources, they still have significant shortcomings in terms of the feasibility and challenge of the task nature. In this paper, we propose a hybrid text categorization model, DeBERTa+BiLSTM+Attention (DBA), that incorporates DeBERTa, Bidirectional Long and Short-Term Memory Network (BiLSTM), and Attention mechanism, aiming at identifying the specific ways of authoring in the humancomputer collaborative generation of text. The model takes full advantage of DeBERTa's strong capability in semantic understanding, combines BiLSTM's advantage in modeling contextual sequential information, and introduces an attention mechanism to highlight key information so as to more accurately capture semantic cues in the text reflecting the characteristics of human-AI interaction. The task focuses on determining the degree of human versus AI involvement in the creation process of a text, which is classified into six categories: fully human-authored, human-initiated then machine-continued, human-authored then machine-polished, machineauthored then robotically-sourced (obfuscated), machine-authored then manually edited, and deep hybrid text. Our proposed DBA model significantly outperforms existing baseline methods (e.g., RoBERTa) in this multicategory discriminative task, fully demonstrating its power in complex semantic disambiguation and mixed writing pattern recognition.
As Natural Language Processing (NLP) continues to evolve, people are increasingly relying on Large
Language Models (LLMs) in their writing, whether they are using them for academic tasks [
CLEF 2025 Working Notes, 9 – 12 September 2025, Madrid, Spain ⋆You can use this document as the template for preparing your publication. We recommend using the latest version of the ceurart style. * Wenyin Yang.
Existing research has mainly focused on distinguishing between texts created entirely by humans
and texts generated entirely by AI, with less attention paid to the subtle diferences between human
writing styles and hybrid texts created by human-computer collaboration [
In order to fill the gaps in existing research in identifying human-machine collaborative texts, this paper explored how texts can be categorized into six diferent categories based on the nature of human and artificial intelligence (AI) contributions to the creation of the text. Completely human authorship, human-initiated then machine-continued, human-authored then machine-polished, machine-authored then robot-sourced (obfuscated), machine-authored then human-edited, and deep hybrid texts. We propose a text categorization method based on DeBERTa+BiLSTM+Attention (DBA). This approach is a hybrid architecture that fuses the pre-trained language model DeBERTa-v3-large with sequence modeling and attention mechanisms. First, the contextual semantic features of the text are extracted using DeBERTa, followed by dimensionality reduction of the high-dimensional features by linear projection. Then, a bidirectional long-short-term memory network (BiLSTM) is employed to further model the time-dependent and bidirectional contextual information in the sequence. In order for the model to better focus on the key content in the text, a pooling layer based on the attention mechanism is designed to weight the sequence features for aggregation. Finally, six types of labels are output after the classification layer. Ranking 10th in the oficial PAN 2025 test results, the three metrics experimented with resulted in Macro F1 at 52.81%, Macro Recall at 48.32%, and Macro F1 at 47.82%, all exceeding the oficial benchmarks.
The dataset for the Generative AI Detection Task (Subtask 2) [? ] @ PAN 2025 [
The validation set provided by the “Human-Computer Collaborative Text Classification Task” in PAN@CLEF plays a crucial role in testing and optimizing the author’s validation model. Its structure is consistent with the previously described format. Notably, the dataset links the label and label type, where the labels 0 to 5 are used to represent diferent types of text during prediction. The mapping between these labels and text types is as follows: {
The goal of the participants’ task was to categorize documents co-created by humans and LLM. This setup tested the model’s ability to recognize micro linguistic and stylistic nuances that typically distinguish human writing from its AI-generated counterpart. Access to the dataset was oficially regulated through the PAN, and participants had to sign up and request access using their CodaLabregistered email to ensure that the use of the data was limited to research purposes and that no redistribution occurred. This controlled distribution ensures compliance with copyright regulations and maintains the integrity of the data for academic and developmental purposes.
This study proposes a hybrid neural network architecture DBA based on the combination of pre-trained language models and sequence modeling for distinguishing the degree of human and machine generated text. The model architecture is shown in Figure. ??. The model first adopts Hugging Face’s pre-trained DeBERTa-v3-large as the backbone network, and makes full use of its powerful contextual semantic encoding capability to perform deep feature extraction on the input text to obtain a high-dimensional hidden state representation with rich semantic information.
In order to reduce the feature dimensionality and enhance the eficiency of subsequent sequence modeling, the model designs a linear projection layer to map the output of DeBERTa from 1024 to 256 dimensions. Subsequently, the projected sequence features are modeled with temporal information through a bi-directional long-short-term memory network (BiLSTM) to capture the contextual dependencies in the text. In order to adaptively aggregate the sequence information, the model introduces Attention Pooling, which weights and sums the BiLSTM outputs by calculating the attention weights to obtain a context vector representing the overall text semantics. The context vector passes through the Dropout layer to mitigate the risk of overfitting, and is finally mapped to the classification space through the fully connected layer to realize multi-category label prediction. The cross-entropy loss function is used to optimize the model during the training process to ensure the robustness and accuracy of the classification performance.
The DBA model efectively combines the semantic representation advantage of the pre-trained Transformer model with the RNN’s sensitive capture ability of temporal features, showing strong diferentiation ability and generalization performance in the Human-AI Collaborative Text Classification, and improving the recognition accuracy of the human-computer hybrid text creation process. The specific challenges of the Human-AI Collaborative Text Classification task are met at PAN@CLEF 2025, demonstrating innovative theoretical approaches and practical diferentiation capabilities.
In our experimental setup for evaluating the ability of the DBA model to distinguish between humancreated and machine-generated text, we train the model on an oficially given training dataset. The model consists of six modules: First, the contextual features of the text are extracted by DeBERTa-v3large, and then the features are downscaled using a linear projection layer. Then, BiLSTM models the contextual dependencies in the sequence, and Attention Pooling aggregates the important information to form an overall representation. Finally, the classification header is used to generate the prediction results, and Dropout is used to prevent overfitting. The experimental configuration is optimized for a 16GB graphics memory environment and uses the DeBERTa model for the text classification task. The training process adopts a round-by-round evaluation and save strategy, keeping at most one best model, with a training batch size of 8, achieving an equivalent large batch through gradient accumulation (8 steps), and turning on mixed-precision training (fp16) to reduce the memory usage and speed up the training. We set the learning rate to 2e-5, train for 3 cycles, add weight attenuation to prevent overfitting, and have clear paths for logs and model outputs, so that the overall configuration takes into account performance, stability, and resource eficiency.
Our evaluation framework has been carefully designed to rigorously evaluate the performance of the DBA model in the Human-AI Collaborative Text Classification task using the following three metrics: Accuracy, Macro F1 and Macro Recall [11][12]. These metrics provide a comprehensive measure of the model’s performance in determining the level of human and AI involvement in a text. The formula for each metric is as follows.
Accuracy measures the proportion of samples correctly predicted by the model to the total sample, with the formula:
Accuracy = TP + TN (1)
TP + TN + FP + FN where TP, TN, FP, and FN denote the true example, true inverse example, false positive example, and false inverse example, respectively.
Macro F1 is the calculation of F1 scores for each category separately and then averaged, which is able to give equal attention to each category when the categories are not balanced, according to the formula: where n is the number of categories and F1 is the F1 value of the i-th category.
Macro Recall is an average of the recall of each class, which is used to measure whether the model recognizes each class completely, with the formula:
Macro F1 = 1 ∑︁ F1
=1 Macro Recall = 1 ∑︁ Recall =1 (2) (3) where n is the number of categories and is the recall of the i-th category.
Together, these metrics are used to comprehensively assess the model’s ability to recognize diferent human and AI engagement patterns, and Macro Recall in particular emphasizes coverage of each category to avoid biasing the model toward the dominant category, a key metric for assessing the fairness and stability of multi-class classification.
Our DBA model in Subtask 2: Human-AI Collaborative Text Classification of PAN 2024Voight-Kampf Generative AI Detection 2025 shows strong performance, showing substantial efectiveness on several key metrics. As shown in Table 1, Accuracy of 61.65% is about 4% higher than the benchmark’s 57.09%, reflecting the overall ability to predict correctly. Macro Recall is 54.06%, indicating that the model has better coverage of the categories. Macro F1 is 52.81%, indicating a more balanced performance across the six diferent categories. The accuracy-related metrics about the benchmarks are 57.09% for Accuracy, 48.32% for Macro Recall, and 47.82% for Macro F1. Our model exceeds all the metrics of the benchmark, reflecting the efectiveness of the model.
In terms of competition rankings, our entry ranked 10th out of 22 participants on the oficial PAN 2025 leaderboard. Notably, the ranking exceeded all baselines on all test datasets, as detailed in the PAN 2025 rankings. This ranking emphasizes the competitive advantage of our model and its remarkable ability to discriminate in challenging environments full of diverse and complex entries. These results confirm that DBA not only represents theoretical innovation, but also demonstrates significant practical capabilities in the area of human-AI collaborative writing recognition. The model’s ability to efectively distinguish between human and machine-generated text makes it a valuable tool for complex text analysis tasks. Future work at will focus on further optimizing the model parameters, enhancing the feature engineering techniques, and expanding the diversity of the training datasets to improve the model’s generalizability and performance in diferent textual contexts. This continuous improvement aims to refine the ability of DBA for higher detection accuracy and wider range of applications in the real world.
This paper details the development and evaluation of the DBA model, which is our contribution to Subtask 2: Human-AI Collaborative Text of Voight-Kampf Generative AI Detection 2025 in PAN. Classification’s innovative contribution. In this study, we propose a hybrid architectural model that fuses DeBERTa semantic feature extraction capability, BiLSTM long-range dependency modeling capability and attention mechanism to efectively identify the generation of human-computer collaborative text. The model balances deep semantic understanding and sequence modeling, and focuses on key text segments through the attention mechanism, which improves the ability to perceive complex linguistic features. In the task of classifying six types of mixed-author text, the model outperforms the existing baseline method in several performance metrics, as shown by the accuracy (Accuracy) of 61.65%, the macro-averaged F1 score (Macro F1) of 52.81%, and the macro-averaged recall (Macro Recall) of 54.06%; in comparison, the baseline model only respectively achieved 57.09%, 47.82%, and 48.32%, respectively. This performance improvement fully verifies the efectiveness and stability of the proposed model in discriminating complex text generation methods.
Looking ahead, we will continue optimizing the DBA model across multiple dimensions. At the model level, we aim to apply more refined parameter tuning and regularization to enhance generalization. In feature engineering, we plan to fuse structured semantic and stylistic features to better capture authorship variations. At the data level, we will expand the training corpus to cover diverse text types such as news, social media, technical documents, and literature. These eforts will not only boost performance in human-AI collaborative text classification but also support the model’s transferability to other NLP tasks.
This work was supported by grants from the Guangdong-Foshan Joint Fund Project (No. 2022A1515140096) and Open Fund for Key Laboratory of Food Intelligent Manufacturing in Guangdong Province (No. GPKLIFM-KF-202305).
Duringthepreparationofthis work, theauthorsusedDeepSeek-R1inorderto: Grammarandspelling check. After using this tool, the authors reviewed and edited the content as needed. full responsibility for the publication’s content. P. Rosso, D. Spina (Eds.), Working Notes of CLEF 2025 – Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings, CEUR-WS.org, 2025. [11] M. Sokolova, G. Lapalme, A systematic analysis of performance measures for classification tasks,
Information Processing & Management 45 (2009) 427–437. [12] J. Opitz, A closer look at classification evaluation metrics and a critical reflection of common evaluation practice, Transactions of the Association for Computational Linguistics 12 (2024) 820–836.