The task of generative AI authorship verification aims to determine if a text is written by a human or generated by AI. In this paper, we treat this task as a binary classification problem and introduce a method called TriSentence Analysis (TSA). TSA captures fine-grained contextual information, enhancing the model's ability to identify the text's source. Additionally, we incorporate the MPU method to improve the model's eficiency and diferentiation for short texts.Finally, we integrated these methods into a pre-trained BERT model. On the test set, our performance metrics for the Minimum, 25-th Quantile, Median, 75-th Quantile, and Maximum scores are 0.883, 0.936, 0.976, 0.989, and 0.999, respectively.
The rapid development of artificial intelligence technology has led to significant advancements in large language models (LLMs) for text generation. These models can produce well-structured and grammatically correct text and are widely used in various fields, including advertising, news writing, storytelling, and code generation. However, some malicious individuals misuse LLMs for harmful purposes, such as creating credible fake news or cheating, which misleads readers and has severe negative societal impacts. Therefore, distinguishing AI-generated text from human-written text to prevent abuse has become urgent. Detection methods for text generated by large models can be divided into two main categories:
The first method is zero-shot detection, which identifies AI-generated text by directly accessing the source model that created the text. This method does not need pre-trained datasets but uses the source model’s output logits or loss values to determine if the text is machine-generated. Examples include the methods by Mitchell et al. [11] and Yang et al. [12] The advantage of zero-shot detection is that it does not require large amounts of training data and can be applied directly to new text. However, its disadvantage is that it depends on the performance of the source model or a proxy model. If there is a significant diference between the proxy model and the source model, the detection efectiveness may be low.
The second method is based on deep neural network (DNN) classifiers, which detect human-written and AI-generated text through supervised training models. The advantage of this method is that it can improve detection performance through large amounts of training data. For example, the method by Guo et al. [13]. However, DNN-based classifiers have high data requirements, poor generalization ability [14], and the trained classifiers are vulnerable to backdoor attacks [15] and adversarial attacks [16].
The dataset for the author verification task in generative AI, provided by ELOQUENT and PAN participants, includes various types of texts such as news articles, Wikipedia summaries, and fan fiction. It covers real and fake news articles from multiple US headlines in 2021. The dataset consists of 14 JSONL files, each containing 24 topics and 1087 articles. One file is written by humans, while the other 13 files are generated by diferent large language models. Each file corresponds to the same row ID, indicating the content pertains to the same topic. In total, the 14 files contain 15,218 articles, as shown in Table 1. Label 1 indicates text written by humans, while label 0 indicates text generated by large language models.
We integrated and categorized the dataset provided by PAN into a new dataset named "combine." This dataset consists of two columns: "text" and "label." The "text" column represents the content of the text, while the "label" column indicates the source of the text, with human-written texts labeled as 1 and machine-generated texts labeled as 0. The combine dataset contains a total of 15,218 articles. We used 80% of the data labeled as 0 and 1 for training and the remaining 20% for validation, resulting in 12,174 samples for training and 3,044 samples for validation.
Our objective is to split long texts into multiple short texts and determine whether the original text was generated by AI or written by humans by analyzing features extracted from each short text.To achieve this, we propose a deep learning-based text classification method called Tri-Sentence Analysis (TSA). TSA works by dividing long texts into short texts, each containing three sentences, and analyzing each independently.The combined results of these analyses are used for the final classification of the entire text.This approach aims to capture fine-grained contextual information more efectively, thereby improving classification accuracy. Additionally, segmenting the text reduces the burden of long texts on the model, enhancing its stability when processing lengthy texts.
During the prediction phase, we use TSA to process new input texts in a similar manner.Each group of three sentences is treated as a short text and input into the trained BERT model. The results of each short text prediction are averaged with weights to classify the original text. During testing, if we need to determine which of two texts is closer to being human-written, we apply the same method to obtain the prediction values for each text. The text with a prediction value closer to human-written is classified as such. We also want to enhance the model’s ability to classify short texts. Tian et al. [8] proposed a new loss function called MPU, which improves the recognition and diferentiation of AI-generated short texts.Therefore, we modified the model’s loss function to MPU [ 8] to improve classification accuracy. Our system architecture is shown in Figure 1.
In this work, we selected the BERT-base-uncased model, featuring 12 layers, 768 hidden units, 12 attention heads, and 110M parameters. The maximum length of the encoder was configured to 512, with a batch size of 32. Meanwhile, we employed MPU [8] as the loss function, utilizing the Adam optimizer with a learning rate of 5e-5. Our experiments were conducted on an A800 server. The optimal performance was achieved after 13 epochs of training.
To evaluate the performance of our model, we used the evaluation platform provided by PAN, which includes the following metrics: • AUC: the conventional area under the curve score. • c@1: rewards systems that leave complicated problems unanswered [17]. • F_0.5u: focus on deciding same-author cases correctly [18]. • F1-score: harmonic way of combining the precision, and recall of the model [19]. • Brier: Brier Score evaluates the accuracy of probabilistic predictions [20].
• Mean: The arithmetic mean of all the metrics above.
We finally submitted the model to TIRA [ 10] for execution to obtain the final metrics. Our model, charitable-mole_v3, performed exceptionally well in the PAN 2024 Generative AI Authorship Verification task. Table 2 presents the outstanding performance of charitable-mole_v3 across various metrics: ROCAUC of 0.991, Brier of 0.991, C@1 of 0.991, F1 of 0.99, F0.5u of 0.989, and Mean of 0.99. Our model outperformed the oficial baselines across all metrics and maintained strong competitiveness in the 95th percentile among participants.
Table 3 further illustrates the average accuracy of charitable-mole_v3 across diferent dataset variants, particularly on the test sets of nine variants. Our model’s Minimum value across all variants was 0.883, with the 25-th and 75-th Quantile at 0.936 and 0.989, respectively, a Median of 0.976, and a Maximum value of 0.999. These results significantly surpass those of all oficial baselines.
Compared to the quantile results of other participants, charitable-mole_v3 is close to or surpasses the models in the 95-th quantile in most metrics and exceeds the 75-th quantile models in all metrics, demonstrating strong competitiveness. This further proves the excellent performance of our model in the PAN 2024 Generative AI Authorship Verification task.
This paper details our achievements in the PAN2024 generative AI author verification task. We proposed a text classification method based on the BERT pre-trained model, called Tri-Sentence Analysis (TSA). The TSA method can capture more fine-grained contextual information, thereby improving the accuracy of text classification. It better understands the semantic relationships and consistency between sentences, enhancing the model’s robustness in handling long texts. Additionally, we integrated the MPU method to improve the eficiency of distinguishing short texts. Ultimately, our method performed excellently in the PAN2024 generative AI author verification test set. The Minimum, 25-th Quantile, Median, 75-th Quantile, and Maximum values were 0.883, 0.936, 0.976, 0.989, and 0.999, respectively. In the future, we plan to further improve this method and explore its potential applications in a broader range of natural language processing tasks to achieve higher detection eficiency and wider application. Analysis, and Generative AI Authorship Verification, in: L. Goeuriot, P. Mulhem, G. Quénot, D. Schwab, L. Soulier, G. M. D. Nunzio, P. Galuščáková, A. G. S. de Herrera, G. Faggioli, N. Ferro (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Fifteenth International Conference of the CLEF Association (CLEF 2024), Lecture Notes in Computer Science, Springer, Berlin Heidelberg New York, 2024. [5] J. Karlgren, L. Dürlich, E. Gogoulou, L. Guillou, J. Nivre, M. Sahlgren, A. Talman, Eloquent clef shared tasks for evaluation of generative language model quality, in: European Conference on Information Retrieval, Springer, 2024, pp. 459–465. [6] A. Hans, A. Schwarzschild, V. Cherepanova, H. Kazemi, A. Saha, M. Goldblum, J. Geiping, T. Goldstein, Spotting llms with binoculars: Zero-shot detection of machine-generated text, arXiv preprint arXiv:2401.12070 (2024). [7] G. Bao, Y. Zhao, Z. Teng, L. Yang, Y. Zhang, Fast-detectgpt: Eficient zero-shot detection of machine-generated text via conditional probability curvature, arXiv preprint arXiv:2310.05130 (2023). [8] Y. Tian, H. Chen, X. Wang, Z. Bai, Q. Zhang, R. Li, C. Xu, Y. Wang, Multiscale positive-unlabeled detection of ai-generated texts, arXiv preprint arXiv:2305.18149 (2023). [9] J. Lee, K. Toutanova, Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 3 (2018) 8. [10] M. Fröbe, M. Wiegmann, N. Kolyada, B. Grahm, T. Elstner, F. Loebe, M. Hagen, B. Stein, M. Potthast, Continuous Integration for Reproducible Shared Tasks with TIRA.io, in: J. Kamps, L. Goeuriot, F. Crestani, M. Maistro, H. Joho, B. Davis, C. Gurrin, U. Kruschwitz, A. Caputo (Eds.), Advances in Information Retrieval. 45th European Conference on IR Research (ECIR 2023), Lecture Notes in Computer Science, Springer, Berlin Heidelberg New York, 2023, pp. 236–241. doi:10.1007/ 978-3-031-28241-6_20. [11] E. Mitchell, Y. Lee, A. Khazatsky, C. D. Manning, C. Finn, Detectgpt: Zero-shot machine-generated text detection using probability curvature, in: International Conference on Machine Learning, PMLR, 2023, pp. 24950–24962. [12] X. Yang, W. Cheng, Y. Wu, L. Petzold, W. Y. Wang, H. Chen, Dna-gpt: Divergent n-gram analysis for training-free detection of gpt-generated text, arXiv preprint arXiv:2305.17359 (2023). [13] B. Guo, X. Zhang, Z. Wang, M. Jiang, J. Nie, Y. Ding, J. Yue, Y. Wu, How close is chatgpt to human experts? comparison corpus, evaluation, and detection, arXiv preprint arXiv:2301.07597 (2023). [14] A. Uchendu, T. Le, K. Shu, D. Lee, Authorship attribution for neural text generation, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 8384–8395. [15] F. Qi, M. Li, Y. Chen, Z. Zhang, Z. Liu, Y. Wang, M. Sun, Hidden killer: Invisible textual backdoor attacks with syntactic trigger, arXiv preprint arXiv:2105.12400 (2021). [16] X. He, X. Shen, Z. Chen, M. Backes, Y. Zhang, Mgtbench: Benchmarking machine-generated text detection, arXiv preprint arXiv:2303.14822 (2023). [17] A. Peñas Padilla, Á. Rodrigo Yuste, A simple measure to assess non-response (2011). [18] J. Bevendorf, B. Stein, M. Hagen, M. Potthast, Generalizing unmasking for short texts, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 654–659. [19] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, et al., Scikit-learn: Machine learning in python, the Journal of machine Learning research 12 (2011) 2825–2830. [20] G. W. Brier, Verification of forecasts expressed in terms of probability, Monthly weather review 78 (1950) 1–3.