Large Language Models (LLMs) have demonstrated remarkable capabilities in generating text that closely resembles human writing across wide range of styles and genres. However, such capabilities are prone to potential misuse, such as fake news generation, spam email creation, and misuse in academic assignments. Hence, it is essential to build automated approaches capable of distinguishing between artificially generated text and human-authored text. In this paper, we propose an architecture which includes three components: transformer model, token-level features, and state-of-the-art embeddings. This approach achieves a mean score of 0.973 on PAN dataset, demonstrating its efectiveness in identifying AI-generated text.
The domain of Natural Language Generation (NLG) is witnessing a remarkable transformation with
the emergence of Large Language Models (LLMs) such as Generative Pre-trained Transformer (GPT-4)
[
Diverse modeling strategies, ranging from simple statistical techniques to cutting-edge
Transformerbased architectures [
Recently, there has been a notable increase in research focused on zero-shot detection techniques for AI-generated text. These methods predominantly involve the analysis of outputs from LLMs, utilizing features such as entropy, log-probability scores, and perplexity [14, 15, 16, 17] to help distinguish between human-authored and AI-generated content. However, the zero-shot detection methods can be more efective when there is direct access to the internal specifics of the LLM that generated the text. This limits the robustness of zero-shot detection methods across diferent scenarios [18, 19].
To boost this area of research further, PAN 2024 workshop introduced ‘generative AI authorship verification’ shared task, which focuses on determining whether a given text is human-authored or AI-generated. In response to this challenge, we proposed an architecture which leverages a pre-trained RoBERTa-base AI-text detector [20], a Bidirectional Long Short-Term Memory (BiLSTM) attention layer for processing token-level perplexity and word-frequency features, and a state-of-the-art EmbEddings from bidirEctional Encoder rEpresentations (E5) model [21]. Our experiments show that our proposed approach outperforms several state-of-the-art approaches based on established metrics.
The PAN dataset, released by the PAN shared task organizers, contains both human-authored and AI-generated text. It includes a total of 15,190 samples, consisting of 1,087 human-authored texts and 14,103 AI-generated texts produced using thirteen diferent LLMs, namely: (i) alpaca-7b, (ii) bigscience-bloomz-7b1, (iii) chavinlo-alpaca-13b, (iv) gemini-pro, (v) meta-llama-llama-2-70b-chat-hf, (vi) meta-llama-llama-2-7b-chat-hf, (vii) mistralai-mistral-7b-instruct-v0.2, (viii) mistralai-mistral-8X7binstruct-v0.1, (ix) qwen-qwen1.5-72b-chat-8bit, (x) text-bison-002, (xi) vicgalle-gpt2-open-instruct-v1, (xii) gpt-3.5-turbo-0125, and (xiii) gpt-4-turbo-preview. More details about the dataset can be found in the PAN overview paper [22].
Our framework consists of three major components for feature representations of input text: (i) RoBERTa base Open AI detector [20] produces document-level representations that capture the overall content’s meaning;
(ii). Token-level features[23] are extracted from various GPT2 variants (DistilGPT2, GPT-2, GPT-2 Medium, and GPT-2 Large) to analyze both the predictability of the word sequence and word frequency. The token level features include: log-probability of the observed token, log-probability of the most likely token, entropy of the token probability distribution at a given position, and word frequency. A BiLSTM with attention layer processes these token-level features to create combined document-level representations; (iii). Document-level feature representation are also extracted using the E5 model.
Finally, the three document-level representations are concatenated into a single representation. The combined representation is then fed into a fully connected layer to generate the final probabilities. Baseline Binoculars [17] Baseline Fast-DetectGPT (Mistral) Baseline Fast-DetectGPT [16]
ROC-AUC Brier C@1 0.987 3.1. Results In this section, we present an evaluation of our AI-generated text detection experiments. In our experiments, 20% of the training data was used for validation. For test run submissions, the validation set was merged back with the training set. We report results using well established metrics [22], and compare our model’s performance with state-of-the-art models, as shown in Table 1. After predicting the label for each input, we produced the final scores of each text pair as recommended by the organizers [22]. The results indicate that our method surpasses the state-of-the-art models, achieving a modest improvement of around 1% over the mean score of Binoculars model.
In this paper, we described our submission to the PAN shared task for detecting the generative AI content. Our experiments demonstrated that our generative AI text detection approach performs well compared to other state-of-the-art approaches in this domain. For future work, we aim to enhance the generalizability of our model by testing it on diverse datasets to evaluate its robustness in real-world applications. [14] J. Su, T. Y. Zhuo, D. Wang, P. Nakov, Detectllm: Leveraging log rank information for zero-shot detection of machine-generated text, arXiv preprint arXiv:2306.05540 (2023). [15] E. Mitchell, Y. Lee, A. Khazatsky, C. D. Manning, C. Finn, Detectgpt: Zero-shot machine-generated text detection using probability curvature, arXiv preprint arXiv:2301.11305 (2023). [16] G. Bao, Y. Zhao, Z. Teng, L. Yang, Y. Zhang, Fast-detectgpt: Eficient zero-shot detection of machine-generated text via conditional probability curvature, arXiv preprint arXiv:2310.05130 (2023). [17] A. Hans, A. Schwarzschild, V. Cherepanova, H. Kazemi, A. Saha, M. Goldblum, J. Geiping, T. Goldstein, Spotting llms with binoculars: Zero-shot detection of machine-generated text, arXiv preprint arXiv:2401.12070 (2024). [18] Y.-F. Zhang, Z. Zhang, L. Wang, R. Jin, Assaying on the robustness of zero-shot machine-generated text detectors, arXiv preprint arXiv:2312.12918 (2023). [19] X. Yang, L. Pan, X. Zhao, H. Chen, L. Petzold, W. Y. Wang, W. Cheng, A survey on detection of llms-generated content, arXiv preprint arXiv:2310.15654 (2023). [20] I. Solaiman, M. Brundage, J. Clark, A. Askell, A. Herbert-Voss, J. Wu, A. Radford, G. Krueger, J. W.
Kim, S. Kreps, et al., Release strategies and the social impacts of language models, arXiv preprint arXiv:1908.09203 (2019). [21] L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, D. Jiang, R. Majumder, F. Wei, Text embeddings by weakly-supervised contrastive pre-training, arXiv preprint arXiv:2212.03533 (2022). [22] J. Bevendorf, M. Wiegmann, J. Karlgren, L. Dürlich, E. Gogoulou, A. Talman, E. Stamatatos, M. Potthast, B. Stein, Overview of the “Voight-Kampf” Generative AI Authorship Verification Task at PAN and ELOQUENT 2024, in: G. Faggioli, N. Ferro, P. Galuščáková, A. G. S. de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings, CEUR-WS.org, 2024. [23] P. Przybyła, N. Duran-Silva, S. Egea-Gómez, I’ve seen things you machines wouldn’t believe: Measuring content predictability to identify automatically-generated text, in: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2023). CEUR Workshop Proceedings, CEUR-WS, Jaén, Spain, 2023.