The rapid advancements of Large Language Models (LLMs) make it increasingly challenging to distinguish between human-written and machine-generated texts, which raises concerns regarding their potential misuse. This paper describes our submission to the PAN: Generative AI Authorship 2024 verification task, which involves identifying the human-authored text from a pair of texts, one written by a human and the other by an LLM. Our approach is based on the assumption that LLMs use a distinct vocabulary. We propose a simple and interpretable method using non-neural machine learning classifiers with lexical features. We evaluate several classification models and feature sets on a validation split and find logistic regression and SVM models using tf-idf feature vectors to be highly efective. Our submissions ofer a more efective alternative to all baseline approaches while also being more eficient and interpretable.
With the rapid advancements of Large Language Models (LLMs), distinguishing between human-written
and machine-generated texts becomes more and more challenging. As a result, the need for reliable
authorship verification methods becomes even more pressing. The ability to distinguish between
human-written and machine-generated texts is crucial for various applications, such as plagiarism
detection [
In this paper, we present our submission to the PAN shared task, where we address the generative
authorship verification problem using non-neural machine learning classifiers based on lexical features.
Our decision to employ non-neural models is motivated by the observation that simple models are often
overlooked in recent research, despite their proven efectiveness and their ability to serve as eficient
baselines for comparison with more complex models [
In our work, we experimented with three classification models and two lexical feature sets. We found logistic regression and SVM models using tf-idf feature vectors are highly efective for the task. Motivated by the performance of our approach, we conducted a qualitative analysis of the most significant lexical features to test our hypothesis that LLMs employ a distinct vocabulary. Our analysis revealed that there is a small set of words that can indicate whether a text is written by an LLM. Overall, our approach ofers a more efective alternative to all baseline approaches while also being more eficient and interpretable.
The remainder of this paper is structured as follows. In Section 2, we provide background information on the PAN: Generative AI Authorship Verification task and review the related work. In Section 3, we describe our system and the components of our submission. In Section 4, we present the results of our submission. Section 5 provides a qualitative analysis of the most important lexical features. We conclude with a discussion of our results in Section 6.
Task Description The PAN: Generative AI Authorship Verification task is organized in collaboration with the Voight-Kampf Task at the ELOQUENT Lab in a builder-breaker style. PAN participants build systems to tell human and machine-generated texts apart, while ELOQUENT participants investigate novel text generation and obfuscation methods to avoid detection. The task is defined as follows:
Given two texts, one authored by a human, one by a machine: pick out the human.
More formally, given a pair of texts (1, 2), one of which is written by a human and the other by an LLM, the system must output a confidence score ∈ [0.0, 1.0]. A score < 0.5 indicates that text 1 is believed to be human-authored, while a score > 0.5 indicates that text 2 is believed to be human-authored. A score of exactly 0.5 means the case is undecidable.
Dataset The task participants were provided with a training dataset of 1,359 U.S. news articles. To ensure that the articles were human-authored, the task organizers collected the articles from Google News, focusing on the period before the release of GPT-3.5. The articles were summarized using GPT-4-Turbo, and the summaries were used as input for 13 downstream LLMs to generate new articles. The dataset consists of pairs of articles, one human-authored and one LLM-generated, and is split into training, validation, and test sets.
To further test the robustness of submissions, the task organizers provided additional test datasets, each applying a diferent obfuscation technique to the original test dataset. The obfuscation techniques include switching the text encoding, prompting the LLMs to generate German instead of English, using contrastive decoding, cropping the text to 35 words, etc. In total, 65 diferent test datasets were created by obfuscation, with ELOQUENT providing another five.
Scoring Function As follows from the task description (cf. Section 2), the generative authorship verification task is formulated as a pairwise classification problem. Given a pair of texts (1, 2), the goal is to determine which text is human-authored. However, we approach this task as a pointwise binary classification problem. That is, given a text , we aim to predict the probability (human|) that the text is human-authored.
By definition, the probability (human|) is equal to 1 − (LLM|). Given that we need to predict the probability that 1 is human-authored while taking into account 2, we average the probabilities of the first text being written by a human and the second text not being written by a human to obtain the ifnal score (human|1): (human|1) = (human|1) + 1 − (LLM|2) 2 (1) Feature Extraction To capture the distinctive vocabulary of LLM-generated texts, we use a bag-ofwords model to represent the texts. We experiment with two feature sets: term frequencies and tf-idf values for all tokens in the training dataset.
Classification Models We experiment with three classifiers: Multinomial Naive Bayes, logistic regression, and a support vector machine (SVM) with a linear kernel. We test the classifiers with both term frequencies and tf-idf values to identify the most efective model and feature combination. Model and Feature Selection To evaluate the performance of the diferent models and feature sets, we use 100 samples from the training dataset as a validation split. The results of the validation are used to select the most efective model and feature combination.
Table 1 shows the accuracy achieved on the validation split for each model. Overall, logistic regression and SVM are more efective than multinomial Naive Bayes. The diferences in efectiveness for diferent feature sets for logistic regression and SVM are minimal. Interestingly, the performance of multinomial naive Bayes is significantly better using raw term frequencies compared to tf-idf values.
Evaluation Setup The PAN: Generative AI Authorship Verification task employed the TIRA platform
[
The arithmetic mean of all metrics is used to rank the submissions.
Baselines The task organizers provided oficial baselines for comparison, which are based on the
performance of various approaches to the task of authorship verification. The baselines include a
simple text length classifier, PPMd Compression-based Cosine [
Evaluation Results Table 2 presents the evaluation results of our submissions to the task, along with the oficial baselines and summary statistics of all submissions. Our best performing submission (SVM) outperforms all oficial baselines across all metrics, with the other two submissions (Multinomial Naive Bayes and Logistic Regression) not outperforming only the Binoculars baseline for the algorithmic mean of all metrics (0.965 vs. 0.956 and 0.958 respectively).
Table 3 shows the summarized results averaged (arithmetic mean) over 10 obfuscated variants of the test dataset. Each dataset variant applies one potential technique to measure the robustness of authorship verification approaches (cf. Section 2). The results show that all our submissions are robust to the obfuscation techniques, as the performance does not drop significantly compared to the baseline approaches. For example, the minimum achieved score for our best submission (SVM) is 0.832, while the minimum score for the best baseline (Binoculars) is 0.342.
Overall, our approach demonstrates that simple and interpretable models can be highly efective for the task of generative authorship verification. The results suggest that the distinctive vocabulary used by LLMs can indeed be efectively captured using simple lexical features and machine learning classifiers. Moreover, our submissions showed to be robust to obfuscation techniques, making them a promising alternative to more complex and computationally expensive methods. litrcea itrcpoanem izsphaedem licscuonon ittsnga lidnehea rrscsedoeun iiszhpagnem liitsghhhg liiitghhgnhg llscgneeah ittcenommm iiiftcsangn irreendm liitghedhhg rsseeaum iiiltscnoapm itsdam rssxeedep ttsdae iszshaeepm rseuen ilircsneee tttsenaem lcyage soend isad lirreae irnognm itshng tsep onw itb oga tgo syunda tfrneanoo sanw ifryad indd skeda tsyeuad yoandm llryea ysa troew tryshdau sysa syedneadw ltdo
In addition to the quantitative evaluation of our submissions, we conducted a qualitative analysis of the most important lexical features identified by the models. This analysis aims to highlight key tokens that contribute to distinguishing between human-written and LLM-generated texts.
The implementation of the multinomial Naive Bayes model allows us to extract the log probabilities of each token belonging to the human-written and LLM-generated classes. By comparing these probabilities, we can identify the tokens that contribute most to the classification decision. We use the following equation to calculate the diference in log probabilities for each token in the feature set: log_dif () = log( (|LLM)) − log( (|human)) (2)
The log diference values are then sorted in descending order to identify the tokens with the largest diferences. The resulting values are interpreted as the importance of each token in distinguishing between human-written and LLM-generated texts. Positive values indicate higher probabilities for LLMgenerated texts, while negative values indicate higher probabilities for human-written texts. Figure 1 presents the top 50 tokens with the largest diferences in log probabilities for the multinomial Naive Bayes model. Here, we observe that LLM-generated texts frequently use specific terms such as “article”, “importance”, “emphasized”, “context”, and “despite”. These terms often relate to structured and formal writing, which is often characteristic of LLM-generated content. On the other hand, human-written texts show a higher probability of tokens related to everyday language and temporal expressions such as “told”, “says”, “asked”, “wrote”, and “really”. These tokens indicate a more narrative and less formal style typical of human writing. The frequent use of days of the week such as “Wednesday”, “Thursday”, and “Friday” and terms like “afternoon” and “morning” in human-written texts can be attributed to their common use in chronological events or planning. Humans often refer to specific days when recounting events, discussing plans, or setting contexts within their narratives. This is particularly relevant in our news articles dataset, where providing temporal context is essential for accurate and engaging reporting. The word “told” is particularly prominent in human-written texts, as it is frequently used in direct and indirect speech, which is also common in news articles. In contrast, LLM-generated texts often prioritize structured content delivery and formal exposition over narrative elements, resulting in frequent use of terms such as “emphasized”, “stating”, and “highlights”. The term “conclusion” is also prevalent in LLM-generated texts, indicating a structured and formal writing style that often includes a summary or final remarks, which is uncommon in human-written news articles.
significant article importance despite safety incident expressed emphasized challenges stating actions ensure potential conclusion efforts stated concerns continues ongoing impact
Logistic Regression significant despite importance expressed emphasized stating continues actions conclusion ensure incident challenges support individuals ongoing stated commitment highlighted article expected 0.0 0.5 1.0 1.5 2.0 2.5 0.0 0.5 1.0 1.5 2.0 2.5
In this paper, we presented our submission to the PAN: Generative AI Authorship Verification task. Our approach is based on the assumption that LLMs use a particular vocabulary, which can be captured using lexical features. We experiment with three classifiers and two feature sets to identify the most efective model and feature combination. Our results show that logistic regression and SVM models using tf-idf feature vectors are highly efective for the task. We find that our submissions outperform all oficial baselines, demonstrating that simple and interpretable models can be more efective than complex and computationally expensive methods. Our qualitative analysis of the most important lexical features confirms that LLM-generated texts often contain terms distinct from human-written texts, which can be efectively captured using lexical features. The robustness of our submissions to obfuscation techniques further highlights the efectiveness of our approach. Overall, our results ofer a more efective alternative to all baseline approaches while also being more eficient and interpretable.
This work originates from a programming assignment from the “Introduction to Natural Language Processing” course at Bauhaus-Universität Weimar during the summer term of 2024. We would like to thank the teaching staf who recognized the potential of our approach and encouraged us to participate in the PAN task. Together we turned these ideas into writing. vectors, in: Data Compression Conference (DCC’06), 2006, pp. 332–341. doi:10.1109/DCC.2006. 13. [10] O. Halvani, C. Winter, L. Graner, On the usefulness of compression models for authorship verification, in: Proceedings of the 12th International Conference on Availability, Reliability and Security, ARES ’17, Association for Computing Machinery, New York, NY, USA, 2017. URL: https://doi.org/10.1145/3098954.3104050. doi:10.1145/3098954.3104050. [11] M. Koppel, J. Schler, Authorship verification as a one-class classification problem, in: Proceedings of the Twenty-First International Conference on Machine Learning, ICML ’04, Association for Computing Machinery, New York, NY, USA, 2004, p. 62. URL: https://doi.org/10.1145/1015330. 1015448. doi:10.1145/1015330.1015448. [12] J. Bevendorf, B. Stein, M. Hagen, M. Potthast, Generalizing unmasking for short texts, in: J. Burstein, C. Doran, T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota, 2019, pp. 654–659. URL: https://aclanthology.org/N19-1068. doi:10.18653/v1/N19-1068. [13] A. Hans, A. Schwarzschild, V. Cherepanova, H. Kazemi, A. Saha, M. Goldblum, J. Geiping, T. Goldstein, Spotting llms with binoculars: Zero-shot detection of machine-generated text, 2024. URL: https://arxiv.org/abs/2401.12070. arXiv:2401.12070. [14] J. Su, T. Y. Zhuo, D. Wang, P. Nakov, Detectllm: Leveraging log rank information for zero-shot detection of machine-generated text, 2023. URL: https://arxiv.org/abs/2306.05540. arXiv:2306.05540. [15] G. Bao, Y. Zhao, Z. Teng, L. Yang, Y. Zhang, Fast-detectgpt: Eficient zero-shot detection of machinegenerated text via conditional probability curvature, 2024. URL: https://arxiv.org/abs/2310.05130. arXiv:2310.05130.