Introduction

Authorship verification can be defined as the task of predicting whether two input texts are written by the same author [1]. For example, previous PAN labs [2,3,4] have led to the development and benchmarking of many excellent methods for human authorship verification. The recent development of Large Language Models (LLMs), however, has introduced new challenges to the field. Since the invention of transformers in 2017 [5] and the release of GPT-2 in 2019 [6], a number of LLMs are now capable of producing texts with human levels of fluency -even when generated via zero-or few-shot in-context learning (i.e. prompting). In turn, this rapid advance in LLM technology has led to a demand for tools capable of automatically detecting LLM-written texts, extending the problem of authorship identification to the analysis of machine-generated writing for the first time [7,8,9,10,8,10]. Notably, although these studies define the task of LLM detection in somewhat different ways, at a basic level, all involve distinguishing machine-written texts from human-written texts.

Building on research in this area, PAN@CLEF2024 released the shared task of Voight-Kampff Generative AI Authorship Verification (AIAV), where the LLM detection problem is reframed as a verification task: given a pair of texts, where one is written by a human and one is written by a machine, the goal is to select the text written by the human (or alternatively the machine) [4,11]. To resolve this tasks, we extended our authorial language models (ALMs) paradigm for authorship attribution [12] to AIAV. When evaluated on the TIRA dataset [13], our method outperforms all baselines methods with a mean benchmarking score of 0.979. transformer deep learning architecture, which was introduced in 2017 [5]. While LLMs consisting of millions of parameters (e.g. GPT-2) have been capable of producing texts with human-level fluency for years, the more recent development of LLMs with billions of parameters (e.g. GPT-3.5, GPT-4, Llama) has made it possible to generate text via prompting. This now allows almost anyone to easily and quickly generate machine-written texts of very high quality. Although this type of automated writing has great potential value for society, there is also considerable concern about its misuse [7,8,9,10,8,10]. While LLM security is a broad topic that requires efforts from across academia and industry to mitigate the risk of LLM misuse and abuse, LLM detection is clearly an indispensable parts of this endeavor.

LLM Detection is a family of tasks within authorship analysis that involve identifying machinewritten texts and distinguishing machine writing from human writing. From this broad definition, several more specific types of LLM detection tasks can be identified [11]. Arguably, at the most basic level, the problem is to distinguish between pairs of texts, where one text is written by a human and one text is written by a machine [11]. This task, which is the focus of the PAN@CLEF2024 shared task, is referred to as AIAV. Several solutions to AIAV have been proposed, including PPMd Compression-based Cosine, Authorship Unmasking, Binoculars, DetectLLM LRR and NPR, DetectGPT and Fast-DetectGPT, which act as the baselines for this shared task [11].

Perplexity (PPL) and perplexity-related measures have been at the core of many attempts to automate LLM detection. Perplexity is defined as the exponentiated mean log-likelihood of a text over a LLM, as described in the following formula.

𝑃 𝑃 𝐿 (𝑀, 𝑋) = 𝑒𝑥𝑝 {︃ − 1 𝑡 𝑡 ∑︁ 𝑖 𝑙𝑜𝑔 (𝑝 𝑀 (𝑥 𝑖 |𝑥 <𝑖 ))

}︃

where 𝑋 = {𝑥 1 , 𝑥 2 , ..., 𝑥 𝑡 } is the sequence of tokens (e.g., the text), 𝑡 is the length of the sequence (i.e. number of tokens), 𝑀 is the LLM, and 𝑝 𝑀 (𝑥 𝑖 |𝑥 <𝑖 ) is the predicted probability of the 𝑖 th token given an LLM and the preceding tokens in the sequence. Perplexity measures the predictability of a text over the LLM, and is commonly used in LLM training as a potential loss function and evaluation metric. The higher perplexity is, the less predictable a text is to the LLM.

Perplexity for LLM detection is a common approach because LLM-generated texts are generally assumed to be associated with a lower perplexity than human-written texts for any given LLM. This approach is especially common in hybrid LLM-detection solutions that are designed to assist humans distinguishing human-written texts from LLM-generated ones [8,14]. LLM detection via perplexity, however, also has clear limitations. From a technical standpoint, it relies heavily on using LLMs for the calculation of perplexity. More fundamentally, it is also entirely possible for human texts to be associated with relatively low perplexity scores for generic LLMs. In general, such approaches therefore appear to be overly simplistic. To mitigate the risks of prediction failure, especially avoiding false positive, where texts written by real human authors are flagged as being machine-generated, researchers have therefore expanded on this approach, for example, by incorporating a pair of pertained LLMs rather than a single LLM into their LLM detection systems [15].

Authorial Language Models (ALMs) is a paradigm for authorship analysis that relies on training a set of fine-tuned authorial models based on the available writing samples for each candidate author [12]. Unlike most previous LLM-based approaches to authorship analysis that use only one LLM [16,17], ALMs involves using multiple LLMs, one for each candidate author, to better capture authorial variation in token predictability. This makes ALMs more resilient to exceptional or extreme cases because this approach does not relying on a single LLM, while allowing for greater amounts of information to be extracted from the underlying textual data, as the LLMs are fine-tuned on a candidate-by-candidate basis. Furthermore, ALMs is also more interpretable as it can provide token-level predictability metrics for the questioned document for each candidate author. Because of these advantages, we have found that ALMs outperforms all other state-of-the-art methods (N-grams NN, BERT, and PPM) for human authorship attribution on the Blogs50 dataset, while nearly matching the performance of N-grams NN, which achieves the best results, on the CCAT50 datasets [12]. For this shared task, we have therefore modified ALMs for AIAV, as we detail in the next section.

Datasets

The PAN@CLEF2024 AIAV shared tasks involves two groups of datasets: the bootstrap group and the testing group.

The bootstrap group was open for method development. In the bootstrap group, one dataset contains 1087 texts that were generated by 13 widely-used LLMs ranging from Llama [18] to GPT-4 [19], together with 1087 texts that were authored by humans. Regardless of the author, texts in bootstrap dataset are full or trimmed news articles. In this study, we used the bootstrap datasets for the fine-tuning of authorial language models and the training of the support vector machine classifier. We then developed our method and submitted it to the tira.io [13],

Meanwhile, the testing group is retained by PAN2024 organizers and was not made available to participants in the shared task. Rather, this dataset was used to independently test the systems submitted to tira.io for assessment. Specifically, the PAN2024 organizers tested our system on the testing group of datasets, which includes one main dataset, plus nine variants of datasets that are obfuscated against AI verification methods. For the details of testing group datasets, PAN2024 organizers plan to release the basic facts and compilation details in the overview notebook [11].

System Overview

ALMs for AIAV is a version of ALMs that is tailored to the needs of AIAV shared task. ALMs for AIAV is based on the idea of using perplexity for LLM detection, where human-authored texts are assumed to have substantially higher perplexity compared to LLM-authored ones. However, this assumption has exceptions if we only consider perplexity from a single LLM: there are human-authored texts with relative low perplexity, and LLM-authored texts with relatively high perplexity, both of which undermine LLM detection using this approach. Hence, during the development of our method, we hypothesized that these exceptions could result from a lack of any attempt to represent in the styles of different LLMs, which we believe could affect LLM detection in two ways.

On the one hand, confounding variables in training corpus, such as genres, registers, and topics, can possibly distort perplexity: for example, a human-written texts in a register that is over-represented in the training corpus would tend to be associated with a relatively low perplexity, whereas an LLMauthored text written in a register that is underrepresented in the training corpus would tend to be associated with a relatively high perplexity. On the other hand, the differences in language modeling and text generating pipelines can also lead exceptional perplexity values: for example, a LLMs that uses a distinctively unique pipeline would potentially generate texts that are more unexpected to other existing LLMs and hence be associated with relatively high perplexity.

Although these issues cannot be completely eliminated, we believe these issues can be mitigated by using not one but many LLMs. By fine-tuning pre-trained LLMs to correspond to each of the potential LLMs in the detection task, we can build perplexity array that take into account the styles of different LLMs. Meanwhile, based on this perplexity array, perplexity values for each of the LLMs can be compared against one another, which further mitigates the risk from under-representing LLM styles.

Like ALMs for human authorship attribution, the first step for ALMs for AIAV is the fine-tuning of a series of pre-trained LLMs that correspond to each of the potential LLM "authors". These fine-tuned authorial language models are then used to extract a perplexity array for each pair of question texts. The perplexity arrays are then used as feature vectors in a pre-trained Support Vector Machine (SVM) classifier to decide which of the two texts is most likely written by a human. Finally, the prediction result is outputted as the 𝑖𝑠_ℎ𝑢𝑚𝑎𝑛 score, as requested by the shared tasks [11]. 𝑖𝑠_ℎ𝑢𝑚𝑎𝑛 ranges between 0 and 1, where 0 means the first text is considered human-written, and 1 means the second text is considered human-written. The workflow of ALMs for AIAV is described as flowchart in Figure 1, Figure 2, and Figure 3. The details of each step are described in the following subsections.

Fine-tuning Authorial Language Models

The first step of the ALMs for AIAV is the fine-tuning of pre-trained LLMs on the texts from each candidate author. However, in the AIAV shared task, candidates are grouped by whether it is human(e.g LLMs group and human group). Though the number of authors in human group is unclear, the number of authors in the LLM groups is specified. Therefore, we can take the LLMs listed in the bootstrap dataset [20] as potential "authors" for the fine-tuning of the authorial language models. We take 80% for each of the LLM datasets as the training data for LLM fine-tuning, and we retain the remaining 20% for use in further steps. As most of these potential models are causal language models, we choose GPT-2 base, a canonical causal language model as the pre-trained model for fine-tuning. We then fine-tune 13 GPT-2 models on the texts from 13 potential LLM "authors". For each case, we fine-tune each LLM for 20 epochs with a weight decay of 0.01, an initial learning rate of 0.00002, and a gradient accumulation step of 16.

Perplexity Array Extraction

Although perplexity is defined as the exponentiated mean log-likelihood of all tokens in a text, for efficiency purposes, we calculate perplexity based on cross entropy using the formula below:

𝑃 𝑃 𝐿 (𝑀, 𝑋) = 𝑒𝑥𝑝 {𝐶𝑟𝑜𝑠𝑠𝐸𝑛𝑡𝑟𝑜𝑝𝑦 (𝐿𝑜𝑔𝑖𝑡𝑠, 𝑋)}

Given an input text 𝑄, and a fine-tuned authorial GPT-2 model 𝑀 , we first pass 𝑄 to the GPT-2 BPE Tokenizer to extract a token sequence 𝑋. 𝑋 is then passed to 𝑀 for language modeling, whose output is 𝐿𝑜𝑔𝑖𝑡𝑠. Here 𝐿𝑜𝑔𝑖𝑡𝑠 reflects the predicted probabilities of all tokens in 𝑋, where 𝑋 represents the ground truth. Therefore, in the next step, we measure the predictability of all tokens in 𝑋 by comparing the predicted 𝐿𝑜𝑔𝑖𝑡𝑠 and the ground truth 𝑋 via cross entropy, which we calculate using 𝑡𝑜𝑟𝑐ℎ.𝑛𝑛.𝐶𝑟𝑜𝑠𝑠𝐸𝑛𝑡𝑟𝑜𝑝𝑦𝐿𝑜𝑠𝑠 from PyTorch. Finally, we obtain the perplexity of 𝑄 under 𝑀 by exponentiated cross entropy.

For each input text 𝑄, we calculate its perplexity under each of the 13 fine-tuned, authorial language models. We store these perplexity values in an 13*1 array, which we flag as a perplexity array for input text 𝑄. The perplexity array is then used in the next step as a feature array to make a prediction on each questioned text pair.

Authorship Prediction via Support Vector Machine

Given an input question text pair 𝑄 1 and 𝑄 2 , we first extract their perplexity arrays from the 13 authorial language models. We then move to authorship prediction based on the two perplexity arrays. For this stage, we trained an SVM using a reconstituted dataset composed of paired perplexity arrays from the human data in bootstrap dataset and the remaining 20% of LLM-generated texts that we retained during ALMs fine-tuning. In this dataset, we paired perplexity arrays following the description of the shared tasks, where, for each pair of texts, we guarantee that one text is human authored and the other text is LLM-generated. We trained the SVM classifier with a radial basis function kernel, a regularization parameter of 1.0, and a tolerance for stopping criterion of 0.001. We did not impose hard training steps or epochs for the SVM classifier.

Results

We submitted our ALMs for AIAV as docker contained software for the benchmarking on the tira.io group of testing datasets [13]. During testing, our method was labeled as "greasy-chest". Table 1 shows the results, initially pre-filled with the official baselines provided by the PAN organizers and summary statistics for all submissions to the shared task (i.e., the maximum, median, minimum, and 95-th, 75-th, and 25-th percentiles over all submissions to the task). We find that our method beat all existing baselines for all evaluation measures and performs in the top 25% for all submissions to this shared task.

In addition, Table 2 shows the summarized results averaged (arithmetic mean) over all 10 variants of the test dataset. Each dataset variant applies one potential technique to measure the robustness of the AIAV systems, including but not limited to switching the text encoding, translating the text, switching the domain, and manual obfuscation by humans. Our method achieves a median score of 0.935 over the 9 variants, which also surpasses all existing baselines and is among the top 25% of all submissions to this shared task. However, we also notice that our method has a relative low minimum score over 9 variants, suggesting that further investigations are needed for the most challenging dataset variant. Our submission(as team "jaha") scores 17th out of 30 on the leaderboard with a ranking score of 0.683.

Conclusion

In this paper, we have introduced ALMs For AIAV, a generative AI verification method that utilizes fine-tuned authorial language models and a support vector machine classifier to predict which text is written by human in a human-and machine-written text pair. We found that our method has a score of 0.979 in ROC-AUC Brier C@1 F1 and F0.5, which is better than all baseline methods. We attribute the excellent performance of ALMs for AIAV's to the ALMs paradigm, which uses many fine-tuned authorial language models, providing greater flexibility and resilience than is possible if only one LLM is used, as has often been the case in previous methods for authorship identification. Future research may focus on the improvement of authorial prediction methods and use a regressor instead of classifier as proposed in this paper. In addition, it is also worthwhile to experiment with in-context learning (ICL) of LLMs with billions of parameters to see whether ICL could be an effective replacement for the fine-tuning approach we have taken, since ICL would potentially enable a few-shot implementation of our method.