This paper proposes a method for fine-tuning the pre-trained language model Text-to-Text Transfer Transformer (T5) for Generative AI Authorship Verification. During the training phase, the input sequence consists of explicit instructions and training samples, while the output sequence represents the classification results in the form of "positive </s>" or "negative </s>". During inference, the model's vocabulary is restricted to "positive" and "negative", selecting the word with the highest probability as the classification result. Finally, on the test set, our performance metrics scored 0.138, 0.529, 0.744, 0.874, and 0.877 for the minimum, 25th percentile, median, 75th percentile, and maximum values, respectively.
Text classification is a fundamental research direction in NLP tasks. The aim of this direction is to determine whether two texts are written by the same person. AI Authorship Verification can be widely applied in environments where the authenticity of information needs to be verified, such as in legal proceedings and news reporting.
In the PAN 2024 AI Authorship Verification Task [1,2], our challenge is to differentiate between human-authored texts and machine-generated texts from two texts with the same topic. In this paper, we focus on using fine-tuning methods to address this problem. We first conducted thorough data preprocessing on the training set provided by the organizers, including data cleaning and character conversion steps, to ensure the quality and consistency of the data. Then, we fine-tuned a T5 model [3,4] on the preprocessed dataset to perform the text classification task. Finally, we submitted our results on TIRA.io [5] to evaluate the performance of our method in real-world applications.
Text detection in machine-generated text is an active research area, primarily employing three distinct approaches to differentiate between human-written text and machine-generated text.
The first approach is traditional statistical methods, which identify anomalies by analyzing statistical characteristics of text samples. For example, the statistical method called the Giant Language Model Test Room (GLTR), designed by Gehrmann et al. [6] This method comprises three testing steps: Tests 1 and 2 examine whether generated words are sampled from the top of the distribution, while Test 3 verifies if the system is overly confident in its next prediction due to familiarity with previously generated contexts. Through a study involving human subjects, GLTR successfully increased the accuracy of identifying fake text from 54% to 72% without any pretraining, significantly enhancing human discernment of the genuineness of generated text.
The second approach is unsupervised learning methods, particularly zero-shot classification. This method utilizes pre-trained large language models (LLMs) to detect their own generated text or text generated by similar models. Solaiman et al. [7] proposed a baseline method that makes classification decisions by evaluating log-probabilities and corresponding thresholds. However, compared to statistical methods, the performance of zero-shot classification methods is typically inferior.
The third approach is supervised learning methods, which involve fine-tuning existing language models to create a text detector. For instance, Zeller et al. [8] utilized fine-tuning linear layers on the hidden states of the GROVER encoder to distinguish whether input text originates from the GROVER model or from human hands. However, compared to the first two methods, supervised learning methods require a significant amount of labeled data for model training, making the training process more time-consuming.
The training data set of the Generative AI Authorship Verification task is a bootstrap dataset of real and fake news articles spanning multiple 2021 U.S. news headlines. It consists of JSON files written by 13 different machine authors and 1 human author. Each file contains a list of articles, and each file contains articles of the same topic. In all files, the ID and row order of the article are the same, so the same row always corresponds to the same topic, but from different "authors". Each document contains 24 topics and 1087 articles. Considering the token length limitations of large language models, we have conducted a token count on these data. The total number of articles and the token length range in the dataset written by different authors are shown in Table 1.
For the provided training set, we initially preprocess texts authored by machines and humans. We remove all empty texts, replace all full-width characters with half-width characters, and remove spaces within the texts. Subsequently, we merge texts authored by machines and humans. Specifically, we create a text tuple ("pair": ["text", "label"]), where "text" represents the content of the article, and "label" indicates whether it is authored by a human (1 for positive, 0 for negative). We use 80% of the training dataset for training and 20% for validation, with 11,393 samples for training and 3,797 samples for validation.
Our method fine-tunes the T5 model for the text classification task. This approach transforms the text classification problem into a sequence-to-sequence (seq-to-seq) problem, enabling the model to handle and understand text data more flexibly.
During the training phase, we utilized the training dataset provided by PAN 2024 and conducted necessary preprocessing to ensure the data was suitable for model training. Since the T5 model was exposed to numerous tasks with explicit instructions or prompts during its pre-training phase, adding a prompt can fully leverage the model's pre-training knowledge, thereby improving its performance on specific tasks. Consequently, the input sequence consists of two parts: one is explicit instructions (for example: "Distinguish whether the following text is written by a human"), and the other is the text sample to be classified. The output sequence represents the classification result in the form of "positive </s>" or "negative </s>", where "</s>" is a sequence-ending token. To adapt the model to this text classification task, we replaced the head of the T5 model with a randomly initialized head.
During the inference phase, we restricted the model's vocabulary to only include the words "positive" and "negative". The model predicts whether the input text is human-written or machine-generated based on the probability distribution of these two words. For two disputed texts in the test dataset, the model first predicts each text individually, outputting either "positive" or "negative". Subsequently, we compare the predicted labels of these two texts. If the predictions differ, a clear conclusion can be drawn. If the predictions are the same, we compare the probability values of the predictions and choose the result with the higher confidence. Additionally, we introduced a special <PAD> token in the T5 model's decoder to help maintain consistent output formats when handling input sequences of different lengths. The detailed design of the entire network architecture is illustrated in Figure 1.
In this work, we adopt T5 as the pretrained model, and we use Pytorch to implement T5. Our hyperparameters are set as follows: the batch size is 512, the loss function is cross entropy, the initial learning rate is set to 5e-4, and 5 epochs are trained. Each training is optimized with AdamW. Our experiment was conducted on the A800 server.
To assess the effectiveness of our proposed model, we utilized the evaluation tool provided by PAN, which includes the following metrics: ROC-AUC: ROC-AUC is a comprehensive evaluation of the balance between the true positive rate and the false positive rate.
Brier: The complement of the Brier score (mean squared loss). C@1: A modified accuracy score that assigns non-answers (score = 0.5) the average accuracy of the remaining cases.
F1: F1 score is the harmonic mean of precision and recall, combining both metrics into a single value. F0.5u: A modified F0.5 measure (precision-weighted F measure) that treats non-answers (score = 0.5) as false negatives.
Mean: The arithmetic mean of all the metrics above.
We evaluated the performance of our model and baselines(Binoculars [9], Fast-DetectGPT (Mistral) [10], PPMd [11,12], Unmasking [13,14] and Fast-DetectGPT [10]) on the official test set provided in PAN 2024.
Table 2 shows the performance of our method across various metrics. Our method surpasses Unmasking and Fast-DetectGPT in ROC-AUC, C@1, F0.5u and mean respectively, but there is still a certain gap with Binoculars.
Table 3 further shows the average accuracy of our model on different dataset variants, particularly on the test sets of 9 variants. Our method surpasses Fast-DetectGPT (Mistral) on the minimum, surpasses Unmasking and Fast-DetectGPT on the median, surpasses PPMd, Unmasking, and Fast-DetectGPT on the 75-th quantile, and surpasses PPMd and Unmasking on the max.
Compared with the quantile results of other participants, our model is close to or exceeds the 25th quantile model on most indicators, and exceeds the Min model on all indicators, indicating that our method performed poorly on the test set and still has a significant gap compared to the current state-of-the-art methods.
The article comprehensively elaborates on our research progress in the field of Voight-Kampff Generative AI Authorship Verification in 2024. In this study, we fine-tuned a T5 pre-trained model to enhance the detection capability of AI text generation. The experimental results demonstrate that this method effectively enhances text detection capability, but there still exists a certain gap compared to the stateof-the-art methods. In the future, we plan to continue optimizing and refining this method to achieve higher levels of precision and efficiency. Additionally, we will explore the potential applications of this method in a wider range of natural language processing tasks, aiming to expand its scope of application and further increase its practical value.
This work is supported by the National Natural Science Foundation of China (No. 61972091), Natural Science Foundation of Guangdong Province of China (No. 2022A1515010101, No. 2021A1515012639).