Generative AI Authorship Verification is the task of distinguishing between human-authored and machinegenerated texts. This paper explores the application of the pre-trained language model Deberta to address this problem. Our approach involves fine-tuning the Deberta model on a curated dataset comprising both human and machine-generated texts. To manage the imbalance in our dataset, we employed a random sampling to ensure a balanced representation of both types of texts during training. Preliminary experiments show that while our method performs comparably with existing approaches, there is significant potential for further optimization and improvement in identifying human-authored texts. Future work will explore advanced techniques and larger datasets to enhance model.
With Large Language Models (LLMs) improving at breakneck speed and seeing more widespread
adoption every day, it is getting increasingly hard to discern whether a given text was authored by a
human being or a machine. These models, such as GPT-3[
Many classification approaches have been devised to help humans distinguish between human and
machine-authored text. Traditional methods rely on surface-level features such as word frequency,
syntactic patterns, and stylistic elements, but these features can be easily mimicked by advanced
LLMs[
Recently, PAN 2024[
To address this task, we use the Deberta[
PAN 2024 provides a guided dataset covering both real and fake news articles from multiple 2021 US news headlines. Each file contains a list of articles, written either by (any number of) human authors or a single machine. Machine text is generated by some large language models such as Gemini Pro[12].
The dataset comprises human-authored text and machine-generated text, with a significant imbalance in the ratio between the two categories (1:13). To address this challenge, we adopted a random sampling approach during model training. Due to the limited data availability and the need to balance computational resources, we trained the classification model for two epochs.
In the first epoch, we randomly selected 1200 samples from the combined machine-generated text to ensure representation from diferent sources and topics. For the second epoch, we increased the sample size to 3000 to further enrich the training data. All human-authored samples were included in both epochs to maintain a balanced representation of human and machine texts.
Subsequently, in each epoch, we combined these two sets of data into a format where a label corresponds a type of text, classifying the two types of texts separately. We then split the data into training, validation, and test sets in a ratio of 0.95, 0.05, and 0.05, respectively. This partitioning strategy ensured that the model was trained on a diverse range of samples while maintaining suficient data for evaluation and testing. Finally, we utilized the Deberta-large model architecture for AI authorship verification on the combined dataset.
We utilized the Deberta-large model, which is characterized by a vocabulary size of 50,000, a hidden size of 1024, 24 layers, and a total of 3.03 billion parameters. This model was selected for its disentangled attention mechanism and enhanced masked decoder. The classification model was built using PyTorch, with training conducted using a batch size of 2. We did not set a maximum encoder length, fully leveraging the model’s capacity to handle long texts. The AdamW[13] optimizer, with a learning rate of 1e-6, was employed to update the model weights, while cross-entropy[14] was used as the loss function to measure prediction error. The network was trained over 2 epochs to ensure thorough learning without overfitting.
In this study, we just utilized the [CLS] token, which is a standard practice in BERT and its derivative models. The [CLS] token is positioned at the onset of the sequence, serving to aggregate information from the entire input sequence, which is crucial for classification tasks. We employed the [CLS] token based on the default settings as per the model’s pre-training, without any modifications to its functionality.
All experiments were conducted on an NVIDIA A800 GPU with 80GB of memory, providing the necessary computational power to handle the large model and extensive dataset. Additionally, data augmentation techniques such as random sampling were applied to enhance the training data diversity, thereby improving the model’s generalization ability. Performance metrics, including accuracy, precision, recall, and F1-score, were used to evaluate the model’s efectiveness on the test dataset.
To process each sample, we compare the confidence scores of Text 1 and Text 2. If Text 1’s confidence score is higher, the final confidence score is 1 minus Text 1’s confidence score; if Text 2’s confidence score is higher, the final confidence score is Text 2’s confidence score.
Table 1 summarizes the performance of the validation and test sets in this experiment, highlighting high accuracy, precision, recall, and F1 scores.Table 2 shows the summarized results averaged (arithmetic mean) over 10 variants of the test dataset. Each variant uses a diferent technique to test the robustness of authorship verification approaches, such as switching text encoding, translating text, changing the domain, and manual obfuscation. Table 3 shows the results, initially pre-filled with the oficial baselines provided by the PAN organizers and summary statistics of all submissions to the task (i.e., the maximum, median, minimum, and 95-th, 75-th, and 25-th percentiles over all submissions to the task). The evaluations for Table 2 and Table 3 were conducted on the PAN 2024 Generative AI Authorship Verification task training dataset using the TIRA[ 15] platform. Our method, referred to as "beige-limit" in the tables, is compared against various baselines.
This paper addresses Generative AI authorship verification using the Deberta model. The goal was to distinguish between human and machine-authored texts. By employing a pre-trained Deberta-large model and random sampling to manage data imbalance, we conducted a series of experiments to evaluate the model’s performance.We ranked 23rd in this task using this method.
Our study indicates that advanced pre-trained language models like Deberta have potential for authorship verification tasks. Future research could explore more eficient training strategies and extend this approach to other domains and languages.
This research was supported by the Natural Science Platforms and Projects of Guangdong Province Ordinary Universities (Key Field Special Projects) (No. 2023ZDZX1023) [12] T. G, R. Anil, S. Borgeaud, et al., Gemini: a family of highly capable multimodal models, arXiv preprint arXiv:2312.11805 (2023). [13] I. Loshchilov, F. Hutter, Decoupled weight decay regularization, arXiv preprint arXiv:1711.05101 (2017). [14] R. Rubinstein, The cross-entropy method for combinatorial and continuous optimization, Methodology and computing in applied probability 1 (1999) 127–190. [15] M. Fröbe, M. Wiegmann, N. Kolyada, B. Grahm, T. Elstner, F. Loebe, M. Hagen, B. Stein, M. Potthast, Continuous Integration for Reproducible Shared Tasks with TIRA.io, in: J. Kamps, L. Goeuriot, F. Crestani, M. Maistro, H. Joho, B. Davis, C. Gurrin, U. Kruschwitz, A. Caputo (Eds.), Advances in Information Retrieval. 45th European Conference on IR Research (ECIR 2023), Lecture Notes in Computer Science, Springer, Berlin Heidelberg New York, 2023, pp. 236–241. doi:10.1007/ 978-3-031-28241-6_20.