Voight-Kampf Generative AI Authorship Verification, a task proposed jointly by PAN and the ELOQUENT Lab [1], aims to diferentiate between human and machine authors by analyzing text features, addressing the escalating text generation abilities of large language models. This paper uses the BERT pre-trained model for binary classification, fine-tuning it to identify distinctive features of human and machine writing. Additionally, we introduced an augmented dataset to enhance the model's recognition capabilities. Through comprehensive evaluation, we achieved a ranking score of 0.778 on all test sets and achieved 11th place.
alpaca-7b text ...
gpt-3.5 text gemini-pro text
... ...
Deactivate Neurons in Dropout
Input layer Hidden layer Output layer
The dataset provided by PAN comprises over a thousand human-written texts, alongside texts generated
by 13 distinct AI models. Each AI model generated an equal number of texts as the human-written
ones. To create a balanced dataset, one-thirteenth of the texts were randomly sampled from each of
the 13 sets of generative AI texts. This approach allowed us to construct a dataset where the number
of human-written texts is equal to the number of generative AI texts, ensuring a 1:1 ratio between
human and machine-generated content. The remaining generative AI texts were not discarded. Instead,
they were supplemented with an equal number of human-written texts collected from Hugging Face’s
Dataset Card for AuTexTification 2023 [
During fine-tuning, no parameters of the base BERT model were fixed. All parameters were updated to adapt to the binary classification task. Upon acquiring the BERT model, a fully connected (Dense) layer was added to map the model’s output to the label space of the binary classicfiation task. This Dense layer has two output units.
To prevent overfitting, Dropout was applied to this fully connected layer, randomly setting a portion
of the neurons’ outputs to zero during each training iteration. This step helps increase the model’s
generalization capability. The Dropout rate was set to 0.1, meaning that 10% of the neurons are dropped
out. Additionally, a softmax activation function was used to convert the model’s outputs into class
probabilities. To train the model, the loss function, optimizer, and evaluation metrics were configured.
Considering that the labels of the task are integer class identifiers, sparse categorical cross-entropy
was selected as the loss function. The optimizer chosen was Adam [
In TIRA [
Based on the aforementioned methods, we identified the texts in the competition and uploaded our results. Table 1 presents the final scores of the Voight-Kampf Generative AI Authorship Verification 2024 shared task, where the individual validity score is an aggregate across all test datasets, corrected by half a standard deviation to penalize unstable classification performance. Rankings are based on the mean average of all individual scores. Our team secured the 11th position out of 30, achieving a score of 0.778 across all test datasets. Table 2 provides an overview of the accuracy in detecting whether a text is written by a human in Task 4 of PAN 2024 (Voight-Kampf Generative AI Authorship Verification). Our model achieved a mean score of 0.906, surpassing most of the published baselines. In addition to the primary test dataset, the PAN organizers evaluated the "Voight-Kampf" Generative AI Authorship Verification on nine additional variants. Table 3 showcases the overview of the mean accuracy across these nine variants of the test set. Among the nine variant datasets, our model demonstrated its lowest accuracy at 0.361, while achieving a 75th percentile score of 0.959, with the highest accuracy reaching 1. Our model consistently outperformed the baselines across most of these variant datasets, underscoring its robustness and efectiveness in diverse scenarios. Baseline Binoculars (Falcon-7B) Baseline DetectLLM-LRR (Mistral-7B) Baseline Fast-DetectGPT (Mistral-7B) Baseline Text Length
SYSTEM
ROC-AUC Brier C@1
F1
F0.5 Mean canary-paint 4. Conclusion In this study, we addressed the challenge of distinguishing between human and machine-generated texts in the Voight-Kampf Generative AI Authorship Verification 2024 task. By leveraging a BERT-based model and incorporating data augmentation techniques, we enhanced the model’s ability to accurately classify texts. Our approach involved constructing a balanced dataset, fine-tuning the BERT model, and implementing a simple classifier for binary predictions.
The evaluation results demonstrated the feasibility and efectiveness of our approach, achieving a mean score of 0.778 and an overall ranking of 11, which outperformed all baseline models. These ifndings suggest that our approach is a promising solution for human-machine authorship discrimination, contributing to the broader field of AI authorship verification.
This work is supported by The Natural Science Foundation of Guangdong Province, China (No.2022A1515011544)