In the task of detecting machine-generated texts, accurately distinguishing between those created by artificial intelligence and those authored by humans is crucial. In this evaluation, we use a method that integrates two BERT models with a causal language model specifically trained on in-distribution (ID) samples. This integration enhances the performance of individual models in distinguishing between machine-generated and human-authored texts. Experimental results indicate that our method achieves a certain level of efectiveness in distinguishing between machine-generated and human-authored texts.
In recent years, the rapid development of large-scale language models such as ChatGPT and Claude has
increased the awareness and usage of artificial intelligence technologies among a wider audience. While
these advancements have improved eficiency in daily tasks, they have also brought about some negative
consequences, such as the unethical use of AI to generate academic papers or complete assignments[
The goal of machine text generation detection is to discern whether a text is generated by a machine
or by a human[
Building upon existing work and considering the excellent performance of BERT models in capturing contextual information and the ability of causal language models to capture the causal relationships in text generation better, we hypothesize that integrating BERT models and causal language models can complement their weaknesses and reduce biases and errors that might occur with a single model. Therefore, we use a method that integrates two BERT models with a causal language model specifically trained on in-distribution (ID) samples.
Our methodology, as depicted in Figure1, involves training two distinct BERT models, namely BERT
Model A and BERT Model B, utilizing the same dataset but employing diferent processing techniques.
Subsequently, we exclusively train a causal language model using machine-generated texts, which can
compute the perplexity of a text segment. Perplexity is a metric used to evaluate the performance of a
language model. It represents the model’s ability to predict the next word1. Due to slight diferences
in causal logic between machine-generated and human-authored texts at the sentence or word level,
we classify a segment as machine-generated if the perplexity score computed by the causal language
model is low; otherwise, it is classified as human-authored[
Here’s the detailed workflow: Initially, our method takes two text segments (referred to as text1 and text2) as input. These segments are then separated by [sep] and fed simultaneously into BERT Model A and BERT Model B for evaluation, yielding label A and label B. Meanwhile, text1 and text2 are independently input into a causal language model. This model evaluates each input text segment separately, computing their respective perplexity scores (Perplexity1 and Perplexity2). Subsequently, using a function to generate the output result label C of the causal language model based on perplexity1 and perplexity2, if Perplexity1 is less than Perplexity2, label C is set to 1; otherwise, it is set to 0.
Each model generates a label based on its analysis results, indicating which segment of text it believes is more likely to be human-authored. Finally, through a voting mechanism, the evaluations from all models are aggregated to determine the final output, identifying which segment of text is authored by a human[10].
text1 [SEP]
text2 Bert Model A
Causal Language Model
Bert Model B label A label B Perplexity 1
Perplexity 2 Function label C VOTE label D
The datasets are sourced from the PAN24 generation-author-news and Kaggle’s DAIGHT-v2-traindataset2. The PAN24 generation-author-news dataset is provided by the evaluation authority, while the DAIGHT-v2-train-dataset is contributed by a participant in Kaggle’s machine-generated text detection competition. The PAN24 dataset includes texts generated by 13 large language models and one text authored by a human. Notably, these 13 machine-generated texts and the single human-written text cover 1,087 descriptions of the same topics. The daigt-v2-train-dataset contains segments of machinegenerated and human-authored texts on 15 same topics.
Considering the token length limitations of large models, we have conducted a token count on these data.
Our analysis of the dataset revealed that some texts exceed 512 tokens. To address this issue, we propose a solution that involves segmenting the texts based on their length and then averaging the perplexity scores of these segments to determine the final result. This method not only overcomes the token length limitation but also has the potential to improve model accuracy. We established text segmentation rules where each segment does not exceed 500 characters and is divided by punctuation marks.
Through experiments, we observed that shorter texts significantly impact model accuracy, so we discarded text segments shorter than 50 characters after segmentation. We randomly selected 1,087 unique topic text segments from 13 machine-generated text datasets and combined them with a humanauthored text dataset to form a new dataset. This new dataset was then segmented according to the text segmentation rules to create a validation set. The remaining machine-generated texts were also segmented according to the same rules and used as the training set.
In the PAN24-Generation-Author-News dataset, we paired the human-generated dataset with 13 machine-generated datasets by matching texts on the same topic. The human-generated texts were designated as text1, and the machine-generated texts as text2, with a label assigned as 0. We applied the same pairing method in the DAIGHT-v2-train dataset, combining human-generated texts with machine-generated texts based on similar topics. After completing all pairings, we swapped the content
of text1 and text2 for half of the data and changed the label to 1, thus forming Dataset A. Considering the maximum number of tokens the model can accept and the relationship between the two text segments, we swapped the content of text1 and text2 in Dataset A again and inverted the label values, creating Dataset B. Both Dataset A and Dataset B were then split into training and validation sets in a 7:3 ratio. Texts were always truncated at the end after concatenation. If the first text alone exceeded the BERT token limit, it was treated as a single-text classification problem. The aforementioned text swapping operations mitigated the potential adverse efects of this issue to some extent.
We employ the GPT-2 model as a causal language model to perform the task of predicting text perplexity. During the training process, we opted for the AdamW optimizer, which utilizes an adaptive learning rate suitable for weight decay regularization. In setting the optimizer parameters, we distinguished between parameters that required decay and those that did not: ’bias’ and ’LayerNorm.weight’ parameters within the model were exempt from weight decay, whereas all other parameters were subjected to it. The initial learning rate was set at 5e-5, and we implemented a ’constant_with_warmup’ learning rate schedule. This strategy starts with a warmup phase and subsequently maintains a constant learning rate. We calculated the number of update steps per training epoch based on the batch size and the gradient accumulation steps. Training cycles were adjusted inversely based on the maximum training steps to ensure the completeness and consistency of the training. Each batch of data is first processed on the device, followed by bidirectional forward propagation, meaning each batch undergoes two separate forward propagations, yielding two sets of outputs. Additionally, to prevent gradient explosion, gradient clipping techniques were applied. All experiments are conducted on NVIDIA A800 GPU with 80GB memory with a batch size of 8.
We trained the BERT model on Dataset A and Dataset B separately, resulting in Model A and Model B, respectively. The BERT models were optimized using the AdamW optimizer, with the learning rate set at 3e-5. The batch size was set to 8, and the models were trained for 3 epochs. Notably, our experiments revealed that although this appears to be a binary classification task, setting the model’s output categories to three significantly enhances its performance.
In this evaluation task, our method surpasses the baseline in both the minimum and median scores, and exceeds most baselines in the 25th quantile, 75th quantile, and maximum scores, as shown in Table 2.
This paper discusses a method used in the authorship verification task of generative AI in the PAN@CLEF 2024 competition, which integrates two trained BERT models with a causal language model. The results indicate that our method is efective in distinguishing between machine-generated and human-authored texts, leveraging the strengths of both types of models. However, our study has certain limitations, and there is room for further improvement in the performance of individual models. This could involve exploring alternative base models or adopting diferent training strategies to optimize the results.
This work is supported by the National Social Science Foundation of China (No.22BTQ101) . [9] F. Harrag, M. Debbah, K. Darwish, A. Abdelali, Bert transformer model for detecting arabic gpt2 auto-generated tweets, arXiv preprint arXiv:2101.09345 (2021). [10] G. Dhaou, G. Lejeune, Comparison between voting classifier and deep learning methods for arabic dialect identification, in: Proceedings of the Fifth Arabic Natural Language Processing Workshop, 2020, pp. 243–249.