In this paper, we use the LoRA method to fine-tune the large language model ChatGLM. To balance the data distribution in the dataset, we modified the labels and transformed it into a multi-classification task. This enables the large language model to better learn the diferences in expression among diferent authors on the same topic, thereby learning the writing styles of humans and machines. During inference, we modify its final output by remapping it into a binary classification task, distinguishing whether the text was authored by a human or a machine. This approach aims to achieve the task of Generative AI Authorship Verification. The evaluation results on the PAN corpus test dataset indicate that this method is efective, with an mean score greater than 0.7.
As artificial intelligence-generated content (AIGC) technology continues to advance, large language
models (LLMs) such as ChatGPT, ChatGLM [
However, with these advancements, several issues have also surfaced. One major concern is the rapid spread of fake news, as LLMs can generate realistic and convincing false information that can be quickly disseminated across various platforms. Additionally, there is the manipulation of public opinion through social media comments, where LLMs are used to produce a large volume of persuasive and biased posts, swaying public perception and discourse. Another significant problem is academic dishonesty, with students using LLMs to complete their assignments, which undermines the integrity of the educational process and presents challenges for educators in assessing genuine student performance.
This paper presents our approach for Generative AI Authorship Verification task [
As large language models rapidly advance in generating extremely high-quality text, powerful LLMs
provide unprecedented convenience to people. These models not only understand and process complex
language inputs but also excel in generating coherent and contextually appropriate text, making
them valuable tools for applications such as customer service, content creation, educational support,
and more.However, the false text generated by powerful LLMs is raising ethical and legal concerns.
Moreover, it has become increasingly dificult for people to rely on their own experience to determine
whether a piece of text was written by a human or a machine. RoFT [
For this task, we utilize the dataset provided by PAN, which includes both genuine and fake news articles spanning multiple headlines from the United States in 2021. It consists of a JSON file written by 13 diferent machine authors and one human author. Each file contains articles on the same topic. The IDs and line order of the articles are the same, so the same line always corresponds to the same topic. Each document contains 24 topics and 1087 articles.
Although this could be considered as a binary classification task, to determine whether the text is written by a human or not.However, due to the extremely high quality of text generated by large language models, we believe that simply dividing this task into a binary classification task may not yield satisfactory results. Since we are also using a large language model to perform this task, in order to enable the model to learn diferent textual features, we have modified the dataset labels to implement a multi-class classification task.
Table 1 shows the specific modifications made to the labels in the dataset. Fortunately, the dataset provided by PAN is very well-organized, with each type of "author" providing the same number of articles, which is 1087.In the form of multi-class tasks, there won’t be exaggerated data proportions like "human:machines=1:13", which could lead to the issue of unbalanced learning data. Finally, during the inference test, we limit the output result to a number between 0 and 1. The closer the number is to 1, the more likely the text is written by a human. Conversely, the closer the number is to 0, the more likely it is written by a machine. When the probabilities are equal, we default to considering the first comment as written by a human and the second comment as written by a machine.
This paper employs LoRA technology to fine-tune ChatGLM. LoRA is a method used for fine-tuning large language models, aimed at enhancing the model’s performance on specific tasks. The core idea of LoRA is to expand the language representation of the model by introducing domain-specific corpora, making it more specialized and adaptable, as shown in Equation 1 = x + ∆ x = x + BAx = ( + )x (1) Where ∈ × represents the weight matrix of the pre-trained model. ∆ represents the change in the weights. ∆ = represents the update part obtained through fine-tuning, which is updated using low-rank decomposition. Where ∈ × , ∈ × , and the rank ≪ min(, ).
During fine-tuning, the weight parameters of the pre-trained model are frozen, and only the parameters and are trained. As shown in Figure 1, LoRA integrates the trained bypass weight parameters with the pre-trained model weights without introducing additional pathways for inference, making it suitable for real-time requirements in vertical domains.
Pretrained Weights (freeze)
B=0
r
A=N(0,σ2) h + x d
In this work, for LoRA, we set the LoRA rank to 4, the batch size to 8, and the learning rate to 1e-4, using FP16 for training. We completed this fine-tuning on a single A800 GPU. For ChatGLM, we set the top p to 0.7, max length to 2, and temperature to 0.2.
We submitted our system to TIRA and utilized the evaluation metrics provided by TIRA, which specifically include the following:
ROC-AUC: The area under the ROC (Receiver Operating Characteristic) curve.
Brier: The complement of the Brier score (mean squared loss).
C@1: A modified accuracy score that assigns non-answers (score = 0.5) the average accuracy of the remaining cases.
F1: The harmonic mean of precision and recall.
F0.5u: A modified F0.5 measure (precision-weighted F measure) that treats non-answers (score = 0.5) as false negatives.
Mean:The arithmetic mean of all the metrics above
We evaluated the performance of our model on the new test set provided by PAN, and the test results are shown in Table 2. Our test results are higher than Baseline Unmasking and Baseline FastDetectGPT, but lower than Baseline Binoculars, Baseline Fast-DetectGPT (Mistral), and Baseline PPMd. Our approach performs poorly on the new test dataset. This suggests that our model approach’s generalization ability is not satisfactory.
Table 3 further shows the average accuracy of ours model on diferent dataset variants, particularly on the test sets of nine variants. Our model’s minimum value across all variants was 0.219,with the 25th and 75th percentiles at 0.691 and 0.776, respectively, a median of 0.725, and a maximum value of 0.907. Our method surpasses PPMd, Unmasking, and Fast-DetectGPT on the 25-th quantile,75-th quantile and Max. Compared to the quantile results of other participants, our model is surpasses the models in the 25-th percentile in most metrics. This indicates that there is still a significant gap between our approach and the current state-of-the-art methods.
In this paper, we propose a method for Generative AI Authorship Verification on PAN 2024. We modified the labels in the training dataset to transform it into a multi-classification task. We fine-tuned ChatGLM with the aim of enabling the large language model to better understand the writing styles of diferent authors, thus learning the diferences between robot and human writing. We utilized LoRA technology for fine-tuning, as LoRA method can extend the language representation of the model, making it more professional and adaptive. From the results, it appears that the method performs poorly on the new test dataset, indicating that our approach lacks some degree of generalization ability.In subsequent work, it is advisable to employ more efective methods to augment the data and enhance the model’s classification ability on open sets.
This work is supported by the National Natural Science Foundation of China (No. 61972091), Natural Science Foundation of Guangdong Province of China (No. 2022A1515010101, No. 2021A1515012639). [10] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [11] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., Language models are unsupervised multitask learners, OpenAI blog 1 (2019) 9. [12] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al., Gpt-4 technical report, arXiv preprint arXiv:2303.08774 (2023).