Introduction

In recent years, generative LLMs have gained recognition for their impressive ability to produce coherent language across different domains. Consequently, detecting machine-generated text has become increasingly vital. The Generative AI Authorship Verification task regarded as detecting machine-generated text task involves two texts, one authored by a human and one by a machine. The primary objective is to determine which of the two texts was written by a human and which was generated by a machine. Furthermore, the Generative AI Authorship Verification task can aid in ensuring the authenticity of information is critical, such as legal proceedings.

Research [1] utilizes Token Prediction as an Implicit Classification for Generative AI Authorship Verification. By assigning distinct tokens to different labels and reformulating the multi-class classification task into a next-token prediction task, this method identifies whether the input sentence was generated by a particular model or authored by a human [1]. The purpose of this approach is to leverage the model's next-token prediction capability for this specific task. Recent studies [2] have employed the fine-tune transformer-based method, which achieved the LLMsgenerated text detection task by training transformer-based classifiers. However, one of the biggest challenges in fine-tuning transformer-based methods is not to directly leverage the next-token prediction capability of the model for this particular task [1]. Fine-tune transformer-based method will increase the gap between downstream tasks and pre-training tasks compared to next-token prediction [3]. Hence here are better solutions than simply fine-tuning transformer-based methods. In this paper, we leverage research [1] to predict whether a given sample text is authored by a human or paraphrased by a machine. Unlike the fine-tuning transformer-based method, we employ the Token Prediction as an Implicit Classification approach. This involves establishing a bijection 𝑓 :𝑌 → 𝒴, where 𝑌 ⊂ Σ. 𝒴 serves as proxy labels such as 'human', 'GPT-3.5', etc. 𝑌 represents the ground truth label. The model then predicts the corresponding proxy labels based on the input text. We have established two sets of proxy labels which are proxy labels in method 1 and proxy labels in method 2. In method 1, the proxy labels can be translated into three outcomes: one indicating human authorship, one indicating AI model rewrites, and one indicating undecidable. Method 2, proxy labels are translated into two outcomes: one for human authorship and one for AI model rewrites. This differentiation allows us to determine whether the text is human-authored, machine-generated, or falls into another category. In detail, the model comprises two parts. The first part is the long-T5 [4] model, which encodes the input text. The second part is a linear layer designed to project the output of long-T5 onto a dimension equivalent to the vocabulary size. This projects the probabilities of the proxy labels, thereby determining whether the input text under examination was generated by a model, authored by a human or undecidable.

Network Architecture

First, the language model is presented with a series of sentences to be tested, each consisting of tokens from 𝐸 1 to 𝐸 𝑛 and 𝐸 <𝑠> . The goal is to utilize the longT5 model to implement the Generative AI Authorship Verification task. The core feature of the model is the method of next-token prediction. After inputting the tokens from 𝐸 1 to 𝐸 𝑛 and 𝐸 <𝑠> into longT5, it obtains the probabilities of the proxy labels. The predicted proxy labels for each sentence are then determined by selecting the label with the highest probability. Then, we convert the proxy labels into the final result, determining whether the text was authored by a human or paraphrased by a machine. According to the model shown in Figure 1, it comprises a longT5 backbone, a next-token prediction layer, and a filter. The first component is the longT5 backbone, which is used to encode the sentences under examination. Following the next-token prediction layer in method 2, where linear layers map the output of longT5 to a dimension equivalent to the vocabulary size, enabling the calculation of probabilities for each proxy label. In method 2, the filter selects the probabilities corresponding to the proxy labels from the output of the next-token prediction layer, which are then processed through a softmax layer. Finally, the proxy labels with the highest probability are chosen, which is then translated into one of two outcomes: whether the text under examination was generated by a specific model or authored by a human. Returning to method 1, it is similar to method 2 but it identifies the text by obtaining the probabilities corresponding to the proxy labels from the next-token prediction layer. In method 1, after obtaining the proxy labels, we translate them into two outcomes: one indicating human authorship and the other AI model rewrites. For method 1, in addition to these two outcomes, we include an additional result labeled as undecidable, making three possible outcomes. The detailed process is described in section 2.1. Overall, the primary loss function ℒ can be defined as follows.

ℒ = ℒ 𝑁 𝐿𝐿 = −𝑙𝑜𝑔𝑃 (𝒴 𝑖 |𝑆 𝑖 ; 𝜃)(1)

The loss ℒ 𝑁 𝐿𝐿 is negative log-likelihood to optimize the longT5 and next-token prediction layer, 𝑆 𝑖 means the sentence under examination, 𝜃 mean the whole model's parameters, and 𝑦 means the ground truth labels.

Self-attention

Feed-forward MLP

next-token prediction

For method 2, we assign a special token "<extra_id_0>" as the proxy label for human-authored text. For other models we designate similar tokens such as "<extra_id_1>", "<extra_id_2>", ... "<extra_id_n>", where 𝑛 ≤ 𝑘 and k represents the number of models involved in PAN dataset [5,6].

In method 1, human-authored texts are tagged with the word "positive" as the proxy label, while all texts rewritten by AI models are labeled "negative". If the highest probability in the next token probability distribution does not fall on either "positive" or "negative", the result is deemed "undecidable".

Both methods involve the model predicting the probabilities of the proxy labels and then converting proxy labels into the actual prediction results. Next, we measure the token length of each human-authored or model-generated text. Our statistical analysis reveals that the vast majority of text lengths are within 2048 tokens. Firstly, the PAN organization has provided datasets for Generative AI Authorship Verification, which include multiple texts authored by humans and subsequently rewritten by various models. Give a batch name as ℬ. The contents of ℬ can define as {(𝑆 1 , 𝒴 1 ), (𝑆 2 , 𝒴 2 )...(𝑆 𝑖 , 𝒴 𝑖 )} ∈ ℬ, where 𝑆 𝑖 means the sentence under examination, and 𝒴 𝑖 is the proxy label.

During training, we feed the ℬ into the pre-training model which is composed of the transformer [7] block to get the corresponding hidden state ℋ 𝑖 . After obtaining the hidden state ℋ 𝑖 we use the next-token prediction layer and softmax layer to obtain the probabilities for all tokens in the vocabulary. That is,

𝜙 𝑖 = (𝒴 1 𝑖 , 𝒴 2 𝑖 , ...𝒴 𝑉 𝑖 ) = ( 𝑒 (𝜑(ℋ 𝑖 ) 1 ) ∑︀ 𝑉 𝑣=1 𝑒 (𝜑(ℋ 𝑖 ) 𝑉 ) , 𝑒 (𝜑(ℋ 𝑖 ) 2 ) ∑︀ 𝑉 𝑣=1 𝑒 (𝜑(ℋ 𝑖 ) 𝑉 ) , ..., 𝑒 (𝜑(ℋ 𝑖 ) 𝑉 ) ∑︀ 𝑉 𝑣=1 𝑒 (𝜑(ℋ 𝑖 ) 𝑉 ) )(2)

where 𝜙 𝑖 is the soft label of sample i , v indicates the position of a token within the vocabulary, V represents the total number of tokens in vocabulary, 𝒴 𝑉 𝑖 represents the probability of the V -th word in vocabulary and 𝒴 𝑖 means proxy label. Then we calculate the negative log-likelihood loss for classification.

𝐿 𝑛𝑙𝑙 = −𝑙𝑜𝑔𝑃 (𝒴 𝑖 |𝜙 𝑖 , 𝜃)(3)

In the inference phase, for method 1, after obtaining 𝜙 𝑖 , we convert 𝜙 𝑖 into three predictive outcomes.

𝑦 ˆ= ⎧ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎩ 1 𝑎𝑟𝑔 max 𝒴∈𝒴 𝑉 𝑖 𝜙 𝑖 = 𝑎 0 𝑎𝑟𝑔 max 𝒴∈𝒴 𝑉 𝑖 𝜙 𝑖 = 𝑏 0.5 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒(4)

In method 1, 𝑎 represents the position of the word "positive" in the vocabulary, while 𝑏 represents the position of the word "negative". 𝑦 ˆrepresent predict label. 𝑦 ˆ= 1 indicates text authored by humans, 𝑦 ˆ= 0 indicates text rewritten by a machine, and 𝑦 ˆ= 0.5 indicates 'undecidable' when a clear determination cannot be made. For method 2, we initially obtain the output from the next-token prediction layer.

𝜑(•) = (𝜑(ℋ 𝑖 ) 1 , 𝜑(ℋ 𝑖 ) 2 , ..., 𝜑(ℋ 𝑖 ) V )(5)

where 𝜑(•) indicates the output of the next-token prediction layer and V is vocabulary size. We then use a filter to select the outputs associated with all the special tokens(proxy label tokens).

𝜑(•) ′ = (𝜑(ℋ 𝑖 ) 1 , 𝜑(ℋ 𝑖 ) 2 , ..., 𝜑(ℋ 𝑖 ) 𝑘 )(6)

where 𝜑(•) ′ indicates the output of the filter and 𝑘 represents the number of all special tokens. After passing through the softmax layer, we obtain the probability distribution of proxy label tokens.

𝜙 ′ 𝑖 = (𝒴 1 𝑖 , 𝒴 2 𝑖 , ...𝒴 𝑘 𝑖 ) = ( 𝑒 (𝜑(ℋ 𝑖 ) ′ ) 1 ∑︀ 𝑘 𝑗=1 𝑒 (𝜑(ℋ 𝑖 ) ′ ) 𝑗 , 𝑒 (𝜑(ℋ 𝑖 ) ′ ) 2 ∑︀ 𝑘 𝑗=1 𝑒 (𝜑(ℋ 𝑖 ) ′ ) 𝑗 , ..., 𝑒 (𝜑(ℋ 𝑖 ) ′ ) 𝑘 ∑︀ 𝑘 𝑗=1 𝑒 (𝜑(ℋ 𝑖 ) ′ ) 𝑗 )(7)

where j ∈ 𝑘 and 𝜙 ′ 𝑖 represent probability distribution of proxy label tokens. Finally, we convert 𝜙 𝑖 ′ into two predictive outcomes:

𝑦 ˆ= ⎧ ⎨ ⎩ 0 𝑎𝑟𝑔 max 𝒴∈𝒴 𝑉 𝑖 𝜙 𝑖 = 𝑐 1 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒(8)

where 𝑐 ∈ {1 . . . 𝑘} indicates the special tokens, 𝑦 ˆ= 1 indicates text authored by humans, and 0 indicates text rewritten by a machine.

Experiments and Result

Experience setting

In this work, we utilize the longT5 model for classification, which consists of 12 transformer layers, with a hidden size of 768. As for the next-token prediction layer, we use randomly initialized parameters before training. For method 1, the training parameters are set with 10 epochs, a batch size of 64, and a learning rate of 5e-4. For method 2, the settings are 15 epochs, a batch size of 16, and a learning rate of 8e-4. Both method's maximum token length is set to 2048. All experiments are conducted on an NVIDIA A800 GPU with 80GB of memory.

Results

We will conduct two experiments using token prediction as an implicit classification for both method 1 and method 2. After training with these methods, the resulting model weights from both experiments will be submitted to the TIRA platform [8] to obtain scores. Table 1 and 2 displays our test set results reported to the TIRA platform. Table 1 shows the summarized results averaged (arithmetic mean) over 10 variants of the test dataset.

Each dataset variant applies one potential technique to measure the robustness of authorship verification approaches, e.g., switching the text encoding, translating the text, switching the domain, manual obfuscation by humans, etc. Table 2 shows the results, initially pre-filled with the official baselines provided by the PAN organizers and summary statistics of all submissions to the task (i.e., the maximum, median, minimum, and 95-th, 75-th, and 25-th percentiles over all submissions to the task).

Conclusion

In this paper, we have completed the tasks set by PAN and have employed the next-token prediction method to tackle the Generative AI Authorship Verification task. Instead of using fine-tuned transformerbased method techniques, we utilize the next-token prediction method to narrow the gap between downstream tasks and pre-training tasks. Finally, on the test set, our performance metrics at the Minimum, 25-th Quantile, Median, 75-th Quantile, and Max were 0.527, 0.896, 0.922, 0.926, and 0.947 respectively. These results certify the effectiveness of our proposed method performing the Generative AI Authorship Verification task.

Limitations

Firstly, the method proposed in this paper does not involve any prompts in the current LLMs-generated text detection task. Using prompts can better leverage the internal knowledge of language models. Therefore, in future work, we plan to incorporate prompts to complete this task. Additionally, transforming the task into a binary AI detection task, rather than judging which AI -authored the text, is another method to accomplish AI detection tasks. However, this approach can easily lead to data imbalance issues, where the amount of human-authored data is not equivalent to that of AI-generated data. To address this, data augmentation techniques could be employed to increase the quantity of human-authored data.