This paper focuses on the text author (human or AI) classification problem, aiming to enhance the model's ability to detect AI-generated texts. We propose an innovative method based on ModernBERT, which introduces a custom gradient loss function and optimizes model training by combining sample weighting strategies, efectively enhancing the model's classification capability for texts of varying dificulties. In implementation, we ifne-tuned the ModernBERT model, optimized the training process using the custom gradient loss function, and constructed an eficient classification system through meticulous data preprocessing and rigorous testing evaluation. Experimental results show that this method achieved good performance metrics on the training set, but there was some overfitting on the test set, with performance metrics declining. Future work will be dedicated to further optimizing the model's generalization ability to improve its potential for application in multi-domain text classification.
With the tremendous success of large language models in recent years and the rapid development
of AI content generation technology, text author classification (human or AI) has gradually become
an important research area. In recent years, the gap between AI-generated texts and human texts in
terms of language fluency and logical coherence has gradually narrowed, and in some scenarios, it is
even dificult to distinguish between them. This makes text author classification crucial for content
authenticity verification, copyright protection, and information security defense. [
This study aims to develop a classification model that can efectively distinguish whether the text author is human or AI. Based on the pre-trained ModernBERT model, we referred to a training method that combines gradient loss and sample weighting. During training, we not only focused on classification accuracy but also controlled the model’s learning intensity for samples of diferent dificulties through the gradient loss function to prevent overfitting to dificult samples. We also assigned weights to the samples in the dataset, with higher weights for easy samples and lower weights for dificult samples, to adjust the loss calculation. Experiments have shown that this method can improve the model’s accuracy on the validation set, verifying its efectiveness in the text author classification task.
Currently, research in this field mainly focuses on the application of text feature analysis and deep
learning models. Traditional methods extract text features such as vocabulary, grammar, and syntax,
and use machine learning algorithms such as Support Vector Machine (SVM), Naive Bayes, Logistic
Regression, and Decision Tree for classification. Although these methods can distinguish between
human and AI-generated texts to some extent, feature engineering is complex and relies on human
experience, making it dificult to efectively extract and utilize complex text features. Moreover, with
the development of deep learning technology, models based on Recurrent Neural Networks (RNN) and
its variants (such as LSTM, GRU) have been applied to text author classification. These models can
automatically learn the sequential features and semantic information of texts, resulting in improved
classification performance compared to traditional methods. [
In recent years, pre-trained language models based on the Transformer architecture (such as BERT
and its variants, GPT series, etc.) have gradually become the mainstream method for text author
classification. [
This experiment is based on the ModernBERT model and employs an eficient text author classification
method, aimed at overcoming the limitations of traditional models such as the original BERT in handling
long texts, computational eficiency, and timeliness of data. [
In the experiment, we first preprocessed the training and validation data, including tokenization, truncation or padding to a fixed length, and assigning weights based on sample dificulty. Then, we finetuned the ModernBERT model using this data by adding a classification layer that utilizes the CLS token as input to capture the global representation of the text for the binary classification task and optimizing model parameters to adapt to specific text classification tasks. Additionally, a custom gradient loss function was introduced, which efectively controlled the learning intensity of the model for samples of varying dificulties by adjusting the gradient of the loss function for model outputs, preventing overfitting. Ultimately, the model performed well on the training set, verifying its efectiveness and superiority in text author classification tasks.
Figure 1 below shows the basic workflow of the ModernBERT model in the experiments of this paper.
This experiment aims to construct a binary classification model and program that can accurately
distinguish whether the text author is human or AI. The experimental dataset is provided by PAN lab
2025 and contains text samples generated by humans or machines in JSONL format. [
The fine-tuning training data for this model comes from the Zenodo platform, uploaded by PAN lab 2025.
The dataset includes training and validation datasets, containing text samples generated by humans
and multiple AI models to achieve training and prediction purposes. [
This program uses the pre-trained ModernBERT-base model from answerdotai company as the base model and fine-tunes it (fine-tuning) with the training dataset to adapt to the text author classification task. The ModernBERT model, based on the Transformer architecture, uses a self-attention mechanism to capture long-range dependencies in the text. The output layer of the model is adjusted for binary classification tasks, with an output category number of 2 (human or AI).
The training and validation datasets are provided in JSONL format. The data preprocessing steps are as follows: 1. Tokenize the text using the ModernBERT tokenizer. 2. Truncate or pad the tokenized sequence to a fixed length (512 tokens). Although ModernBERT can handle sequences of up to 8192 tokens, we truncate or pad the text to 512 tokens here to align with the input requirements of the dataset and to ensure uniformity of model input, which can also improve the computational eficiency during training. 3. Assign weights to each sample based on the dificulty label. The dificulty of a sample is determined by its distinguishability from human-written text. Samples that are more easily distinguishable (i.e., those with more obvious AI characteristics) are labeled as easy and assigned higher weights, while samples that are harder to distinguish (i.e., those closely resembling human writing style) are labeled as dificult and assigned lower weights. This strategy allows the model to focus more on dificult samples during training, thereby improving its generalization ability.
The model training process includes several key steps: 1. Define a custom dataset class: Create a custom dataset class MyDataset to load and preprocess training and validation data. This class is also responsible for reading JSONL files, tokenizing, truncating, and padding the text, and assigning weights to each sample based on the dificulty label. 2. Custom gradient loss function: A custom gradient loss function is defined in the experimental program, combining the standard cross-entropy loss and its gradient for model outputs, while adjusting the gradient size based on sample weights. The formula for the custom gradient loss function is as follows:
⃦ ℒgrad = 1 ∑=︁1 ⃦⃦⃦ ∇ CE(logits, labels) · weights⃦⃦ 2 where CE is the cross-entropy loss function, logits represents the gradient of the model’s output logits, weights are the sample weights, and N is the batch size. This formula calculates the gradient of the cross-entropy loss for the model’s output, adjusts the gradient size based on sample weights, and ultimately derives the gradient loss to optimize model training and prevent overfitting. 3. Training process: During the training phase, the preprocessed data is loaded onto the GPU, and the AdamW optimizer is used to zero the model’s gradients. In training, the model performs forward propagation to obtain output results, calculates the loss using the custom gradient loss function, and then updates the model parameters through backpropagation and the optimizer. The entire training process lasts for three epochs, and the best model parameters are saved by monitoring the validation set accuracy.
The prediction process includes the following steps: 1. Define the test dataset class: Load the trained model parameters and define the test dataset class TestDataset to load and preprocess the test data, similar to the custom dataset class MyDataset used in the training model above. 2. Prediction function: The prediction function loads the test data, performs inference through the model, converts the predicted confidence scores into probabilities through the sigmoid function, and then converts them into binary labels based on a threshold of 0.5. Finally, the prediction results are saved in JSONL format.
This experimental program has been uploaded to the oficial testing platform tria of PAN lab 2025 for
oficial testing [
Tables 4-1 and 4-2 below show the evaluation results of the test and training datasets on the tria online platform, respectively.
F1
Based on multiple tests of the test and training datasets, overall, the program using the pre-trained ModernBERT model and fine-tuning it to adapt to the text author classification task has achieved good results. However, there is still room for improvement in the model’s generalization ability. The high metric values on the training set indicate that the model fits the training data well, but the performance drop on the test set suggests insuficient adaptability to new data. In the future, methods such as data augmentation, regularization, or adjusting the model architecture could be considered to enhance the model’s generalization ability to cope with the diversity of test data.
This thesis innovatively proposes a text author classification method based on ModernBERT, conducting in-depth research on the detection of human and AI texts. For the first time, a gradi-ent loss function is applied to the fine-tuning process of ModernBERT, combined with a sample weighting strategy to optimize model training, efectively improving the classification perfor-mance for texts of diferent dificulties. In implementation, through meticulous data prepro-cessing, custom gradient loss calculation, model fine-tuning, and rigorous testing evaluation, an eficient classification system is constructed. Experiments have verified the advantages of this method over traditional models, providing new ideas for the identification of AI-generated con-tent and potentially expanding its application in multi-domain text classification in the future.
This work is supported by the Social Science Foundation of Guangdong Province, China (No.GD24CZY02)
During the preparation of this work, the author utilized Kimi K2 and DeepSeek-V3 to accomplish drafting content, text translation, and content enhancement tasks. After employing these tools, the author reviewed and edited the content as necessary and took appropriate measures to assume full responsibility for the content of the publication.