As large language models (LLMs) become more accessible, machine-generated content is rapidly increasing in many fields. These models can produce fluent and coherent text, making them useful for automating various writing tasks. However, their wide use has also raised concerns about misinformation, academic honesty, and the authenticity of content. Therefore, it is important to identify how much of a text is created by humans and how much by machines. In this study, we introduce StarBERT, a new hybrid model that combines DeBERTav3-large with StarBlock2d. This model focuses on classifying texts written through human-AI collaboration. Specifically, after dividing the texts into six categories based on the type of human and machine contribution, StarBERT uses the deep language understanding of DeBERTa-v3-large and the high-dimensional mapping ability of StarBlock2d. Our results show that StarBERT performs significantly better than existing baseline models, such as RoBERTa-base.
The rapid spread of generative AI models, such as GPT-4o and Claude 3.5, has ushered in a new era of human-AI collaborative creation, where machine-generated text is deeply integrated with humanauthored content. While this collaboration boosts productivity, it also presents serious challenges for tracing the origin of texts and identifying authorship—both of which are essential for maintaining academic integrity, combating misinformation, and ensuring content authenticity.
Although some studies have ofered valuable insights [
Traditional AI text detection models that rely solely on binary classification (i.e., human vs. machine)
are not well-suited to handle the full spectrum of human-AI collaborative writing. These models
often fail to identify complex co-writing patterns, ranging from light AI-assisted editing to deeply
integrated joint narratives. To help people better understand human-AI collaboration and reduce the
risks associated with synthetic texts, PAN@CLEF 2025 [
StarBERT is trained using a stable cross-entropy loss function to enhance training robustness. It was fine-tuned on a JSONL-formatted dataset provided by the organizers to better distinguish among six types of human-AI collaboration. Following local training and initial validation, we submitted the model’s predictions—formatted as "id": "identifier of the test sample", "label": 1—to the CodaLab platform. This platform ofers a rigorous and controlled evaluation environment, ensuring fair and transparent benchmarking against standard baselines. StarBERT achieved strong results across several key metrics: Macro Recall of 57.46%, Macro F1 of 56.31%, and Accuracy of 66.81%, highlighting its capability to distinguish human-AI collaborative writing efectively. These results demonstrate the practical value of StarBERT in real-world collaborative text classification tasks.
In natural language processing tasks, data preprocessing is a crucial step. It not only lays the foundation for model training but also directly impacts the model’s final performance. Our preprocessing pipeline includes two main components: label conversion and text tokenization.
We begin by converting the labels in the dataset. In the original data, labels are stored as strings, while deep learning models typically require labels in integer format. To handle this, we define a mapping function that converts the label field in each data sample to its corresponding integer value. The label range spans from 0 to 5, representing six distinct categories. Next, we tokenize the text using the tokenizer aligned with DeBERTa-v3-large. The tokenizer converts each text sample into a sequence of token indices from the model’s vocabulary, while also indicating which tokens are valid and which are padding. After tokenization, we apply truncation and padding to ensure that all input sequences conform to the model’s maximum input length. We set this maximum to 512 tokens, so all inputs are either truncated or padded to exactly 512 tokens. Once tokenization is complete, the data is ready for model training. We ensure the quality of the input by carefully preprocessing the dataset, which transforms raw data into a format suitable for deep learning models.
This comprehensive preprocessing pipeline not only prepares the dataset for efective training of StarBERT, but also enhances the model’s accuracy in distinguishing human-AI collaborative text—a key requirement for this evaluation task.
In this study, we introduce StarBERT, a hybrid neural network model that combines the powerful feature extraction capabilities of DeBERTa-v3-large with the implicit high-dimensional feature fusion mechanism of StarBlock2d to handle complex textual features and distinguish subtle diferences between categories of human-AI collaborative writing. To help readers better understand StarBERT, we use Figure 1 to illustrate its structure.
We adopt the pretrained roberta-v3-large model from Hugging Face’s Transformers library as the backbone BERT layer, leveraging its large-scale pretrained parameters for advanced natural language understanding. This layer captures rich contextual information from complex inputs such as JSONLformatted data and applies dropout between transitions to reduce overfitting and enhance robustness.
On top of this, we integrate StarBlock, an eficient neural module centered on element-wise
multiplication (the Star Operation [
During testing, StarBERT independently processes each text sample in JSONL format, evaluates its
likelihood of belonging to one of six categories—Fully human-written, Human-initiated then
machinecontinued, Human-written then machine-polished, Machine-written then humanized (obfuscated),
Machine-written then human-edited, or Deeply-mixed text—and classifies it based on the highest score.
By fine-tuning hyperparameters such as learning rate and batch size, and optimizing with binary
cross-entropy loss, the model is calibrated for high performance on key metrics such as Macro Recall
and Macro F1. This setup enables StarBERT to meet the specific challenges of the PAN@CLEF 2025
[
text
DeBERTa-v3- Dropout large
StarBlock2d
Linear
output six labels
DW-Conv2d FC
GELU
FC sum
StarBlock2d
In our experimental setup, to evaluate the performance of StarBERT in the human-AI collaborative text classification task, we used the oficially provided subtask2_train.jsonl as the training set and subtask2_dev.jsonl as the validation set for training or fine-tuning. To ensure the reproducibility of our experiments, we fixed the random seed to 42. This setting guarantees consistent results across all stages of the training process, including data loading and model initialization. To balance computational eficiency and learning depth, we trained the model on a CUDA-enabled GPU using the AdamW optimizer. The maximum input length for the model was set to 512 tokens. The learning rate was set to 2e-5, which is a commonly efective value for most text classification tasks. The batch size was 8, and the model was trained for one epoch.
In this task, three evaluation metrics were defined: Macro Recall, Macro F1, and Accuracy. These metrics are designed to comprehensively reflect the model’s performance on the collaborative human-AI text classification task. A detailed introduction to each metric is provided below.
Macro Recall: Macro Recall is the arithmetic mean of the recall scores across all classes, giving equal
importance to the recognition ability of each class [
Macro Recall = 1 ∑︁
=1 + ( = 6) where denotes the number of true positives for class (correctly predicted samples), and represents the number of false negatives for class (missed samples).
Macro F1: Macro F1 is the arithmetic mean of the F1-scores across all classes, which balances both
precision and recall [
Macro F1 = =1 1 ∑︁ 2 · Precision · Recall
Precision + Recall
This macro-averaging strategy calculates the F1-score for each class independently and then averages them, which avoids bias toward majority classes.
Accuracy: Accuracy measures the proportion of correctly predicted samples among all samples,
reflecting the overall classification performance [
total Accuracy =
(total = total number of samples)
These three metrics jointly provide an efective and balanced evaluation of our model’s capability in the human-AI collaborative text classification task.
Our StarBERT model demonstrated strong and stable performance in the PAN 2025 human-AI collaborative text classification task, showing substantial efectiveness across several key evaluation metrics. As shown in Table 1, StarBERT achieved a Macro Recall of 57.46%, significantly outperforming the baseline model’s 48.32%. This reflects its strong ability to accurately identify instances across all categories. Moreover, StarBERT achieved a Macro F1 score of 56.31%, well above the baseline’s 47.82%, indicating (1) (2) (3) that the model efectively balances predictive performance across categories. It avoids bias toward classes with more samples, which often occurs in imbalanced datasets. In terms of Accuracy, StarBERT reached 66.81%, again far exceeding the baseline score of 57.09%, ofering a clear and intuitive measure of the model’s overall classification capability in this task. In the oficial PAN 2025 leaderboard, our submission ranked 4th out of 22 participating teams. This result underscores the competitive advantage of our approach in efectively capturing discriminative textual features in the context of human-AI collaboration.
These outcomes validate that StarBERT not only embodies theoretical innovation but also demonstrates substantial practical efectiveness in this domain. The model is capable of distinguishing collaborative human-AI texts with high accuracy, making it a valuable tool for complex text analysis tasks.
StarBERT integrates the deep semantic understanding capabilities of DeBERTa-v3-large with the high-dimensional feature mapping power of StarBlock2d, achieving—for the first time—precise sixclass classification of human-AI collaborative texts. Experimental results demonstrate that the model significantly outperforms baseline models (e.g., RoBERTa-base) in the PAN 2025 task, achieving a Macro Recall of 57.46%, a Macro F1 score of 56.31%, and an Accuracy of 66.81%. These results validate StarBERT’s superiority in handling mixed-authorship texts.
The innovative Star Operation and enhanced positional encoding efectively capture the nonlinear characteristics and contextual dependencies inherent in collaborative texts. This research provides a scalable technical solution for promoting academic integrity and supporting content provenance in the era of AI-assisted writing.
Future work will focus on further optimizing model parameters, enhancing feature engineering techniques, and expanding the diversity of the training dataset. These improvements aim to increase the model’s generalization ability and performance across various textual contexts. This continued development is expected to further improve StarBERT’s detection accuracy and extend its applicability in real-world scenarios.
This work was supported by grants from the Guangdong-Foshan Joint Fund Project (No. 2022A1515140096) and Open Fund for Key Laboratory of Food Intelligent Manufacturing in Guangdong Province (No. GPKLIFM-KF-202305).
During the preparation of this work, the author(s) used GPT-4 and DeepSeek-R1 in order to: Grammar and spelling check. After using these tool(s)/service(s), the author(s) reviewed and edited the content as needed. full responsibility for the publication’s content.