This paper describes the approach of the Unibuc - NLP team in tackling the "Voight-Kampf" Generative AI Detection task from PAN at CLEF 2025 [1, 2]. In the Subtask 1: Voight-Kampf AI Detection Sensitivity we have explored diferent approaches of leveraging embeddings extracted from LLMs, using either classical ML models or further fine-tuning LLMs with a classification head for the downstream binary classification task. Our experiments showed that using a majority vote ensemble of classical ML models (Light GBM, XGB, Logistic Regression and SVM) is comparable with fine-tuning for downstream task achieving around 99.4% F1 score. For the Subtask 2: Human-AI Collaborative Text Classification we explored a way of combining features at diferent layers extracted using transformer backbone with layer-wise projection and attentive pooling which proved modest results.
As artificial intelligence continues to advance and generative AI technologies are more and more popular,
distinguishing between machine-generated and human-authored texts has become very dificult. The
"Voight-Kampf" Generative AI Detection task from PAN at CLEF 2025 [
For the Subtask 1: Voight-Kampf AI Detection Sensitivity we experimented various methods of utilizing the embeddings from large language models (LLMs), including classical machine learning approaches and fine-tuning classification head for the binary classification task. The results indicate that a majority vote ensemble of classical models - specifically LightGBM, XGBoost, Logistic Regression and SVM - ofers a slightly better performance and faster training times and resources compared to the ifne-runing approach.
For the Subtask 2: Human-AI Collaborative Text Classification, we have tried a diferent approach by fusing together layers of a transformer backbone model with layer-wise projection and attentive pooling strategies.
The detection of AI-generated text has become a rapidly evolving research area, particularly with the advent of large language models. Recent shared tasks such as the PAN 2024 "Voight-Kampf"
Generative AI Authorship Verification challenge have provided valuable benchmarks and insights into
the capabilities and limitations of current detection systems [
The first subtask is formulated as a binary classification problem and the dataset contains either humanwritten texts (label 0) or ai-written texts (label 1). The genre of the text is also available and it can be one of the following: essays, news or fiction. Some of the news texts were sampled from the last year task dataset [12].
Overall the train split has 23, 707 samples and dev split has 3, 589. The distribution of the labels in each split is shown in Table 1. The predominant genre for both labels is fiction, followed by essays and news. As for the models used to generate text, gpt [13] comes first followed by mistral [ 14], gemini [15], llama [16] and deepseek [17] (more info in section 7 and Figure 2).
The second subtask is a multi-class classification problem where the goal is to classify texts into six distinct categories based on the nature of human and machine contributions: • Fully human-written: The document is entirely authored by a human without any AI assistance. • Human-initiated, then machine-continued: A human starts writing, and an AI model completes the text. • Human-written, then machine-polished: The text is initially written by a human but later refined or edited by an AI model. • Machine-written, then machine-humanized (obfuscated): An AI generates the text, which is later modified to obscure its machine origin. • Machine-written, then human-edited: The content is generated by an AI but subsequently edited or refined by a human. • Deeply-mixed text: The document contains interwoven sections written by both humans and
AI, without a clear separation.
Overall the train split has 288, 919 samples and dev split has 72, 661. The distribution of the labels in each split is shown in Table 2.
In the train split, models used for text rewriting and generation are mostly from mistral [14], gpt [13], gemini [15] and llama [16] (more info in Figure 3). Most of the texts that are fully human-written, machine-written then machine-humanized or human-written then machine-polished come from the LLM-DetectAlve dataset [18]. The next big source of the texts is TriBERT [19] where ChatGPT is mainly used to produce deeply-mixed texts and human-initiated then machine-continued.
On the other hand, in the dev split llama [16] is mainly used for text generation followed by gpt [13] and mistral [14], gemini [15] being less used in this split. While in this split a good amount of fully human-written, machine-written then machine-humanized or human-written then machine-polished texts come from the LLM-DetectAlve dataset [18], there is a huge amount of human-written and human-initiated then machine-continued texts that are sampled from m4gt-bench [20]. Also in dev set there are more samples from RoFT [21] than in the train set (more information in Figure 4).
We explored 3 diferent ways of laveraging LLM embeddings for either binary or multi-class classification tasks. For subtask 1 we conducted experiments with systems described in sections 4.1 and 4.2. For the second subtask we did our experiments with systems from sections 4.2 and 4.3.
This approach is using the last layer embeddings extracted from Qwen3-0.6B [22] to train diferent
classic machine learning classifiers:
• Light GBM [23]
• XGBoost [
In the final approach we used an ensemble of the above models combining predictions in a majority voting manner. So the final prediction is the class voted by most of the models in the ensemble. The embeddings model was kept frozen and used only to extract embeddings from the last layer.
The next approach also laverages the rich representational capacity of large language models, which can capture both complex linguistic patterns and semantic relationships through their high-dimensional vector space representations, and used as input for the final classification head. Experiments were conducted with embeddings extracted using difernet LLM backbones: Qwen2.5-0.5B 1, Qwen3-0.6B 2, Mistral7B-v0.1 3, Llama-3.1-8B 4 and fine-tuning them with a classification head on top of the last token.
We experimented this approach only for subtask 2. It uses embeddings from transformer-based models, combines them using frame-wise projection, pooling strategies and a lightweight classification head. 4.3.1. Backbone and Feature Extractor The model uses a pretrained transformer as a backbone to extract contextualized token embeddings. This part of the model is always kept frozen during the training process. 4.3.2. Layer Aggregation Block The information across multiple hidden layers of the transformer are aggregated using a learnable, softmax-normalized weighted scheme. The weighted sum of hidden states is normalized via layer normalization (without element-wise afine tranformation), producing a single sequence of contextual embeddings for each input. 4.3.3. Frame-wise Processing Block The aggregated embeddings are further processed by a configurable frame-wise module, which can be either a simple linear projection or a small neural network. This step reduces the dimensionality and introduces additional non-linearity, preparing the features for temporal pooling. 4.3.4. Time Pooling Strategies To summarize the sequence of embeddings into a fixed-size representation, the model supports several time pooling strategies: • Statistical Pooling (SP): Concatenates the mean and standard deviation across the sequence. • Attentive Statistical Pooling (ASP): Computes attention-weighted statistics using a learnable attention mechanism.
For both subtasks, we kept 20% of the training set for validation and hyperparameter tuning (taking into
consedireation the label distribution as well), and the dev split was used to test the final models. For
the dataset splits and implementation for classical machine learning models (4.1) we used scikit-learn
[
In order to find the best parameters for each of the classical models from 4.1 we did hyperparameter
tuning using optuna [
For the second approach 4.2 we loaded the backbone LLMs using QLoRA [
The backbone transformer model used for the last approach 4.3 is RoBERTa [
The evaluation platform used for this challenge was TIRA [
The metrics used to evaluate the system are the following: • ROC-AUC: The area under the ROC (Receiver Operating Characteristic) curve. • Brier: The complement of the Brier score (mean squared loss). • C@1: A modified accuracy score that assigns non-answers (score = 0.5) the average accuracy of the remaining cases. • F1: The harmonic mean of precision and recall. • F0.5u: A modified F0.5 measure (precision-weighted F measure) that treats non-answers (score = 0.5) as false negatives.
• Mean: The arithmetic mean of all the metrics above.
The overall results of the first approach 4.1 are available in Table 3 and those using the second approach 4.2 are available in Table 4.
Looking at Figure 6 it seems like the biggest problem for the model is that is classifies fiction texts generated with gemini-1.5-pro and llama-3.1-8b-instruct as human-written fiction texts. In the embeddings visualization plot Figure 5 we can see that there are some red squares (ai-written fiction) that are really close to the blue squares (human-written fiction).
The metrics used to evaluate the system are the following: • Recall (Macro): Macro recall is the average recall computed independently for each class, measuring the proportion of true positives identified out of all actual instances for each class, and then averaging across all classes. • F1 (Macro): Macro F1 is the average of the F1 scores calculated for each class, where the F1 score is the harmonic mean of precision and recall, providing a balanced measure of a model’s accuracy across all classes. • Accuracy: Accuracy is the proportion of all predictions that are correct, calculated as the number of correct predictions divided by the total number of predictions.
Looking at Figure 7 we can see that the hardest dataset is m4gt-bench (which was not present in the training dataset at all) with 28, 192 samples that were not classified as human-initiated, then machinecontinued (97% of the samples from m4gt-bench with this label) and 1, 750 were not classifier as fully human-written (55% of the samples from m4gt-bench with this label). Furthermore looking at Figure 8 and Figure 9 we see that llama2 is the hardest model to be detected (also not present in the training split at all) with 11, 793 samples that were human-written, then machine-continued misclassified as human-written, then machine-polished. We have also reported the metrics we got on the dev set when removing the samples from m4gt-bench Table 5 to prove that the architecture we used for this type of classification fails to generalize well on unseen data, but manages to learn good enough representations from the training set.
The experimental results demonstrate that both classical machine learning models and fine-tuned large language models (LLMs) achieve exceptionally high performance on the subtask 1 of distinguishing between human-written and ai-written texts, with F1, ROC-AUC, and related metrics consistently above 0.99 for most models. However, detailed error analysis reveals that the main challenge lies in distinguishing between human-written and AI-generated fiction, especially for texts produced by advanced models such as gemini-1.5-pro and llama-3.1-8b-instruct, which were also fewer in the train set, where the model tends to misclassify AI-generated fiction as human-written. This is further supported by the embeddings visualization, which shows significant overlap between these classes.
For the human-AI collaborative text classification subtask, the results are more modest, with macro F1 and recall scores around 0.43 and accuracy at 0.51, suggesting that this remains a challenging problem and that current models struggle to generalize in this setting, the diference between human-written then machine-continued and human-written then machine-polished being very subtle and hard to be detected, especially when the generative model has never been seen in the training dataset.
Future work will focus on addressing these limitations by exploring more sophisticated representation learning techniques, incorporating additional context or metadata, and experimenting with advanced ensemble methods. Furthermore, targeted data augmentation and adversarial training could help the models better distinguish subtle diferences between human and machine-generated texts, particularly in challenging genres such as fiction. Finally, interpretability analyses and error-driven model refinement will be essential to further improve robustness and reliability in real-world applications.
This research is supported by the project “Romanian Hub for Artificial Intelligence - HRIA”, Smart Growth, Digitization and Financial Instruments Program, 2021-2027, MySMIS no. 334906. Computational Linguistics, Punta Cana, Dominican Republic, 2021, pp. 2001–2016. URL: https: //aclanthology.org/2021.findings-emnlp.172/. doi: 10.18653/v1/2021.findings-emnlp.172. [9] A. Hans, A. Schwarzschild, V. Cherepanova, H. Kazemi, A. Saha, M. Goldblum, J. Geiping, T. Goldstein, Spotting llms with binoculars: Zero-shot detection of machine-generated text, 2024. URL: https://arxiv.org/abs/2401.12070. arXiv:2401.12070. [10] M. Jakesch, A. Bhat, D. Buschek, L. Zalmanson, M. Naaman, Co-writing with opinionated language models afects users’ views, in: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, ACM, 2023, p. 1–15. URL: http://dx.doi.org/10.1145/3544548.3581196. doi:10.1145/3544548.3581196. [11] V. S. Sadasivan, A. Kumar, S. Balasubramanian, W. Wang, S. Feizi, Can ai-generated text be reliably detected?, 2025. URL: https://arxiv.org/abs/2303.11156. arXiv:2303.11156. [12] J. Bevendorf, M. Wiegmann, J. Karlgren, L. Dürlich, E. Gogoulou, A. Talman, E. Stamatatos, M. Potthast, B. Stein, Overview of the ‘voight-kampf’ generative AI authorship verification task at PAN and ELOQUENT 2024, in: CEUR Workshop Proceedings, volume 3740, 2024, p. Paper 225.
URL: https://ceur-ws.org/Vol-3740/paper-225.pdf, accessed: 2025-05-30. [13] OpenAI, Gpt-4 technical report, 2024. URL: https://arxiv.org/abs/2303.08774. arXiv:2303.08774. [14] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, W. E. Sayed, Mistral 7b, 2023. URL: https://arxiv.org/abs/2310.06825. arXiv:2310.06825. [15] G. Team, Gemini: A family of highly capable multimodal models, 2025. URL: https://arxiv.org/abs/ 2312.11805. arXiv:2312.11805. [16] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, G. Lample, Llama: Open and eficient foundation language models, 2023. URL: https://arxiv.org/abs/2302.13971. arXiv:2302.13971. [17] DeepSeek-AI, Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL: https://arxiv.org/abs/2501.12948. arXiv:2501.12948. [18] M. Abassy, K. Elozeiri, A. Aziz, M. N. Ta, R. V. Tomar, B. Adhikari, S. E. D. Ahmed, Y. Wang, O. Mohammed Afzal, Z. Xie, J. Mansurov, E. Artemova, V. Mikhailov, R. Xing, J. Geng, H. Iqbal, Z. M. Mujahid, T. Mahmoud, A. Tsvigun, A. F. Aji, A. Shelmanov, N. Habash, I. Gurevych, P. Nakov, LLMDetectAIve: a tool for fine-grained machine-generated text detection, in: D. I. Hernandez Farias, T. Hope, M. Li (Eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, Miami, Florida, USA, 2024, pp. 336–343. URL: https://aclanthology.org/2024.emnlp-demo.35/. doi:10. 18653/v1/2024.emnlp-demo.35. [19] Z. Zeng, L. Sha, Y. Li, K. Yang, D. Gašević, G. Chen, Towards automatic boundary detection for human-ai collaborative hybrid essay in education, 2023. URL: https://arxiv.org/abs/2307.12267. arXiv:2307.12267. [20] Y. Wang, J. Mansurov, P. Ivanov, J. Su, A. Shelmanov, A. Tsvigun, O. M. Afzal, T. Mahmoud, G. Puccetti, T. Arnold, A. F. Aji, N. Habash, I. Gurevych, P. Nakov, M4gt-bench: Evaluation benchmark for black-box machine-generated text detection, 2024. URL: https://arxiv.org/abs/2402. 11175. arXiv:2402.11175. [21] L. Kushnareva, T. Gaintseva, G. Magai, S. Barannikov, D. Abulkhanov, K. Kuznetsov, E. Tulchinskii, I. Piontkovskaya, S. Nikolenko, Ai-generated text boundary detection with roft, 2024. URL: https: //arxiv.org/abs/2311.08349. arXiv:2311.08349. [22] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, Z. Qiu, Qwen3 technical report, 2025. URL: https://arxiv.org/abs/2505.09388. arXiv:2505.09388. [23] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, T.-Y. Liu, Lightgbm: A highly eficient
During the preparation of this work, the authors used ChatGPT in order to: Generate literature review, Grammar and spelling check. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content.