This submission addresses Subtask 1 of the Voight-Kampf Generative AI Detection task, which is part of the PAN 2025 lab. The goal of the subtask is to distinguish AI-generated texts from human-written ones, even when the machine-generated texts have been intentionally obfuscated to appear more human-like. As Large Language Models (LLMs) continue to improve in fluency and coherence, this distinction becomes increasingly dificult and requires robust detection strategies. This submission introduces a zero-shot method based on token-level statistics, which are extracted from two pre-trained LLMs: a base model and an instruction-tuned model. This method LOG-AID computes five core features: mean surprisal under each model, Jensen-Shannon divergence between their predictive distributions, average entropy diference, the mean entropy of the base model and the average logarithmic rank of the ground-truth tokens. These features are combined into a fixed-size vector and classified using a logistic regression model. On the oficial test set, the proposed system achieved a mean score of 0.827 across five metrics, surpassing strong baselines such as Binoculars (0.818) and PPMd Compression (0.758). In particular, the combination of uncertainty-based measures (surprisal, entropy) and rank-based features (log-rank) enhances discriminative power. This contribution ofers a simple, interpretable and self-contained classification approach that does not require any fine-tuning. The method relies solely on internal probability structures of pre-trained models and may serve as a lightweight baseline for future work in AI text detection.
LLMs ofer considerable benefits across domains. For example, they can enhance productivity by
accelerating content creation and software development [
The Voight-Kampf Generative AI Detection (VKGen) task [
These five features are aggregated and passed to a logistic regression classifier.
Unlike many existing systems, this method does not rely on text similarity, fine-tuned classifiers or data augmentation. Instead, it exploits model-intrinsic probabilities to characterize how expected a text appears from the perspective of diferent LLMs. This lightweight framework aims to provide a transparent and modular baseline for robust detection.
The participating methods in the PAN 2024 ’Voight-Kampf Generative AI’ Competition showed a wide
range of methods. Many models were based on BERT or DeBERTa, either fine-tuned or combined with
additional components such as R-Drop, LSTM, CNN or contrastive learning [
The GLTR (Giant Language model Test Room) method, presented by Gehrmann et al., is a visualization tool for the recognition of AI-generated texts [12]. It is assuming that generative language models prefer to choose words from the upper range of the probability scale. GLTR uses the prediction distributions of an LLM (e. g., GPT-2) to calculate the probability, rank and entropy of the prediction distribution for each token. Here, the rank refers to the occurrence in the vocabulary when this is sorted according to the probability of a token in its previous text sequence. In a user study, the tool increased the recognition rate of generated texts from 54% to 72% [12]. In the publication on DetectLLM, two powerful zero-shot methods for recognizing machine-generated texts were presented [13]. The methods DetectLLM-LRR and DetectLLM-NPR use either the ratio of log likelihood to log rank or the change of the log rank value under specific text perturbations to identify generated texts based on their typical statistical instabilities. However, the average logarithmic rank position of the tokens (log(rank)) already proved to be a particularly strong baseline, achieving higher ROC values compared to the non-logarithmic rank [13].
In addition Venkatraman et al. showed that surprisal can be used for the AI detection task [14]. Surprisal describes how unexpected or surprising a token is in a given context. Their method GPT-who calculates its mean value, variance and the diferences between consecutive tokens. Furthermore, texts were segmented in 20 token long subtexts. For all 20-token-segments the surprisal features were calculated using a sliding window. The most extreme surprisal variances were extracted as additional features. These features were classified using a logistic regression. The approach achieved an average 1 score of 0.88 on the TuringBench dataset [14].
Overall, logit-based metrics, including surprisal or entropy, in combination with simple classifiers like logistic regression, can achieve remarkably strong performance in recognising machine-generated text.
Subtask 1 of the PAN 2025 VKGen Challenge addresses the binary classification problem of determining
whether a given text was authored by a human or generated by an AI system. In contrast to previous
years, this year’s task introduces additional challenges by incorporating adversarially obfuscated texts,
designed to humanize the writing style [
It should be emphasized that the LOG-AID method always outputs predictions without defining an
uncertainty range for which = 0.5 applies. In other words, we do not have any non-predictions,
which have an efect on performance metrics such as @1. In this specific case, @1 equals the standard
accuracy metric, according to Stamatatos et al. [15]. The participants were provided with two data
sets: a training set with 17,730 texts (9,101 human, 8,629 machine) and a validation set with 3,589 texts
(1,277 human, 2,312 machine). The texts come from three genres (fiction, news, essays) and exhibit
a wide variety of styles and models. A total of 29 diferent AI models were used, with GPT models
dominating. Each entry contains the label, the genre, the text and, where applicable, the generating
model. A separate test set was retained and used exclusively for the final evaluation [
The proposed detection method LOG-AID for the PAN25 Voight-Kampf Challenge adopts a two-stage architecture. In the first stage, each input text is analyzed independently using two pre-trained causal language models: Falcon-7B1 and its instruction-tuned variant Falcon-7B-Instruct2. These models operate in a zero-shot, autoregressive setting to compute token-level output distributions (e. g., logits). From these, six interpretable statistical features are derived, including mean surprisal, entropy diference and Jensen-Shannon divergence for both models. As well as entropy and the mean logit rank for the base model. To ensure eficiency, models are loaded sequentially. In the second stage, the feature vector is standardized using a z-score normalization and passed to a logistic regression classifier trained to distinguish between human and AI-generated texts.
Surprisal quantifies how surprising a token is for a language model in the given context < and corresponds to the negative logarithm of the associated prediction probability [14]: Surprisal () = − log ( | <) (1) In our approach, this metric is averaged over all tokens of a text. This reflects the average model uncertainty in predicting the words that actually occur. A lower mean surprisal value indicates a higher predictability of the text [14]. With Shanon entropy, the entropy of the prediction probability distribution ( | <) is calculated for each token : 1https://huggingface.co/tiiuae/falcon-7b 2https://huggingface.co/tiiuae/falcon-7b-instruct () = − ∑︁ log =1 Mean Entropy = 1 ∑︁ ()
=1 Δ = 1 ∑︁ ⃒⃒ () () − () ()⃒⃒⃒ =1 ⃒ In addition, the diference between the mean entropies of two models is calculated to determine divergent uncertainties between a base model and an instruct model: To quantify the typicality of a token under the model’s predictive distribution, we compute the logarithmic rank of each observed token within the vocabulary. The rank measures how highly the actual token is located in the model’s predicted probability distribution ( | <), where < denotes the preceding context [12]. Formally, let rank () ∈ N denote the position of the token in the list of all vocabulary items ∈ , sorted by descending probability ( | <). The average log-rank over a sequence of tokens provides a robust scalar feature that reflects the model’s overall perception of how typical the observed sequence is: Here is the size of the vocabulary and is the predicted probability for the th token. In this implementation, entropy is computed per token using the scipy.stats.entropy 3 function applied to the softmax-normalized logit distributions. The average entropy is then calculated as the average of all token-specific entropy values over the entire text: (2) (3) (4) (5) (6) (7) Mean log - Rank() = 1 ∑︁ log (rank()) =1 where is the token at position and rank () is the index (starting at 1 ) of in the sorted distribution. Low log-rank values indicate that tokens consistently appear near the top of the model’s predicted distribution-suggesting a more stereotyped or expected sequence-while high values reflect less typical or more surprising lexical choices [12]. The Jensen-Shannon divergence (JSD) is used to quantify the divergence between the conditional probability distributions of two language models at each token position. It compares two probability distributions and (in this case the logit probabilities of the base-model and the instruct-model) over the vocabulary . The JSD is defined as [16]: JSD( ‖) = Here, KL denotes the Kullback-Leibler divergence and the mixed distribution. The Kullback-Leibler divergence between two discrete probability distributions and over the vocabulary is defined as [16]: KL( ‖) = ∑︁ log =1 where and denote the predicted probabilities of token under and , respectively. Since JSD is symmetric and restricted to the interval [0, log 2], it is particularly suitable for comparing probabilistic model [16]. In this method, the JSD is calculated for each token based on the previous tokens <. The mean JSD across all tokens serves as a measure of the average prediction deviation between the base and instruct models. 3https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.entropy.html
The final prediction is generated using a logistic regression classifier trained on the six-dimensional feature-vectors. Prior to classification, all feature values are standardized using z-score normalization (StandardScaler4). The logistic regression model5 is configured with L2 regularization (default penalty), a regularization strength parameter C=1.0, and a maximum iteration limit of 1000. To account for class imbalance, the class_weight parameter is set to ’balanced’. The texts are processed in batches of the size 8. Training was conducted on a single NVIDIA H100 GPU (80GB HBM3) with CUDA 12.7, using approximately 60 GB of GPU memory. The program was then made accessible to the standardized evaluation procedures on the TIRA platform during the course of the competition and tested there. TIRA is a sandbox-based evaluation platform that enables the reproducible, isolated and fair execution of participant solutions [17].
(8)
(9)
(10)
To evaluate the efectiveness of the proposed detection method, we report results on the oficial PAN25
test set. Evaluation was carried out by the competition organizers on data unknown to participants
(see Tab. 1). All scores were computed using the PAN evaluation toolkit and include the following
metrics: AUC, Brier score complement, C@1, F1 score, F0.5u, and the arithmetic mean of these five
values [
+ 1 (︂ · )︂ Where is the total number of instances, is the number of correctly classified cases and is the · disappears. The accuracy can be derived from the c@1 term as follows: number of non-predicted cases. Since LOG-AID always makes a prediction, = 0, which is why [21]: 4https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html 5https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html =
TP + TN TP + TN + FP + FN = Accuracy As this is an unbalanced data set, the larger class is more important in the evaluation. For this reason, an additional metric, the balanced accuracy curve (BAC), is listed in Table 1. This is defined as follows BAC =
TPR + TNR 2 = 1 (︂
TP 2 TP + FN +
TN FP + TN ︂) 5. Conclusion This work presents a lightweight approach to detecting machine-generated text, based on a compact set of statistical features extracted from the output distributions of two pre-trained language models. Without relying on fine-tuning or large-scale training, the system achieves competitive results in the PAN25 Voight-Kampf Challenge, surpassing several strong baselines. By combining token-level metrics such as surprisal, entropy, log-rank, and Jensen-Shannon divergence into a logistic regression classifier, the system captures robust signals of artificiality and human-like variation. Empirical results demonstrate high accuracy on the test set. The method could be supplemented with additional features in future work. For example, not only the mean of entropy and surprisal could be computed, but also their standard deviations or burstiness. Moreover, it would be worthwhile to explore topic-masking techniques such as POSNoise [22] to guarantee a detection robustness against topic-related biases.
This research work was supported by the National Research Center for Applied Cybersecurity ATHENE. ATHENE is funded jointly by the German Federal Ministry of Education and Research and the Hessian Ministry of Higher Education, Research and the Arts.
Mean
During the preparation of this paper, the authors used ChatGPT and DeepL for rewording, grammar and spelling checks. All content was subsequently reviewed and edited by the authors, who take full responsibility for the final version of the publication. of Large Language Models for Machine-Generated Text Detection, in: G. Faggioli, N. Ferro, P. Galuščáková, A. G. S. Herrera (Eds.), Working Notes Papers of the CLEF 2024 Evaluation Labs, CEUR-WS.org, 2024, pp. 2901–2912. URL: http://ceur-ws.org/Vol-3740/paper-281.pdf. [10] A. Hans, A. Schwarzschild, V. Cherepanova, H. Kazemi, A. Saha, M. Goldblum, J. Geiping, T. Goldstein, Spotting LLMs with Binoculars: Zero-Shot Detection of Machine-Generated Text, in: Proceedings of the 41st International Conference on Machine Learning, ICML’24, JMLR.org, 2024, pp. 17519 – 17537. doi:10.5555/3692070.3692768. [11] J. Wu, S. Yang, R. Zhan, Y. Yuan, L. S. Chao, D. F. Wong, A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions, Computational Linguistics 51 (2025) 275–338. URL: https://aclanthology.org/2025.cl-1.8/. doi:10.1162/coli_a_00549. [12] S. Gehrmann, H. Strobelt, A. Rush, GLTR: Statistical Detection and Visualization of Generated Text, in: M. R. Costa-jussà, E. Alfonseca (Eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Association for Computational Linguistics, Florence, Italy, 2019, pp. 111–116. doi:10.18653/v1/P19-3019. [13] J. Su, T. Zhuo, D. Wang, P. Nakov, DetectLLM: Leveraging Log Rank Information for ZeroShot Detection of Machine-Generated Text, in: H. Bouamor, J. Pino, K. Bali (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Association for Computational Linguistics, Singapore, 2023, pp. 12395–12412. doi:10.18653/v1/2023.findings-emnlp.827. [14] S. Venkatraman, A. Uchendu, D. Lee, GPT-who: An Information Density-based Machine-Generated Text Detector, in: K. Duh, H. Gomez, S. Bethard (Eds.), Findings of the Association for Computational Linguistics: NAACL 2024, Association for Computational Linguistics, Mexico City, Mexico, 2024, pp. 103–115. doi:10.18653/v1/2024.findings-naacl.8. [15] E. Stamatatos, W. Daelemans, B. Verhoeven, M. Potthast, B. Stein, P. Juola, M. Sanchez-Perez, A. Barrón-Cedeño, Overview of the Author Identification Task at PAN 2014, in: L. Cappellato, N. Ferro, M. Halvey, W. Kraaij (Eds.), Working Notes Papers of the CLEF 2014 Evaluation Labs, volume 1180 of Lecture Notes in Computer Science, 2014. URL: https://ceur-ws.org/Vol-1180/ CLEF2014wn-Pan-StamatosEt2014.pdf. [16] F. Nielsen, On the Jensen–Shannon Symmetrization of Distances Relying on Abstract Means,
Entropy 21 (2019). doi:10.3390/e21050485. [17] M. Fröbe, M. Wiegmann, N. Kolyada, B. Grahm, T. Elstner, F. Loebe, M. Hagen, B. Stein, M. Potthast, Continuous Integration for Reproducible Shared Tasks with TIRA.io, in: J. Kamps, L. Goeuriot, F. Crestani, M. Maistro, H. Joho, B. Davis, C. Gurrin, U. Kruschwitz, A. Caputo (Eds.), Advances in Information Retrieval. 45th European Conference on IR Research (ECIR 2023), Lecture Notes in Computer Science, Springer, Berlin Heidelberg New York, 2023, pp. 236–241. URL: https://link. springer.com/chapter/10.1007/978-3-031-28241-6_20. doi:10.1007/978-3-031-28241-6_20. [18] C. Cortes, M. Mohri, Auc optimization vs. error rate minimization, in: S. Thrun, L. Saul, B. Schölkopf (Eds.), Advances in Neural Information Processing Systems, volume 16, MIT Press, 2003. URL: https://proceedings.neurips.cc/paper_files/paper/2003/file/ 6ef80bb237adf4b6f77d0700e1255907-Paper.pdf. [19] S. A. Hicks, I. Strümke, V. Thambawita, M. Hammou, M. A. Riegler, P. Halvorsen, S. Parasa, On evaluation metrics for medical applications of artificial intelligence, Scientific Reports 12 (2022) 5979. URL: https://doi.org/10.1038/s41598-022-09954-8. doi:10.1038/s41598-022-09954-8. [20] L. Hoessly, On misconceptions about the brier score in binary prediction models, 2025. URL: https://arxiv.org/abs/2504.04906. arXiv:2504.04906. [21] K. H. Brodersen, C. S. Ong, K. E. Stephan, J. M. Buhmann, The balanced accuracy and its posterior distribution, in: 2010 20th International Conference on Pattern Recognition, 2010, pp. 3121–3124. doi:10.1109/ICPR.2010.764. [22] O. Halvani, L. Graner, POSNoise: An Efective Countermeasure Against Topic Biases in Authorship Analysis, in: Proceedings of the 16th International Conference on Availability, Reliability and Security, ARES ’21, Association for Computing Machinery, New York, NY, USA, 2021. URL: https://doi.org/10.1145/3465481.3470050. doi:10.1145/3465481.3470050.