Detecting AI-generated text is increasingly important to prevent misuse in education, journalism, and social media, where synthetic fluency can obscure misinformation. This paper presents our solution for the Generative AI Authorship Verification Task at PAN 2025, where the objective is to distinguish machine-generated text from human-written content. We propose DivEye, a novel detection framework that leverages surprisal-based features to capture fluctuations in lexical and structural unpredictability, a signal more prominent in human-authored text. Our method performs competitively across diverse text domains and models, especially on challenging cases where model-generated text closely resembles human writing, and also outperforms the four oficial baselines of the PAN 2025 task.
Large Language Models (LLMs) are widely used in tasks from personal assistance to content creation
[
Reliable AI-text detection is vital for combating risks like misinformation, academic dishonesty, professional misconduct, and the suppression of genuine human writing [10, 11, 12]. Traditional supervised detectors [13, 14, 15] rely on labeled datasets but often fail to generalize to unseen models or domains [16, 11], especially as new LLMs emerge. Zero-shot detectors [17, 18, 19, 20] address this by leveraging statistical signals or LLMs at inference time, ofering scalable, model-agnostic detection critical for maintaining platform integrity.
Contributions. In this work, we present DivEye, a zero-shot framework for AI-generated text detection submitted to the PAN@CLEF 2025 Generative AI Authorship Verification task [ ? ? ]. The challenge centers on identifying the human-written text when presented with a pair comprising one human and one machine-authored sample. Our method leverages diversity-based statistical features computed over token-level surprisal [21] sequences from a reference language model. Unlike finetuned classifiers or signature-based detectors, DivEye captures distributional irregularities inherent in AI-generated content by measuring surprisal variance, entropy, and other diversity features. These metrics are grounded in linguistic theory and require no access to a specific text-generation LM. DivEye is model-agnostic, scalable, and can operate without retraining, making it suitable for real-world deployment. Notably, it complements existing detectors by revealing statistical signals often missed by black-box or fine-tuned approaches. Our results show strong generalization across domains and model families, achieving competitive performance in this challenging verification setting.
Computation of Surprisal
via a Frozen LLM
LLM Input Text
Frozen LLM
Extraction of Surprisal per Token
Split into tokens & compute per token
DivEye Feature
Extraction Distribution Features Mean, Variance, Skew, Kurtosis First-Order Features Second-Order Features Variance-II, Entropy, Autocorrelation
External AI-Detection Framework (if any)
Feature Concatenation
Feature Concatenation + Classification Binary Classifier
(XGBoost) Framework to be
Boosted
Human Text
AI Text
The rise of LLMs has enabled machine-generated text that closely mimics human writing by approximating the true conditional distribution of natural language, human( | <), through training on large human-written corpora [22, 23]. The LLM’s learned distribution, LLM( | <), is used to sequentially generate tokens during inference via sampling [24]. Despite their fluency, LLMs imperfectly approximate human language (LLM ̸= human) [25, 26], and this subtle diference is the crux of AI text detection.
Existing detection methods fall into three categories: watermarking, supervised / fine-tuned and zero-resource detection. Watermarking [27, 28, 29] embeds patterns in generated text but requires model access or fine-tuning, limiting use in black-box or adversarial settings. Zero-resource methods need no model knowledge and rely on statistical or learned diferences between human and AI text, further divided into statistical and training-based approaches.
Supervised / Fine-tuned detection methods [30, 31, 32] train classifiers, such as fine-tuned transformers on a labeled corpora of human and AI text. While these models can be accurate, they often fail to generalize across domains or against adversarial paraphrasing, especially when trained on specific generators or prompts. Statistical / Zero-shot detection methods refers to identifying AI-generated text without task-specific training, either by leveraging LLM probability cues or prompting LLMs directly as detectors. For example, methods like Entropy [33], LogRank [34], DetectGPT [17, 19], and Binoculars [35] use of-the-shelf LLMs to evaluate the consistency of token predictions under masked or perturbed inputs.
Despite progress, AI-text detection remains unsolved. We move beyond individual token probabilities to measure statistical diversity across token sequences, capturing variation in surprise and predictability.
One of the main challenges in detecting AI-generated text [34, 36] lies in the fact that while modern LLMs excel at generating fluent and coherent text, they often do so at the expense of variability and diversity [37].
We hypothesize that human-authored text naturally displays greater stylistic diversity and unpredictability than text produced by AI. Human writing tends to include creative and impulsive choices that introduce unexpected shifts, whereas large language models prioritize high-probability sequences [38], resulting in more uniform and predictable outputs. This hypothesis is supported by both intuitive reasoning and empirical findings (see Remark 1).
Remark 1: Proof Sketch Consider a text sequence = (1, 2, . . . , ) generated either by a human or by a language model . The language model defines a probability distribution () = ∏︀ =1 ( | <) where each token is chosen to maximize overall likelihood.
Humans, however, produce language through a complex, multi-layered cognitive process that balances informativeness, creativity, and contextual appropriateness, rather than strictly maximizing statistical likelihood. Formally, the surprisal of token under model is defined as: () = − log ( | <) Since is trained to assign high probability to plausible continuations, its outputs tend to minimize surprisal on average, implying that maximum likelihood generation compresses diversity:
E∼ [ ()] ≤ E∼ [ ()] where denotes the distribution of human-generated text.
Similarly, human language exhibits higher variance in surprisal due to spontaneous creative choices, idiomatic expressions, and stylistic variation, causing:
Var∼ [ ()] < Var∼ [ ()] We validate this theoretical intuition through empirical experiments detailed below, which confirm statistically significant diferences in surprisal and diversity metrics between human-written and AI-generated texts.
We empirically validate these theoretical claims using a dataset of 200 human-written essays and 200 GPT-4-Turbo-generated essays on matched topics, sourced from BiScope [39]. For each essay, we compute token-level surprisal scores using a fixed language model evaluator (GPT-2), then calculate the mean and variance of surprisal per essay. Figure 2a presents the distribution of mean surprisal scores across both groups, while Figure 2b shows the corresponding variance distributions. Human-written texts demonstrate a broader spread and heavier tails in both metrics, indicating greater unpredictability and stylistic richness. In contrast, AI-generated texts are more tightly clustered with lower mean surprisal and significantly reduced variance. These findings empirically corroborate our hypothesis: human language inherently reflects higher diversity and surprise, whereas AI-generated language, optimized for likelihood, tends toward more predictable and homogeneous patterns.
DivEye computes higher-order statistical features over surprisal sequences, enabling the capture of structural patterns that go beyond aggregate likelihood. More detailed theoretical foundations and experimental results are presented in the original DivEye paper [40].
Surprisal. Human language balances consistency with creative bursts, introducing novel expressions and stylistic variation. This diversity can be quantified using surprisal [ 41], the negative log-probability of a token given its context () = − log ( | 1, 2, . . . , − 1). For a sequence = 1, . . . , , surprisal ofers a principled measure of local unpredictability based on model log-probabilities.
Rather than examining individual token surprisals in isolation, we summarize their behavior through aggregate metrics. The mean surprisal ( ) serves as a coarse indicator of how “expected” a text is on average: Lower values suggest closer conformity to the model’s distribution, whereas higher values signal greater unpredictability. Moreover, human writing also exhibits fluctuations in predictability due to stylistic shifts, topic changes, or bursts of creativity, motivating the use of surprisal variance ( 2) 2.0 2.5 3.0 3.5 4.0 5 6 7 8 9 10 11 12 Mean Surprisal Surprisal Variance (a)
(b)
1 ∑︁(() − )2 Mean and Variance are not suficient.
Mean and variance capture surprisal’s central tendency and spread but miss deeper structural signals distinguishing human from AI text. Human writing often shows asymmetric surprisal distributions with bursts of creativity, causing occasional spikes in unpredictability. AI-generated text, optimized for consistency, tends toward more symmetrical distributions centered on high-probability tokens [25]. Skewness ( 1) measures this asymmetry, positive values indicate rare, surprising tokens typical of human writing, while kurtosis ( 2) reflects the frequency of extreme deviations, signaling stylistic diversity. These higher-order moments enable DivEye to detect subtle irregularities overlooked by methods focusing only on average behavior.
1 = 1 ∑︁ (︂ () − )︂ 3 =1 ; 2 = 1 ∑︁ (︂ () − )︂ 4 =1 abrupt shifts in topic or tone common in human writing.
Static metrics still miss temporal structure.
While static surprisal metrics (mean, variance, skewness, kurtosis) summarize overall unpredictability, they miss how it evolves across a sequence, a key trait separating human from AI text. To model these dynamics, we compute the first-order diference Δ = () − (− 1), with its mean (Δ ) and variance (Δ 2) capturing stylistic volatility, such as ( rate of surprisal change. From this, we extract: (1) variance ( Δ22 ) for erratic transitions; (2) entropy ℋΔ2 ) for irregularity; and (3) autocorrelation ( (Δ2)) for clustering of unpredictability bursts. These metrics uncover rhythmic, non-stationary patterns typical of human text but rare in the smoother, more uniform outputs of LLMs, ofering a richer signal for detection. These have been formally defined as: We also compute the second-order diference Δ2 = Δ − Δ− 1 to track fluctuations in the Δ = () − (− 1), Δ2 = Δ − Δ− 1, Δ = Δ22 = (Δ2) = 1 1 − 1 − 2 =2 =3 ∑︁ Δ,
Δ 2 = ∑︁(Δ2 − Δ2 )2,
ℋΔ2 = − E ︀[ (Δ2 − Δ2 )(Δ2+1 − Δ2 )
︀] ∑︁(Δ − Δ)
2 ∑︁ log , (1) (2) (3) (4) (5) where Δ2 is the mean of second-order diferences, and is the empirical probability of a value falling into bin after discretizing Δ2 for entropy computation. We provide empirical validation of these temporal features and their individual contributions to detection performance in Appendix A.
Combinations. Collectively, DivEye, formalized as () in Equation (6), encapsulates critical aspects of text generation that distinguish human creativity from algorithmically generated predictability, thereby serving as a robust basis for our detection framework. is a 9-dimensional vector of distributional, first-order, and second-order statistics, derived by passing text through an autoregressive LLM. These features feed a binary classifier, optionally combined with existing detector outputs. See Algorithm 1 for details.
DivEye as a booster. Existing detectors often fail against high-quality adversarial text that mimics human writing. DivEye provides a complementary signal by capturing statistical and temporal patterns of token-level unpredictability, orthogonal to traditional features. We enhance detectors by appending DivEye ’s feature vector to their outputs and training a lightweight meta-classifier (e.g., XGBoost [ 42], Random Forest [43]) on the combined representation. This fusion significantly improves performance on adversarial and out-of-distribution text, without retraining or altering the base model.
Datasets. We evaluate our zero-shot DivEye framework on a diverse suite of datasets that span a
wide range of generative models, domains, and adversarial strategies. To substantiate the methodology
described above1, our primary benchmark is the MAGE benchmark [44]. MAGE comprises eight distinct
testbeds covering multiple domains (e.g., Yelp [45], XSum [46], SciXGen [
This fine-grained evaluation setup enables us to isolate and analyze the contribution of diversitybased metrics across diferent domains and model architectures. Each testbed includes predefined training and evaluation splits, which we use accordingly. For full implementation details and extended experimental results on MAGE, we request the reader to refer to our original paper [40], where all MAGE-related experiments are presented and discussed in depth.
PAN Dataset. The PAN@CLEF 2025 Generative AI Author Verification Task [
Implementation Details & Metrics. Unless stated otherwise, we use GPT-2 to compute all DivEye
feature vectors. In score-only detection scenarios, predictions are based solely over concatenated
DivEye features. For both standalone and boosted setups, we train a lightweight XGBoost [42] classifier
as a meta-model, using only DivEye features in the former, and concatenating them with the original
detector’s prediction scores in the latter. We use an XGBoost classifier for binary classification as a
preliminary choice, without extensive comparison to other classifiers, leaving exploration of alternative
models for future work. We evaluate our method using the oficial PAN@CLEF 2025 evaluation platform
(TIRA [
• Brier Score: Measures the accuracy of probabilistic predictions by computing the mean squared error between predicted probabilities and true labels.
Baselines. We compare DivEye against a diverse set of baselines under two evaluation settings. As detailed in our original paper [40], for the MAGE benchmark, we evaluate both traditional statistical detectors and recent fine-tuned models, including RADAR [ 32], FastDetectGPT [17], Binoculars [35], and BiScope [39].
Although we do not explicitly report quantitative results in this manuscript for MAGE, we kindly refer readers to our original paper [40] for full empirical comparisons. Nevertheless, we summarize and discuss the key findings and core observations here to provide insight into the comparative performance of DivEye.
For the PAN@CLEF 2025 task, we follow the oficial evaluation protocol and compare against the
provided baselines: Linear SVM with TF-IDF features, Binoculars [35], and a PPMd compression-based
cosine similarity method [
We evaluate DivEye across a wide range of challenging testbeds to assess its robustness and adaptability under both domain and model distribution shifts. Our experiments span multiple testbeds from the MAGE benchmark [44], including both in-distribution and out-of-distribution scenarios. Across all settings, DivEye consistently outperforms existing zero-shot and fine-tuned baselines in terms of AUROC and average accuracy, demonstrating strong generalization to both familiar and novel generation patterns. In our original paper [40], we report detailed performance metrics for DivEye across these testbeds. The results highlight high AUROCs (e.g., 0.98 and 0.93 across domains and generator families), along with strong accuracy, underscoring the stability and robustness of our approach across diverse evaluation conditions.
To further assess robustness, we evaluate DivEye under adversarial conditions such as paraphrasing attacks. Even in these challenging scenarios, DivEye outperforms strong fine-tuned baselines by notable margins in both AUROC and average accuracy. In addition to accuracy, DivEye is highly eficient, processing each input in approximately 0.01 seconds due to its lightweight GPT-2 backbone and fast statistical feature computations. This makes it particularly well-suited for deployment in real-time or resource-constrained environments.
Finally, we demonstrate that DivEye’s diversity-based surprisal features substantially enhance the performance of existing detectors when used in combination. Fusing DivEye with other diverse detectors results in AUROC and accuracy gains exceeding 18.7%, showing that these features ofer complementary signals to traditional methods. For full experimental results, including quantitative comparisons and boosting analyses, we refer reviewers to our original paper [40].
We successfully participated in the PAN@CLEF2025 Generative AI Authorship Verification task using our proposed framework, DivEye, which leverages surprisal diversity for robust zero-shot detection. By integrating DivEye with BiScope, we achieved strong performance in distinguishing human-written from machine-generated texts, overperforming all given baselines. Our method shows high adaptability across domains and paraphrased inputs, indicating its efectiveness in real-world authorship verification scenarios.
We would like to thank the Data, Systems and High Performance Computing (DaSH) Lab2 and the PI, Prof. Arnab K. Paul, for providing the computational resources necessary to conduct our experiments.
The authors have not employed any Generative AI tools. [10] S. Abdali, R. Anarfi, C. Barberan, J. He, Decoding the ai pen: Techniques and challenges in detecting ai-generated text, 2024. URL: https://arxiv.org/abs/2403.05750. doi:https://doi.org/10.1145/ 3637528.3671463. arXiv:2403.05750. [11] H. D. S. Gameiro, A. Kucharavy, L. Dolamic, Llm detectors still fall short of real world: Case of llm-generated short news-like posts, 2024. URL: https://arxiv.org/abs/2409.03291. arXiv:2409.03291. [12] J. Wu, S. Yang, R. Zhan, Y. Yuan, L. S. Chao, D. F. Wong, A survey on llm-generated text detection: Necessity, methods, and future directions, Comput. Linguistics 51 (2025) 275–338. URL: https: //doi.org/10.1162/coli_a_00549. doi:10.1162/COLI\_A\_00549. [13] S. M. Shukla, C. Magoo, P. Garg, Comparing fine tuned-lms for detecting llm-generated text, in: 2024 3rd Edition of IEEE Delhi Section Flagship Conference (DELCON), IEEE, 2024, pp. 1–8. [14] I. Tolstykh, A. Tsybina, S. Yakubson, A. Gordeev, V. Dokholyan, M. Kuprashevich, Gigacheck:
Detecting llm-generated content, arXiv preprint arXiv:2410.23728 (2024). [15] R. Wang, H. Chen, R. Zhou, H. Ma, Y. Duan, Y. Kang, S. Yang, B. Fan, T. Tan, Llm-detector: Improving ai-generated chinese text detection with open-source llm instruction tuning, arXiv preprint arXiv:2402.01158 (2024). [16] J. Doughman, O. M. Afzal, H. O. Toyin, S. Shehata, P. Nakov, Z. Talat, Exploring the limitations of detecting machine-generated text, 2024. URL: https://arxiv.org/abs/2406.11073. arXiv:2406.11073. [17] G. Bao, Y. Zhao, Z. Teng, L. Yang, Y. Zhang, Fast-detectgpt: Eficient zero-shot detection of machinegenerated text via conditional probability curvature, 2024. URL: https://arxiv.org/abs/2310.05130. arXiv:2310.05130. [18] S. Gehrmann, H. Strobelt, A. M. Rush, Gltr: Statistical detection and visualization of generated text, 2019. URL: https://arxiv.org/abs/1906.04043. arXiv:1906.04043. [19] E. Mitchell, Y. Lee, A. Khazatsky, C. D. Manning, C. Finn, Detectgpt: Zero-shot machinegenerated text detection using probability curvature, 2023. URL: https://arxiv.org/abs/2301.11305. arXiv:2301.11305. [20] H. Wang, X. Luo, W. Wang, X. Yan, Bot or human? detecting chatgpt imposters with a single question, 2024. URL: https://arxiv.org/abs/2305.06424. arXiv:2305.06424. [21] E. G. Wilcox, T. Pimentel, C. Meister, R. Cotterell, R. P. Levy, Testing the predictions of surprisal theory in 11 languages, 2025. URL: https://arxiv.org/abs/2307.03667. arXiv:2307.03667. [22] B. Chen, X. Wang, S. Peng, R. Litschko, A. Korhonen, B. Plank, “seeing the big through the small”: Can LLMs approximate human judgment distributions on NLI from a few explanations?, in: Y. Al-Onaizan, M. Bansal, Y.-N. Chen (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, Association for Computational Linguistics, Miami, Florida, USA, 2024, pp. 14396–14419. URL: https://aclanthology.org/2024.findings-emnlp.842/. doi: 10.18653/v1/2024. findings-emnlp.842. [23] Y. Lu, J. Huang, Y. Han, B. Bei, Y. Xie, D. Wang, J. Wang, Q. He, Llm agents that act like us: Accurate human behavior simulation with real-world data, 2025. URL: https://arxiv.org/abs/2503.20749. arXiv:2503.20749. [24] Z. Zhou, X. Ning, K. Hong, T. Fu, J. Xu, S. Li, Y. Lou, L. Wang, Z. Yuan, X. Li, S. Yan, G. Dai, X.-P.
Zhang, Y. Dong, Y. Wang, A survey on eficient inference for large language models, 2024. URL: https://arxiv.org/abs/2404.14294. arXiv:2404.14294. [25] D. Ippolito, D. Duckworth, C. Callison-Burch, D. Eck, Automatic detection of generated text is easiest when humans are fooled, in: D. Jurafsky, J. Chai, N. Schluter, J. Tetreault (Eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online, 2020, pp. 1808–1822. URL: https://aclanthology.org/2020.acl-main.164/. doi:10.18653/v1/2020.acl-main.164. [26] C. R. Jones, S. Trott, B. Bergen, Comparing humans and large language models on an experimental protocol inventory for theory of mind evaluation (epitome), Transactions of the Association for Computational Linguistics 12 (2024) 803–819. URL: https://doi.org/10.1162/tacl_a_00674. doi:10. 1162/tacl_a_00674. [27] J. Kirchenbauer, J. Geiping, Y. Wen, J. Katz, I. Miers, T. Goldstein, A watermark for large language models, 2024. URL: https://arxiv.org/abs/2301.10226. arXiv:2301.10226. [28] Y. Liang, J. Xiao, W. Gan, P. S. Yu, Watermarking techniques for large language models: A survey, 2024. URL: https://arxiv.org/abs/2409.00089. arXiv:2409.00089. [29] A. Liu, L. Pan, Y. Lu, J. Li, X. Hu, X. Zhang, L. Wen, I. King, H. Xiong, P. S. Yu, A survey of text watermarking in the era of large language models, 2024. URL: https://arxiv.org/abs/2312.07913. arXiv:2312.07913. [30] Y. Chen, H. Kang, V. Zhai, L. Li, R. Singh, B. Raj, Token prediction as implicit classification to identify LLM-generated text, in: H. Bouamor, J. Pino, K. Bali (Eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Singapore, 2023, pp. 13112–13120. URL: https://aclanthology.org/2023.emnlp-main.810/. doi:10.18653/v1/2023.emnlp-main.810. [31] C. Mao, C. Vondrick, H. Wang, J. Yang, Raidar: generative ai detection via rewriting, 2024. URL: https://arxiv.org/abs/2401.12970. arXiv:2401.12970. [32] X. Hu, P.-Y. Chen, T.-Y. Ho, Radar: Robust ai-text detection via adversarial learning, 2023. URL: https://arxiv.org/abs/2307.03838. arXiv:2307.03838. [33] T. Lavergne, T. Urvoy, F. Yvon, Detecting fake content with relative entropy scoring, in: Proceedings of the 2008 International Conference on Uncovering Plagiarism, Authorship and Social Software Misuse - Volume 377, PAN’08, CEUR-WS.org, Aachen, DEU, 2008, p. 27–31. [34] S. S. Ghosal, S. Chakraborty, J. Geiping, F. Huang, D. Manocha, A. S. Bedi, Towards possibilities impossibilities of ai-generated text detection: A survey, 2023. URL: https://arxiv.org/abs/2310.15264. arXiv:2310.15264. [35] A. Hans, A. Schwarzschild, V. Cherepanova, H. Kazemi, A. Saha, M. Goldblum, J. Geiping, T. Goldstein, Spotting llms with binoculars: Zero-shot detection of machine-generated text, 2024. URL: https://arxiv.org/abs/2401.12070. arXiv:2401.12070. [36] V. S. Sadasivan, A. Kumar, S. Balasubramanian, W. Wang, S. Feizi, Can ai-generated text be reliably detected?, 2025. URL: https://arxiv.org/abs/2303.11156. arXiv:2303.11156. [37] C. Yang, A. Holtzman, How alignment shrinks the generative horizon, 2025. URL: https://arxiv.
org/abs/2506.17871. arXiv:2506.17871. [38] B. Park, J. Choi, Identifying the source of generation for large language models, 2024. URL: https://arxiv.org/abs/2407.12846. arXiv:2407.12846. [39] H. Guo, S. Cheng, X. Jin, Z. ZHANG, K. Zhang, G. Tao, G. Shen, X. Zhang, Biscope: AI-generated text detection by checking memorization of preceding tokens, in: The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL: https://openreview.net/forum? id=Hew2JSDycr. [40] A. R. Basani, P.-Y. Chen, Diversity boosts AI-generated text detection, in: Data in Generative Models - The Bad, the Ugly, and the Greats, 2025. URL: https://openreview.net/forum?id=QuDDXJ47nq. [41] T. Kuribayashi, Y. Oseki, S. B. Taieb, K. Inui, T. Baldwin, Large language models are human-like internally, 2025. URL: https://arxiv.org/abs/2502.01615. arXiv:2502.01615. [42] T. Chen, C. Guestrin, Xgboost: A scalable tree boosting system, in: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, ACM, 2016, p. 785–794. URL: http://dx.doi.org/10.1145/2939672.2939785. doi:10.1145/2939672.2939785. [43] L. Breiman, Random forests, Machine Learning 45 (2001) 5–32. URL: https://doi.org/10.1023/A: 1010933404324. doi:10.1023/A:1010933404324. [44] Y. Li, Q. Li, L. Cui, W. Bi, Z. Wang, L. Wang, L. Yang, S. Shi, Y. Zhang, Mage: Machine-generated text detection in the wild, 2024. URL: https://arxiv.org/abs/2305.13242. arXiv:2305.13242. [45] X. Zhang, J. Zhao, Y. LeCun, Character-level convolutional networks for text classification, in: C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, R. Garnett (Eds.), Advances in Neural Information Processing Systems, volume 28, Curran Associates, Inc., 2015. URL: https://proceedings.neurips. cc/paper_files/paper/2015/file/250cf8b51c773f3f8dc8b4be867a9a02-Paper.pdf. [46] S. Narayan, S. B. Cohen, M. Lapata, Don‘t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization, in: E. Rilof, D. Chiang, J. Hockenmaier, J. Tsujii (Eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language
While static surprisal statistics such as mean, variance, skewness, and kurtosis provide useful summaries of token-level unpredictability, they overlook the evolution of this unpredictability over time, a dimension critical to distinguishing human and AI-generated text. Human authors naturally embed stylistic variability through temporal fluctuations, such as abrupt topic shifts, tonal changes, and bursts of creativity, which manifest as distinctive temporal dynamics in surprisal sequences. Intuitively, these temporal features, as listed in Section 3, expose rhythmic and non-stationary patterns characteristic of human creativity and coherence, typically absent in the more uniform output of large language models.
Furthermore, through an ablation study on Testbed 4 of the MAGE benchmark, we empirically show that augmenting static surprisal features with temporal metrics leads to a measurable improvement in classification accuracy. This highlights the complementary value of temporal dynamics in enhancing the robustness of AI-generated text detection. Moreover, an analysis of feature importance reveals that temporal features collectively contribute more than static features, consistently ranking among the most informative signals for distinguishing between human and AI-generated text. Both these experiments are thoroughly detailed in our original paper.
Overall, these findings motivate the inclusion of temporal surprisal features as integral components of our DivEye framework.
is human-written (ℓ = 1) or machine-generated (ℓ = 0) Require: Pretrained auto-regressive language model (e.g., GPT-2) Require: XGBoost classifier with hyperparameters Θ (Appendix C) Algorithm 1 DivEye: Algorithm for Feature Extraction & Training Require: Text dataset = {(, ℓ)}=1, where is a text input and ℓ ∈ {0, 1} indicates whether it Ensure: Trained binary classifier
for each (, ℓ) ∈ do Initialize an empty feature matrix ℱ ← Compute token-level log-likelihoods: ← Convert to token-level surprisals: ← −
() Append (DivEye(), ℓ) to ℱ
Compute diversity features DivEye() ∈ R9 as described in Equation (6) using end for return
Train binary classifier on feature set ℱ using XGBoost with hyperparameters Θ
For further analysis, we refer readers to our main paper [40], which includes: (1) domain-specific performance of DivEye, (2) model-specific performance of DivEye, (3) the relative importance of DivEye when used in a boosted ensemble, (4) DivEye’s efectiveness across diferent base detectors, and (5) a breakdown of feature importance within DivEye. These additional evaluations further support the generality, complementarity, and interpretability of our approach.
classifier with standard but tuned settings to handle class imbalance and optimize detection performance. For our proposed method DivEye, we set the number of bins for entropy computation to 20 and truncate input sequences at a maximum length of 1024 tokens. All experiments were run on a two NVIDIA RTX 4060Ti (16 GB each), and reported results reflect the median of three runs.
XGBoost Hyperparameter Value 42 (len(train) − ∑︀ train)/ ∑︀ train 12 200 0.8 0.7 5 1.0 Value 20 1024 + Truncation