With the growing sophistication of generative language models, distinguishing between human- and AI-generated text has become increasingly complex. This paper presents a hybrid DistilBERT-based approach evaluated on the PAN 2025 Voight-Kampf authorship verification task. The method demonstrates robust performance across multiple genres and employs standard evaluation metrics such as ROC-AUC, Brier Score, F1, and C@1. Our results highlight both the promise and limitations of current authorship detection systems, paving the way for future work in this domain.
Authorship verification has been extensively studied, traditionally focused on attributing anonymous
texts to known authors [
Recent work in the PAN lab has explored stylometric methods and deep learning approaches for
authorship verification [
The dataset[
Our hybrid system leverages DistilBERT fine tuned on the training dataset with binary cross entropy loss. Feature engineering included entropy measures and stylistic statistics. We used a logistic regression layer on top of DistilBERT embeddings to classify authorship.
We fine tuned DistilBERT using the HuggingFace Transformers library. The model was trained for 4 epochs using a batch size of 16 and a learning rate of 5 × 10− 5 with the AdamW optimizer. Early stopping was used based on validation loss to prevent overfitting.
In addition to the transformer-based embeddings, we extracted shallow stylometric features from the text: • Average word length: Calculated as the total number of characters divided by the number of words. • Average sentence length: Measured as the average number of words per sentence. • Punctuation frequency: Relative frequency (per 100 characters) of specific punctuation marks including periods (.), commas (,), semicolons (;), colons (:), exclamation marks (!), and question marks (?). • Type-token ratio: A measure of vocabulary richness, computed as the number of unique words divided by the total number of words in the text. • Shannon entropy of character distribution: Entropy was computed over character 1-grams (i.e., single-character sequences) to quantify the unpredictability or information density of the text. These features were concatenated with the [CLS] token embedding from DistilBERT and passed through a logistic regression layer.
Following PAN standards [
We observed a ROC-AUC of 0.87, F1 score of 0.84, and a Brier score of 0.15. These indicate strong performance and good calibration. Confusion matrix results suggest that the model was most confused on high quality AI fiction and news.
F0.5u
Mean 0.89 0.87 0.91 0.79 0.81 0.16 0.15 0.14 0.21 0.19 0.84 0.82 0.85 0.76 0.78
F1 0.86 0.84 0.88 0.75 0.78 0.83 0.81 0.86 0.72 0.76
TP
Manual inspection of false positives and false negatives revealed common themes: • False Positives: Human authored news articles with formulaic structure were occasionally misclassified as AI generated. • False Negatives: AI generated fiction that incorporated creative punctuation and varied sentence structure often fooled the model.
Excerpt “The central bank announced a hike in interest rates, citing inflation concerns...” “Her eyes flickered two glimmers of silver in the fog of an endless, coded dream...” Our method captures subtle linguistic anomalies indicative of AI authorship. However, genre specific overlaps and high quality machine generated news texts challenge detection. Future improvements may involve: • Integrating adversarial examples during training • Using genre aware embeddings • Ensembling outputs from multiple transformer variants
Our work aligns with ongoing debates around AI content regulation. As governments and platforms
grapple with the detection and labeling of AI generated content [
While transformer models provide strong performance, their lack of transparency remains a concern. Our hybrid approach, which supplements deep embeddings with interpretable features, provides a step toward more explainable detection systems that can ofer justifications for their predictions.
This paper presented a DistilBERT based solution to the 2025 PAN Voight-Kampf authorship verification task. Our hybrid system, which combines deep contextual representations with interpretable stylistic features, demonstrated competitive performance across genres and evaluation metrics. With a ROCAUC of 0.87 and an F1 score of 0.84, our approach shows that even lightweight transformers can be efectively adapted for AI authorship detection under constrained settings.
Beyond quantitative metrics, the system also revealed valuable qualitative insights highlighting, for instance, that human like AI generated fiction poses the greatest detection challenge. These results underline the urgent need for robust, generalizable verification models as generative models continue to evolve in quality and diversity.
For future work, we aim to extend the system’s capabilities to multilingual datasets and mixed
authorship scenarios where AI and human text are interwoven. Additionally, integrating instruction
tuned LLM detectors and adversarial training could further improve robustness. Emphasis will also
be placed on explainability developing techniques that not only classify text accurately, but also ofer
human understandable justifications for their decisions, which is vital for adoption in legal, educational,
and journalistic domains [
This work was supported by IIIT Surat. We thank the PAN organizers[
During the preparation of this work, the author(s) used ChatGPT and Perplexity in order to: Grammar and spelling check, Paraphrase and reword. After using these tool(s)/service(s), the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content.
The fine-tuned model used in this work has been made publicly available on Hugging Face: https: //huggingface.co/OjaswaVarshney/PAN-Clef_Updated