=Paper=
{{Paper
|id=Vol-3740/paper-290
|storemode=property
|title=Team cnlp-nits-pp at PAN: Leveraging BERT for Accurate Authorship Verification: A
Novel Approach to Textual Attribution
|pdfUrl=https://ceur-ws.org/Vol-3740/paper-290.pdf
|volume=Vol-3740
|authors=Annepaka Yadagiri,Dimpal Kalita,Abhishek Ranjan,Ashish Kumar Bostan,Parthib Toppo,Partha Pakray
|dblpUrl=https://dblp.org/rec/conf/clef/YadagiriKRBTP24
}}
==Team cnlp-nits-pp at PAN: Leveraging BERT for Accurate Authorship Verification: A
Novel Approach to Textual Attribution==
https://ceur-ws.org/Vol-3740/paper-290.pdf
Team cnlp-nits-pp at PAN: Leveraging BERT for Accurate
Authorship Verification: A Novel Approach to Textual
Attribution
Notebook for the PAN Lab at CLEF 2024
Annepaka Yadagiri1,* , Dimpal Kalita1 , Abhishek Ranjan1 , Ashish Kumar Bostan1 ,
Parthib Toppo1 and Partha Pakray1
1
Computer Science & Engineering, National Institute of Technology, Silchar, Assam, India
Abstract
The launch of ai-generated tools has attracted a lot of interest from the academic and business worlds. Effectively
handling a broad spectrum of human inquiries, ai-generated tools offer clear, thorough responses that far
outperform earlier open-source chatbots regarding security and use. People are interested in learning how
powerful AI is and how far it has come from human specialists. However, concerns about the possible detrimental
effects large language models like ChatGPT can have on society—including fake news, plagiarism, and social
security problems—are beginning to surface. In this work, The dataset is provided from CLEF PAN-24 human-
written text data and 13 different types of ai-generated models text data like alpaca-7b,bigscience-bloomz-7b1,
chavinlo-alpaca-13b, Gemini-pro, gpt-3.5-turbo-0125,gpt-4-turbo-preview,meta-llama-llama-2-7b-chat-hf,meta-
llama-llama-2-70b-chat-hf,mistralai-mistral-7b-instruct-v0.2,mistralai-mixtral-8x7b-instruct-v0.1,qwen-
qwen1.5-72b-chat-8bit,text-bison-002,vicgalle-gpt2-open-instruct-v1. which approximately provides imbalanced
data. The comparison of human-written and ai-generated data. We examine the features of ChatGPT’s replies,
the distinctions and shortcomings of human experts, and the prospects for LLMs based on the pan-24 dataset.
We conducted extensive human assessments and linguistic examinations of ai-generated content compared to
human content, yielding several intriguing findings. Then, we conduct in-depth research on the best ways to
identify whether a given text was produced by ai-generated or humans. We construct three distinct detection
systems, investigate critical variables affecting their performance, and test them in various contexts. Our
solution approach for this task involves using the BERT model with a preprocessing model, where we achieved
classification results with over 97.6% ROC-AUC for all the results included in this challenge.
Keywords
Large Language Models,, AI-Generated Content Detection, Natural Language Processing, Generative AI, BERT
1. Task
In cooperation with the Voight-Kampff Task at the ELOQUENT Lab, the Generative AI Authorship
Verification Task at PAN uses a builder-breaker approach. ELOQUENT participants research innovative
text creation and obfuscation techniques to evade detection, while PAN participants develop systems to
distinguish between human and AI-generated content [1].
Detecting whether text is human or AI-generated is challenging due to several factors. First, AI-
generated text from LLMs like GPT-4 is often highly coherent and contextually appropriate, making
it difficult to distinguish from human writing. Additionally, LLMs can mimic human writing styles
and nuances, further complicating detection. Statistical methods used to differentiate text, such as
analyzing word frequency and sentence structure, often find significant overlap between human and
CLEF 2024: Conference and Labs of the Evaluation Forum, September 09–12, 2024, Grenoble, France
⋆
You can use this document as the template for preparing your publication. We recommend using the latest version of the
ceurart style.
*
Corresponding author.
$ annepaka22rs@cse.nits.ac.in (A. Yadagiri); kalitadimpal112@gmail.com (D. Kalita); abhiran14082002@gmail.com
(A. Ranjan); kumarashishbostan@gmail.com (A. K. Bostan); toppoparthib@gmail.com (P. Toppo); partha@cse.nits.ac.in
(P. Pakray)
0000-0003-1326-1448 (A. Yadagiri); 0009-0008-0054-7459 (D. Kalita); 0009-0000-3721-430X (A. Ranjan);
0009-0007-0286-5162 (A. K. Bostan); 0009-0004-9756-1951 (P. Toppo); 0000-0003-3834-5154 (P. Pakray)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
AI text. Moreover, detection models trained on specific text types may not perform well on others,
requiring extensive retraining and resources. Ethical and practical concerns also arise, such as the risk
of false positives and negatives, privacy issues in data analysis, and the ongoing need to adapt to new
AI techniques. Addressing these issues involves continuous advancements in detection algorithms and
comprehensive research efforts.
2. Dataset Description
This section outlines the classification methods and specific model training approaches, Section 3.2
discusses the model’s overall structure and Section 3.3 focuses on the key points of model training.
The dataset, acquired via CLEF 2024 PAN [1], consists of about 1,087 rows of text composed by humans
and approximately 14,131 rows of text produced by AI. The text comprises a combination of authentic
and fraudulent news stories from different 2021 U.S. news headlines. Initially, the dataset contained
numerous JSON encodings, which were removed in the first step. During further analysis of the cleaned
dataset, NAN values were identified. These were addressed by consolidating all data into a single
data frame. Using linguistic analysis, the text column extracted features such as average line length,
vocabulary, word density, and POS tags. This provides an overview of the data processing steps, as
shown in Figure 1. From this dataset, we extracted feature statistics. Table 1 represents statistics and
feature extraction data.
Figure 1: Data Processing steps
Table 1
The following table represents pan-24 dataset statistics and feature extraction data.
Model Dataset size Word density Vocab Avg line length NOUN PUNC VERB ADP DET PRON ADJ AUX ADV CCONJ PROPN PART SCONJ NUM X INTJ
human 1087 4.56 29820 27.39 17.17 10.61 14.0 10.25 8.6 3.94 5.77 0.81 3.23 2.59 10.64 0.29 10.25 1.72 0.012 0.013
alpaca-7b 1087 4.79 14409 30.82 16.38 10.52 9.04 9.88 8.63 1.98 5.37 0.76 1.83 2.99 15.99 0.12 9.88 1.4 0.0045 0.0065
bigscience-bloomz-7b1 1087 3.32 13938 24.18 17.66 8.72 10.54 11.06 9.41 2.7 5.36 0.7 2.03 2.35 15.48 0.19 11.06 2.1 0.005 0.0011
chav-inlo-alpaca-13b 1087 5.60 14476 28.82 16.37 9.07 10.17 9.97 8.43 1.95 5.92 1 2.06 3.15 20.28 0.1 9.97 1.48 0.0009 0.0009
gemini-pro 1087 4.15 24347 22.33 17.97 9.41 11.17 9.37 8.37 1.95 5.92 0.71 2.05 2.77 11.67 0.12 9.37 1.85 0.001 0.0013
gpt-3.5-turbo-0125 1087 4.57 22776 28.92 20.51 8.61 12.48 11.37 9.51 2.36 6.83 0.46 2.14 2.58 10.61 0.13 11.37 0.95 0.001 0.0011
gpt-4-turbo-preview 1087 3.99 26175 28.26 20.56 9.94 12.02 10.35 9.56 2.81 6.91 0.47 2.26 2.6 11.37 0.17 10.35 0.9 0.0014 0.0013
metallama 2-7b 1087 3.88 21431 26.58 19.07 10.28 10.88 8.99 9.99 3.66 5.98 0.74 2.21 3.14 9.13 0.12 8.99 1.4 0.003 0.0013
metallama-270b 1087 3.52 22422 25.16 18.52 9.98 12.3 9.39 8.98 3.6 5.91 0.62 2.26 3.04 11.56 0.12 9.39 1.15 0.0064 0.0032
mistralai-mistral-7b 1087 3.62 25147 24.67 18.3 11.15 9.55 9.76 8.43 3.2 5.62 0.67 2.26 3.04 11.56 0.12 9.76 1.24 0.0064 0.0032
mistralai-mixtral-8x7b 1087 3.92 26549 26.28 19.14 10.98 10.02 9.35 9.56 3.67 5.62 0.62 2.22 3.06 11.56 0.12 9.35 1.24 0.006 0.0024
qwen-qwen1.5-72b 1087 4.38 32658 27.84 18.19 10.56 10.6 9.7 9.36 3.16 5.35 0.59 2.1 3.05 11.44 0.1 9.7 1.51 0.003 0.0022
text-bison-002 1087 3.98 25960 26.01 19.16 10.56 12.83 9.22 9.19 3.66 6.65 0.59 2.23 2.88 11.44 0.1 9.22 1.01 0.0016 0.0016
vicugalle-gpt2-open-instruct-v1 1087 2.53 16920 30.03 17.68 9.41 12.8 10.45 9.35 3.16 5.9 0.69 2.02 3 13.32 0.13 10.45 1.56 0.0045 0.0047
3. System Overview
This section examines the linguistic differences between human-written and AI-generated texts. Next,
the performance of existing detection algorithms is assessed using the PAN-24 dataset [2]. Finally, the
criteria used by deep learning-based detection methods are investigated.
3.1. Vocabulary Features
This section examines the vocabulary characteristics of the PAN-24 dataset. The study is focused on
the word choices made by AI-generated text and humans when responding to identical queries. Given
the diversity of texts written by humans and AI, these differences are analyzed during the statistical
procedure. The following traits were computed: in addition to lexicon measure (V), which measures
the total number of unique words used in all responses, and average length (L), which measures the
average number of words in each text, an additional characteristic named word density (D) is proposed.
Word density is determined by the formula D = 100 × V / (L × N), where N is the number of answers.
Density quantifies the degree to which words are employed intensively in a text. For instance, if 1,000
words of the text are published but only 100 distinct words are used, the density is 100 × 100 / 1,000 =
10. The higher the density, the more different words are used in the same text length. [3].
Lexical analysis Within the domain of NLP, every word can be categorized into one of several
lexical categories. The part-of-speech (POS) tagging task aims to identify each word’s grammatical
class within a given phrase. In this section, the lexical distributions of various AI-generated and human
texts in the PAN-24 dataset are computed using the POS module in NLTK [4]. The data is then arranged
according to lexical percentage. As illustrated in Figures 2 and 3, various parts of speech are displayed.
Figures 4 and 5 present punctuation and adposition tags, respectively. Finally, Figures 6 and 7 show
determiners and pronouns. The statistics for the top ten lexical categories are displayed. Nouns (NOUN)
make up the largest proportion of all lexical categories, while punctuation (PUNCT), verbs (VERB),
adpositions (ADP), adjectives (ADJ), and determiners (DET) constitute most of the remaining categories.
When comparing human-written texts to AI-generated texts, the following observations can be made:
AI-generated texts have higher proportions of nouns (NOUN), verbs (VERB), determiners (DET),
adjectives (ADJ), auxiliaries (AUX), coordinating conjunctions (CCONJ), and particles (PART) than
human-written texts. This suggests that the rich knowledge embedded in AI-generated texts offers a
more varied vocabulary, enhancing their informativeness.
Human-written texts contain higher proportions of adverbs (ADV) and punctuation (PUNCT) than
AI-generated texts. This indicates that humans prioritize structure, consistency, and logical flow, in
which AI-generated texts are comparatively weaker.
3.2. Model
A BERT-based sequence classification [5] and transformer-based model designed to understand the
context of a word in search queries. Unlike traditional models that process text sequentially (either left-
to-right or right-to-left), BERT considers the entire sequence of words simultaneously. This bidirectional
approach allows BERT to grasp the context of a word based on its surrounding words, leading to better
performance in NLP tasks.
Key Features of BERT:
• Bidirectional Training: BERT uses a Transformer architecture that reads text bi-directionally.
This helps the model understand the context of each word more comprehensively.
• Pre-training and Fine-tuning: BERT involves two main stages:
– Pre-training: The model is trained on a large corpus of text, learning to predict miss-
ing words in sentences (Masked Language Model) and the next sentence (Next Sentence
Prediction).
– Fine-tuning: The pre-trained BERT model is then fine-tuned on tasks such as text classifi-
cation, named entity recognition, or question answering using task-specific data.
Our team plans to extract features from the original dataset, including the text and numerical columns.
Initially, this dataset was utilized as training data for 3 epochs to train a new model, referred to as model
A, using BERT. BERT, an enhanced version of previous models, incorporates a more significant number
of parameters, more extensive training data, and larger batch sizes. It is trained significantly more
significantly than CNN-BILSTM, which takes considerably longer. This extensive training allows BERT
representations to generalize more effectively to downstream tasks and deliver superior performance
compared to other models. As a result, the BERT model demonstrates high accuracy and faster processing
speeds, as illustrated in Figure. 8.
Figure 2: Pos tag Noun Figure 3: Pos tag Verb
Figure 4: Pos tag Punctuation Figure 5: Pos tag Adposition
Figure 6: Pos Tag Determiner Figure 7: Pos tag Pronoun
3.3. Model Training
First, with Intel(R) Xeon(R) Gold 6348 CPU @ 2.60GHz and NVIDIA A800-SXM4-80GB hardware
platform, we will split the dataset into 80% training and 20% validation sets. Each training iteration will
utilize a batch size of 32. The data consists of a text column along with numerical columns ([‘Vocabulary’,
‘Noun Count’, ‘Verb Count’, ‘AUX Count’, ‘NUM Count’, ‘PRON Count’, ‘ADV Count’, ‘INTJ Count’,
‘PART Count’]). This dataset will be input into the BERT model for sequence classification, incorporating
the numerical features using PyTorch and Hugging Face Transformers.
The CustomDataset class inherits from torch.utils.data.Dataset.
• __init__: Initializes the dataset with text, numerical data, and labels, converting numerical data
and labels to tensors.
• __len__: Returns the length of the dataset.
• __getitem__: Tokenizes text data, processes numerical features, and returns a dictionary with
input IDs, attention mask, and label for a given index.
then Load a pre-trained BERT tokenizer and model from Hugging Face’s model hub. Create instances
of the CustomDataset class for the training and validation sets. Create data loaders for the training
Figure 8: overall prediction process
and validation datasets with a batch size 32. AdamW: Optimizer with weight decay. CrossEntropyLoss:
Loss function for classification tasks. Then, given to the model. tain() After 3 epochs of training, Model
A is generated. Model A’s training round takes about 20 minutes, while prediction time takes about 15
minutes. Then, based on the output results of Model A, our team has established the following criteria
to evaluate the classification between the text and labels: In the data processing step, we are going to
call the label data universally human ‘0’ and ai-generated ‘1’. Based on the label data, we will predict
which was a human-written text and which was ai-generated text. After training the model, we are
going to check after training. We will take 20 percent of the data for testing purposes, whether it is
predicted wrong or correct results. It will give good accuracy to the exact results that came. The model
training process is shown in the figure 9.
3.3.1. Execution Steps
We have written software that can be run from the command line. An input file (an absolute path to the
input JSONL file) and an output directory (an absolute path to the location where the results will be
written) are the two arguments that the script requires.
We execute the command as follows in the terminal:
python3 model.py
Here, model.py is the main Python file that loads and runs the model. The is
the path of the file containing the input texts, and the is the directory where
the output file is saved.
Figure 9: Model Training Process
3.4. Hyperparameters
The precise adjustments a user makes to control the learning process are known as hyperparameters.
The best/optimal hyperparameters for learning algorithms must be selected during training to yield
the most meaningful results. The hyperparameters used in our recommended techniques are shown in
Table 2 we selected these values by analyzing the performance of the suggested methods with different
combinations of hyperparameters.
Table 2
Hyperparameters that were applied to every experiment
Parameter Value
Activation Function Sigmoid
Optimizer AdamW
Loss Function nn_crossentropy
Learning Rate 2 × 10−5
Batch Size 32
Number of Epochs 3
Dropout 0.2
ModelCheckpoint Yes
EarlyStopping Yes
Patience 5
3.5. Features Extracted
Feature extraction in NLP involves transforming raw text data into a structured representation that
machine learning algorithms can use for various NLP tasks. The following features were extracted, and
our model was trained on those parameters.
3.5.1. Average Line Length
In NLP, average line length is the mean number of characters or words per line in a text dataset like
PAN-24. A sample text has been taken from this dataset. For example,
• Text: “President Joseph R. Biden Jr. calls for unity and a renewed commitment to democracy".
• Average characters per line: 74
• Average words per line: 12
3.5.2. Vocabulary
In NLP, vocabulary (vocab) refers to the set of unique words or tokens in a text dataset like PAN-24. A
sample text has been taken from this dataset. For example,
• Text: “Biden’s inauguration is impacted by the pandemic and security threats."
• Vocabulary: "Biden’s," “inauguration," “is," “impacted," “by," “the," “pandemic," “and," “security,"
“threats"
• Size of vocabulary: 10
3.5.3. Word Density
In NLP, word density measures how many unique words (vocabulary) appear per unit of text, calculated
as 100 times the vocabulary size divided by the product of the number of lines and the average line
length.
A sample text has been taken from this dataset. For example, “A new chapter of American democracy
begins amidst unprecedented times."
Step-by-Step Calculation:
• Vocabulary:
– Unique words: "A," "new," "chapter," "of," "American," "democracy," "begins," "amidst," "un-
precedented," "times"
– Vocabulary size: 10
• Number of lines:
– There is 1 line in the text.
• Average line length:
– Line 1: "A new chapter of American democracy begins amidst unprecedented times." (70
characters)
– Average line length: 70 characters
• Word Density Calculation: The word density (𝑊 𝐷) can be calculated using the formula:
100 × Vocabulary Size
𝑊𝐷 = (1)
No of Lines × Average Line Length
Where:
𝑊 𝐷 : Word Density
Vocabulary Size : Number of unique words in the text
No of Lines : Total number of lines in the text
Average Line Length : Average number of characters per line
So, the word density of the text is approximately 14.29.
3.5.4. POS Tags
Part-of-speech (POS) tags are labels assigned to each word in a text to indicate its grammatical category,
such as noun, verb, adjective, etc. POS tagging is a fundamental task in NLP that helps understand
sentences’ syntactic structure and meaning. Explanation of POS Tags:
• Noun
– Definition: Words representing people, places, things, or ideas.
– Examples: “cat," “city," “happiness."
– Usage: “The cat is sleeping."
• Verb
– Definition: Words that describe actions, states, or occurrences.
– Examples: “run," “is," “seem."
– Usage: "She runs every morning."
• Punctuation
– Definition: Symbols used to separate sentences and their elements and to clarify meaning.
– Examples: ".", ",", "!"
– Usage: "Hello, world!"
• Determiner
– Definition: Determiners are words placed before nouns to specify quantity or definiteness.
– Examples: "the," "a," "some."
– Usage: "The apple is red."
• Pronoun
– Definition: Pronouns are words that replace nouns.
– Examples: "he," "they," "it."
– Usage: "She loves her dog."
• Proper Noun
– Definition: Proper nouns are specific names of people, places, or organizations.
– Examples: "John," "Paris," "Google."
– Usage: "Google is a search engine."
• Adjective
– Definition: Adjectives are words that describe or modify nouns.
– Examples: "happy," "blue," "tall."
– Usage: "The tall building is new."
• Auxiliary Verb
– Definition: Auxiliary verbs are used with main verbs to express tense, mood, or voice.
– Examples: "is," "have," "will."
– Usage: "She is running."
• Adverb
– Definition: Adverbs modify verbs, adjectives, or other adverbs.
– Examples: "quickly," "very," "well."
– Usage: "He ran quickly."
• Particles
– Definition: Particles are small words with grammatical functions that do not fit into other
categories.
– Examples: "to" (in "to go"), "not" (in "do not")
– Usage: "She decided to go."
• Subordinating conjunctions
– Definition: Subordinating conjunctions connect clauses to show a relationship between
them.
– Examples: "because," "although," "if"
– Usage: "She stayed home because it was raining."
• Numerals
– Definition: Numerals are words that represent numbers.
– Examples: "one," "two," "third."
– Usage: "She has two cats."
• X
– Definition: Other categories of words that do not fit into the standard parts of speech.
– Examples: Foreign words, typos
– Usage: "She said ’ciao’ as she left."
3.6. Implementation
There are three major steps of our implementation as follows:
• Tokenization and Model Loading: This part sets up the tokenizer and the model into 19 distinct
features as shown in Table 1. From among these features, only suitable features, The tokenizer
and model configuration, are loaded from the ‘bert-base-uncased’ pre-trained model, and the
actual model weights are loaded from a specified path. The model is set to evaluation mode and
moved to the appropriate device (CPU or GPU).
• TextDetector Class: This class takes a text string as input tokenizes it, and then uses the model
to get the logits (Logits are a neural network model’s raw, unnormalized outputs). The logits are
converted to probabilities using the softmax function. It assumes a binary classification model
and returns the second class’s probability (index 1).
• Comparative Score Function:
comparative_score(score1, score2, epsilon=1e-3)
This function compares two scores with a small threshold (epsilon) to avoid floating-point
precision issues. It returns a value between 0 and 1 based on the comparison:
– Returns a value between 0.5 and 1 if the first score is significantly higher.
– Returns a value between 0 and 0.5 if the second score is significantly higher.
– Returns 0.5 if the scores are very close (within epsilon).
In the final function of calculating the result, it reads the line and parses it as JSON, then extracts
the two texts (text1 and text2) and computes scores for both texts. It uses a comparative score
function to determine a final score. Finally, the results are written in a JSONL file in the specified
output directory.
4. Results
Table 3
Overview of the mean accuracy over 9 variants of the test set. We report the minimum, median, maximum, the
25th, and the 75th quantile, of the mean per the 9 datasets. direct-velocity is the name of the LLM model
implemented in this paper. The submission scores 8th out of 30 on the PAN CLEF generated content analysis
leaderboard.
Approach Minimum 25-th Quantile Median 75-th Quantile Max
direct-velocity 0.395 0.905 0.937 0.958 0.978
Baseline Binoculars 0.342 0.818 0.844 0.965 0.996
Baseline Fast-DetectGPT (Mistral) 0.095 0.793 0.842 0.931 0.958
Baseline PPMd 0.270 0.546 0.750 0.770 0.863
Baseline Unmasking 0.250 0.662 0.696 0.697 0.762
Baseline Fast-DetectGPT 0.159 0.579 0.704 0.719 0.982
95-th quantile 0.863 0.971 0.978 0.990 1.000
75-th quantile 0.758 0.865 0.933 0.959 0.991
Median 0.605 0.645 0.875 0.889 0.936
25-th quantile 0.353 0.496 0.658 0.675 0.711
Min 0.015 0.038 0.231 0.244 0.252
4.1. Evaluation Metrics
Systems are assessed using the PAN authorship verification tasks as a benchmark. The metrics listed
below are employed:
4.1.1. ROC-AUC
The region that falls within the receiver operating characteristic (ROC) curve. Characteristics of the
Receiver Operating Area An indicator of the actual positive rate against the false positive rate at different
threshold settings is the area under the receiver operating characteristic curve, or "Under the Curve."
Higher numbers indicate better discrimination performance. It offers a total assessment of a model’s
capacity to distinguish between positive and negative classes.
4.1.2. Brier
The Brier score’s complement (mean squared loss). For binary classification problems, the Brier score
calculates the mean squared difference between the expected probability and the actual result (0 or 1).
Lower Brier ratings indicate better calibration and accuracy of the probability predicted by the model.
4.1.3. C@1
A modified accuracy score that uses the average accuracy of the remaining instances to assign non-
answers (score = 0.5). C@1 quantifies the percentage of cases in which the model’s top-ranked prediction
corresponds with the ground truth label. It is a typical assessment metric for recommendation or
information retrieval systems.
4.1.4. F1
The harmonic mean between recall and accuracy. The F1 score is calculated by taking the harmonic
mean of the two variables: recall, the ratio of accurate optimistic predictions to all real positives,
and precision, which is the ratio of accurate optimistic predictions to all projected positives. Better
performance is indicated by higher numbers, which strike a balance between recall and precision.
4.1.5. F0.5u
A precision-weighted F measure (modified F0.5 measure) that considers non-answers (score = 0.5) to be
false negatives. While recall is less important than precision, the F0.5 score is comparable to the F1
score. When recall is less important than precision, like in situations where erroneous positives are
more expensive than false negatives, it might be helpful.
4.1.6. Mean
The sum of all of the following measurements. The mean score indicates the average performance
across all samples or occurrences in the evaluation dataset.
These metrics collectively provide insights into different aspects of model performance, including
discrimination ability, calibration, accuracy, ranking quality, and the balance between precision and
recall.
Table 3 shows the results, initially pre-filled with the official baselines provided by the PAN organizers.
4.2. Baseline Models
Baseline models are simple reference models used to establish a benchmark for evaluating the perfor-
mance of more complex models in machine learning and natural language processing tasks. These
models provide a standard or point of comparison, allowing researchers and practitioners to assess
whether new models offer improvements in accuracy, efficiency, or other relevant metrics. By comparing
against baseline models, it is possible to quantify the gains achieved by novel techniques and ensure
that the advancements are meaningful and not merely coincidental. Six LLM detection baselines are
used as references for the model results. These six LLM detection baselines are re-implementations
from the original papers:
Table 4
Overview of the accuracy in detecting if a text is written by an human in task 4 on PAN 2024 (Voight-Kampff
Generative AI Authorship Verification). We report ROC-AUC, Brier, C@1, F1 , F0.5𝑢 and their mean.direct-
velocity is the name of the LLM model implemented from this paper. The submission scores 8th out of 30 on
the PAN CLEF generated content analysis leaderboard
Approach ROC-AUC Brier C@1 F1 F0.5𝑢 Mean
direct-velocity 0.976 0.877 0.959 0.934 0.94 0.937
Baseline Binoculars 0.972 0.957 0.966 0.964 0.965 0.965
Baseline Fast-DetectGPT (Mistral) 0.876 0.8 0.886 0.883 0.883 0.866
Baseline PPMd 0.795 0.798 0.754 0.753 0.749 0.77
Baseline Unmasking 0.697 0.774 0.691 0.658 0.666 0.697
Baseline Fast-DetectGPT 0.668 0.776 0.695 0.69 0.691 0.704
95-th quantile 0.994 0.987 0.989 0.989 0.989 0.990
75-th quantile 0.969 0.925 0.950 0.933 0.939 0.941
Median 0.909 0.890 0.887 0.871 0.867 0.889
25-th quantile 0.701 0.768 0.683 0.657 0.670 0.689
Min 0.131 0.265 0.005 0.006 0.007 0.224
• Baseline Binoculars[6]
• Baseline DetectGPT [7]
• Baseline PPMd [8]
• Baseline Unmasking [9]
• Baseline Fast-DetectGPT [10]
5. Conclusion
This paper presents a bootstrap dataset of actual and false news items encompassing multiple 2021
U.S. news headlines, using the shared task on the PAN-24 dataset, which includes almost 1087 rows of
human-written text. And different in almost 13 LLMs with 14181 rows( ai-generated text). Based on the
PAN-24 dataset, we conduct broad considers counting human-written content assessments, phonetic
investigation, and ai-generated content discovery tests. The human-written content assessments and
phonetics analysis provide us with knowledge about the specific contrasts between human-written
content and AI-generated text, which persuade our considerations of LLMs’ future headings. The
ai-generated content substance detection experiments outline a few imperative conclusions that can
give advantageous guides to the research and improvement of AIGC-detection instruments. We make
all our data, code, and models publicly available to facilitate related research and applications at our git
hub repository AI vs Human
Acknowledgments
We express our gratitude to the National Institute of Technology Silchar’s Department of Computer
Science and Engineering and the Center for Natural Language Processing (CNLP) for providing the
necessary infrastructure and assistance for this study.
References
[1] J. Bevendorff, M. Wiegmann, J. Karlgren, L. Dürlich, E. Gogoulou, A. Talman, E. Stamatatos,
M. Potthast, B. Stein, Overview of the “Voight-Kampff” Generative AI Authorship Verification
Task at PAN and ELOQUENT 2024, in: G. Faggioli, N. Ferro, P. Galuščáková, A. G. S. de Herrera
(Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, CEUR
Workshop Proceedings, CEUR-WS.org, 2024.
[2] A. A. Ayele, N. Babakov, J. Bevendorff, X. B. Casals, B. Chulvi, D. Dementieva, A. Elnagar, D. Freitag,
M. Fröbe, D. Korenčić, M. Mayerl, D. Moskovskiy, A. Mukherjee, A. Panchenko, M. Potthast,
F. Rangel, N. Rizwan, P. Rosso, F. Schneider, A. Smirnova, E. Stamatatos, E. Stakovskii, B. Stein,
M. Taulé, D. Ustalov, X. Wang, M. Wiegmann, S. M. Yimam, E. Zangerle, Overview of PAN 2024:
Multi-Author Writing Style Analysis, Multilingual Text Detoxification, Oppositional Thinking
Analysis, and Generative AI Authorship Verification, in: L. Goeuriot, P. Mulhem, G. Quénot,
D. Schwab, L. Soulier, G. M. D. Nunzio, P. Galuščáková, A. G. S. de Herrera, G. Faggioli, N. Ferro
(Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of
the Fifteenth International Conference of the CLEF Association (CLEF 2024), Lecture Notes in
Computer Science, Springer, Berlin Heidelberg New York, 2024.
[3] B. Guo, X. Zhang, Z. Wang, M. Jiang, J. Nie, Y. Ding, J. Yue, Y. Wu, How close is chatgpt to human
experts? comparison corpus, evaluation, and detection, arXiv preprint arXiv:2301.07597 (2023).
[4] S. Bird, Nltk: the natural language toolkit, in: Proceedings of the COLING/ACL 2006 Interactive
Presentation Sessions, 2006, pp. 69–72.
[5] C. A. C. Sáenz, K. Becker, Understanding stance classification of bert models: an attention-based
framework, Knowledge and Information Systems 66 (2024) 419–451.
[6] A. Hans, A. Schwarzschild, V. Cherepanova, H. Kazemi, A. Saha, M. Goldblum, J. Geiping, T. Gold-
stein, Spotting llms with binoculars: Zero-shot detection of machine-generated text, 2024. URL:
https://arxiv.org/abs/2401.12070. arXiv:2401.12070.
[7] E. Mitchell, Y. Lee, A. Khazatsky, C. D. Manning, C. Finn, Detectgpt: Zero-shot machine-
generated text detection using probability curvature, 2023. URL: https://arxiv.org/abs/2301.11305.
arXiv:2301.11305.
[8] D. Sculley, C. Brodley, Compression and machine learning: a new perspective on feature space
vectors, in: Data Compression Conference (DCC’06), 2006, pp. 332–341. doi:10.1109/DCC.2006.
13.
[9] J. Bevendorff, B. Stein, M. Hagen, M. Potthast, Generalizing unmasking for short texts, in:
J. Burstein, C. Doran, T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume
1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota,
2019, pp. 654–659. URL: https://aclanthology.org/N19-1068. doi:10.18653/v1/N19-1068.
[10] G. Bao, Y. Zhao, Z. Teng, L. Yang, Y. Zhang, Fast-detectgpt: Efficient zero-shot detection of machine-
generated text via conditional probability curvature, 2024. URL: https://arxiv.org/abs/2310.05130.
arXiv:2310.05130.