AI text. Moreover, detection models trained on specific text types may not perform well on others, requiring extensive retraining and resources. Ethical and practical concerns also arise, such as the risk of false positives and negatives, privacy issues in data analysis, and the ongoing need to adapt to new AI techniques. Addressing these issues involves continuous advancements in detection algorithms and comprehensive research efforts.

This section outlines the classification methods and specific model training approaches, Section 3.2 discusses the model's overall structure and Section 3.3 focuses on the key points of model training. The dataset, acquired via CLEF 2024 PAN [1], consists of about 1,087 rows of text composed by humans and approximately 14,131 rows of text produced by AI. The text comprises a combination of authentic and fraudulent news stories from different 2021 U.S. news headlines. Initially, the dataset contained numerous JSON encodings, which were removed in the first step. During further analysis of the cleaned dataset, NAN values were identified. These were addressed by consolidating all data into a single data frame. Using linguistic analysis, the text column extracted features such as average line length, vocabulary, word density, and POS tags. This provides an overview of the data processing steps, as shown in Figure 1. From this dataset, we extracted feature statistics. Table 1 represents statistics and feature extraction data.

This section examines the linguistic differences between human-written and AI-generated texts. Next, the performance of existing detection algorithms is assessed using the PAN-24 dataset [2]. Finally, the criteria used by deep learning-based detection methods are investigated.

This section examines the vocabulary characteristics of the PAN-24 dataset. The study is focused on the word choices made by AI-generated text and humans when responding to identical queries. Given the diversity of texts written by humans and AI, these differences are analyzed during the statistical procedure. The following traits were computed: in addition to lexicon measure (V), which measures the total number of unique words used in all responses, and average length (L), which measures the average number of words in each text, an additional characteristic named word density (D) is proposed. Word density is determined by the formula D = 100 × V / (L × N), where N is the number of answers. Density quantifies the degree to which words are employed intensively in a text. For instance, if 1,000 words of the text are published but only 100 distinct words are used, the density is 100 × 100 / 1,000 = 10. The higher the density, the more different words are used in the same text length. [3]. Lexical analysis Within the domain of NLP, every word can be categorized into one of several lexical categories. The part-of-speech (POS) tagging task aims to identify each word's grammatical class within a given phrase. In this section, the lexical distributions of various AI-generated and human texts in the PAN-24 dataset are computed using the POS module in NLTK [4]. The data is then arranged according to lexical percentage. As illustrated in Figures 2 and 3, various parts of speech are displayed. Figures 4 and 5 present punctuation and adposition tags, respectively. Finally, Figures 6 and 7 show determiners and pronouns. The statistics for the top ten lexical categories are displayed. Nouns (NOUN) make up the largest proportion of all lexical categories, while punctuation (PUNCT), verbs (VERB), adpositions (ADP), adjectives (ADJ), and determiners (DET) constitute most of the remaining categories.

When comparing human-written texts to AI-generated texts, the following observations can be made: AI-generated texts have higher proportions of nouns (NOUN), verbs (VERB), determiners (DET), adjectives (ADJ), auxiliaries (AUX), coordinating conjunctions (CCONJ), and particles (PART) than human-written texts. This suggests that the rich knowledge embedded in AI-generated texts offers a more varied vocabulary, enhancing their informativeness.

Human-written texts contain higher proportions of adverbs (ADV) and punctuation (PUNCT) than AI-generated texts. This indicates that humans prioritize structure, consistency, and logical flow, in which AI-generated texts are comparatively weaker.

A BERT-based sequence classification [5] and transformer-based model designed to understand the context of a word in search queries. Unlike traditional models that process text sequentially (either leftto-right or right-to-left), BERT considers the entire sequence of words simultaneously. This bidirectional approach allows BERT to grasp the context of a word based on its surrounding words, leading to better performance in NLP tasks. Key Features of BERT:

• Bidirectional Training: BERT uses a Transformer architecture that reads text bi-directionally.

This helps the model understand the context of each word more comprehensively. • Pre-training and Fine-tuning: BERT involves two main stages:

-Pre-training: The model is trained on a large corpus of text, learning to predict missing words in sentences (Masked Language Model) and the next sentence (Next Sentence Prediction). -Fine-tuning: The pre-trained BERT model is then fine-tuned on tasks such as text classification, named entity recognition, or question answering using task-specific data.

Our team plans to extract features from the original dataset, including the text and numerical columns. Initially, this dataset was utilized as training data for 3 epochs to train a new model, referred to as model A, using BERT. BERT, an enhanced version of previous models, incorporates a more significant number of parameters, more extensive training data, and larger batch sizes. It is trained significantly more significantly than CNN-BILSTM, which takes considerably longer. This extensive training allows BERT representations to generalize more effectively to downstream tasks and deliver superior performance compared to other models. As a result, the BERT model demonstrates high accuracy and faster processing speeds, as illustrated in Figure . 8. • __init__: Initializes the dataset with text, numerical data, and labels, converting numerical data and labels to tensors. • __len__: Returns the length of the dataset.

• __getitem__: Tokenizes text data, processes numerical features, and returns a dictionary with input IDs, attention mask, and label for a given index.

We have written software that can be run from the command line. An input file (an absolute path to the input JSONL file) and an output directory (an absolute path to the location where the results will be written) are the two arguments that the script requires. We execute the command as follows in the terminal:

Here, model.py is the main Python file that loads and runs the model. The <input_file_path> is the path of the file containing the input texts, and the <output_directory> is the directory where the output file is saved.

The precise adjustments a user makes to control the learning process are known as hyperparameters.

The best/optimal hyperparameters for learning algorithms must be selected during training to yield the most meaningful results. The hyperparameters used in our recommended techniques are shown in Table 2 we selected these values by analyzing the performance of the suggested methods with different combinations of hyperparameters.

Feature extraction in NLP involves transforming raw text data into a structured representation that machine learning algorithms can use for various NLP tasks. The following features were extracted, and our model was trained on those parameters.

In NLP, average line length is the mean number of characters or words per line in a text dataset like PAN-24. A sample text has been taken from this dataset. For example,

• Text: "President Joseph R. Biden Jr. calls for unity and a renewed commitment to democracy".

• Average characters per line: 74 • Average words per line: 12

In NLP, vocabulary (vocab) refers to the set of unique words or tokens in a text dataset like PAN-24. A sample text has been taken from this dataset. For example,

• Text: "Biden's inauguration is impacted by the pandemic and security threats. "

• Vocabulary: "Biden's," "inauguration," "is," "impacted," "by," "the," "pandemic," "and," "security," "threats" • Size of vocabulary: 10

In NLP, word density measures how many unique words (vocabulary) appear per unit of text, calculated as 100 times the vocabulary size divided by the product of the number of lines and the average line length.

A sample text has been taken from this dataset. For example, "A new chapter of American democracy begins amidst unprecedented times. "

Step-by-Step Calculation:

• Vocabulary:

-Unique words: "A," "new," "chapter," "of," "American," "democracy," "begins," "amidst," "unprecedented, " "times" -Vocabulary size: 10

-There is 1 line in the text.

-Line 1: "A new chapter of American democracy begins amidst unprecedented times.

Part-of-speech (POS) tags are labels assigned to each word in a text to indicate its grammatical category, such as noun, verb, adjective, etc. POS tagging is a fundamental task in NLP that helps understand sentences' syntactic structure and meaning. Explanation of POS Tags: -Examples: "quickly, " "very, " "well. " -Usage: "He ran quickly. " • Particles -Definition: Particles are small words with grammatical functions that do not fit into other categories. -Examples: "to" (in "to go"), "not" (in "do not") -Usage: "She decided to go. " • Subordinating conjunctions -Definition: Subordinating conjunctions connect clauses to show a relationship between them. -Examples: "because, " "although, " "if" -Usage: "She stayed home because it was raining. " • Numerals -Definition: Numerals are words that represent numbers.

-Examples: "one, " "two, " "third. " -Usage: "She has two cats. " • X -Definition: Other categories of words that do not fit into the standard parts of speech.

-Examples: Foreign words, typos -Usage: "She said 'ciao' as she left. "

There are three major steps of our implementation as follows:

• Tokenization and Model Loading: This part sets up the tokenizer and the model into 19 distinct features as shown in Table 1. From among these features, only suitable features, The tokenizer and model configuration, are loaded from the 'bert-base-uncased' pre-trained model, and the actual model weights are loaded from a specified path. The model is set to evaluation mode and moved to the appropriate device (CPU or GPU). • TextDetector Class: This class takes a text string as input tokenizes it, and then uses the model to get the logits (Logits are a neural network model's raw, unnormalized outputs). The logits are converted to probabilities using the softmax function. It assumes a binary classification model and returns the second class's probability (index 1). • Comparative Score Function:

This function compares two scores with a small threshold (epsilon) to avoid floating-point precision issues. It returns a value between 0 and 1 based on the comparison:

-Returns a value between 0.5 and 1 if the first score is significantly higher.

-Returns a value between 0 and 0.5 if the second score is significantly higher. -Returns 0.5 if the scores are very close (within epsilon).

In the final function of calculating the result, it reads the line and parses it as JSON, then extracts the two texts (text1 and text2) and computes scores for both texts. It uses a comparative score function to determine a final score. Finally, the results are written in a JSONL file in the specified output directory.

Overview of the mean accuracy over 9 variants of the test set. We report the minimum, median, maximum, the 25th, and the 75th quantile, of the mean per the 9 datasets. direct-velocity is the name of the LLM model implemented in this paper. The submission scores 8th out of 30 on the PAN CLEF generated content analysis leaderboard.

Systems are assessed using the PAN authorship verification tasks as a benchmark. The metrics listed below are employed:

The region that falls within the receiver operating characteristic (ROC) curve. Characteristics of the Receiver Operating Area An indicator of the actual positive rate against the false positive rate at different threshold settings is the area under the receiver operating characteristic curve, or "Under the Curve. " Higher numbers indicate better discrimination performance. It offers a total assessment of a model's capacity to distinguish between positive and negative classes.

The Brier score's complement (mean squared loss). For binary classification problems, the Brier score calculates the mean squared difference between the expected probability and the actual result (0 or 1). Lower Brier ratings indicate better calibration and accuracy of the probability predicted by the model.

A modified accuracy score that uses the average accuracy of the remaining instances to assign nonanswers (score = 0.5). C@1 quantifies the percentage of cases in which the model's top-ranked prediction corresponds with the ground truth label. It is a typical assessment metric for recommendation or information retrieval systems.

The harmonic mean between recall and accuracy. The F1 score is calculated by taking the harmonic mean of the two variables: recall, the ratio of accurate optimistic predictions to all real positives, and precision, which is the ratio of accurate optimistic predictions to all projected positives. Better performance is indicated by higher numbers, which strike a balance between recall and precision.

A precision-weighted F measure (modified F0.5 measure) that considers non-answers (score = 0.5) to be false negatives. While recall is less important than precision, the F0.5 score is comparable to the F1 score. When recall is less important than precision, like in situations where erroneous positives are more expensive than false negatives, it might be helpful.

The sum of all of the following measurements. The mean score indicates the average performance across all samples or occurrences in the evaluation dataset. These metrics collectively provide insights into different aspects of model performance, including discrimination ability, calibration, accuracy, ranking quality, and the balance between precision and recall.

Table 3 shows the results, initially pre-filled with the official baselines provided by the PAN organizers.

Baseline models are simple reference models used to establish a benchmark for evaluating the performance of more complex models in machine learning and natural language processing tasks. These models provide a standard or point of comparison, allowing researchers and practitioners to assess whether new models offer improvements in accuracy, efficiency, or other relevant metrics. By comparing against baseline models, it is possible to quantify the gains achieved by novel techniques and ensure that the advancements are meaningful and not merely coincidental. Six LLM detection baselines are used as references for the model results. These six LLM detection baselines are re-implementations from the original papers: • Baseline Binoculars [6] • Baseline DetectGPT [7] • Baseline PPMd [8] • Baseline Unmasking [9] • Baseline Fast-DetectGPT [10]

This paper presents a bootstrap dataset of actual and false news items encompassing multiple 2021 U.S. news headlines, using the shared task on the PAN-24 dataset, which includes almost 1087 rows of human-written text. And different in almost 13 LLMs with 14181 rows( ai-generated text). Based on the PAN-24 dataset, we conduct broad considers counting human-written content assessments, phonetic investigation, and ai-generated content discovery tests. The human-written content assessments and phonetics analysis provide us with knowledge about the specific contrasts between human-written content and AI-generated text, which persuade our considerations of LLMs' future headings. The ai-generated content substance detection experiments outline a few imperative conclusions that can give advantageous guides to the research and improvement of AIGC-detection instruments. We make all our data, code, and models publicly available to facilitate related research and applications at our git hub repository AI vs Human