Data plays a crucial role in finetuning. Training the pretrained model using a specific dataset to adapt it to a particular task or domain improves relevance and enhances adaptation of the stylistic features. The study employs a distinctive data-centric approach by collecting and preprocessing several renowned datasets to improve text quality and generate human-like sentences for given test questions in Voight Kampf 2025. The findings indicate that the data preprocessing resulted in fruitful text generations, and future work needs a more data-centric approach.
The study acknowledges the crucial steps of researching the detectors used in the prior shared tasks
(PAN) before developing an LLM that can fool an LLM detector for Voight Kampf 2025 [
The study reports the methodology in the following three sections: (1) Data Collection, (2) Data Preprocessing, and (3) Finetune. Lastly, the findings section includes the preliminary results and shared results evaluated by the Voight Kampf (2025) committee.
Four open-access datasets are collected from sources such as Zenedo and Kaggle. All the datasets are selected to capture human linguistic features and address the misinformation concerns mentioned in the literature.
1. First, as the use of the Bootstrap dataset has been encouraged by the PAN Clef 2024 committee,
we have downloaded it from Zenedo, acquiring the permissions [
Overall, all four datasets, as shown in Table 1, are chosen to capture the human linguistic styles and address misinformation that has been a major concern in prior studies. All the datasets are primarily collected/used for detection tasks. However, as per Voight Kampf 2025, this study focuses on capturing more human linguistic styles that the Voight Kampf LLM evaluator cannot detect.
The final datasets selected in the data collection process are subject to data preprocessing. follows three stages, i.e., data retrieving, text extraction, and classification. However, not all the selected datasets go through all the Data preprocessing steps.
The data retrieval is only attempted on “Politofact_real.csv,” where there are three columns (e.g., id, news URL, title) for which each column is associated with a PolitiFact ID. However, after inspection, there are missing ID for some URLs and titles. To ensure that we collect data from PolitiFact, we removed all the URLs and titles that do not have a PolitiFact ID. That leaves 797 rows with the PolitiFact ID, associated URL, and title. Next, using the URLs and required Python packages, the text (at least 500 words) from retrievable (open-source) articles is retrieved. We were able to retrieve text from 195 articles. To ensure that the finetuning model captures meaningful patterns in text, 195 retrieved articles are further preprocessed to remove unwanted articles that have noise characters, whitespaces normalization, and handling missing values. Therefore, the final cleaned version includes 193 cleaned articles in CSV format. The text from other datasets is plain and needs no additional cleaning.
Now that there is text available, the next stage involves extracting text from all four datasets. Since a couple of files (cleaned politofact and ISOT true) are in the same CSV format, the plain text is simply extracted to an output file. Since LIAR (train.tsv) is labelled on six classes, where true is legitimate statements, only the statements from the true class are extracted and augmented to the output file. Lastly, the text from the Bootstrap dataset (human-written) is extracted and augmented to the output ifle. Overall, the output file contains 24,284 text entries, with approximately 88 percent from ISOT, 6 percent from LIAR, 4.5 percent from Bootstrap, and 0.8 percent from FakeNewsNet. The output file is ifltered out by removing the text entries of less than 100 words, leaving 19,401 entries in the final file. This truncation of the sentences is performed in the bold belief that longer paragraphs will give the ifnetuning model more clues.
Since output files contain raw text with 19,401 entries, the necessity of understanding the genres and style during finetuning is essential. Therefore, a BART Large MNLI model provided by Facebook is utilized for identifying genre and style for each entry. In other words, the Zero Shot Classification across all the entries is enabled, which allows the BART model to generate genre and style even if the model hasn’t explicitly trained. To allow the model to capture diferent styles and genres, various categories such as science fiction, fantasy, romance, narrative style, persuasive style, and so forth, are predefined. Next, to match the Voight Kampf 2024 data sample format, along with genre and style, we extract the keywords (using YAKE extractor) for each entry for better finetuning. The final result is matched with the task requirement and converted to JSON format before finetuning. The final dataset is referred to as finetune dataset. The genres, styles, and keywords from the result is depicted in figures 1-3.
Llava 3 using a Low Rank Adaptation (LoRa) approach is utilized for eficient memory usage. The ifnetuning involves customizing the model to generate nuanced responses based on finetune dataset. The generated responses are expected to be more human-written when evaluated by the Voight Kampf 2025 evaluator. Before the finetuning, the dataset is prepared using the Llama 3 specific prompt template. Each entry in the finetuning dataset contains examples with an instruction, input, and output for the Llama 3 to follow structured guidance and generate relevant responses.
The Llama 3 model, loaded in 4-bit quantization for eficiency, is set up using a specific training configuration. Several experiments have been conducted to track the training loss to keep it minimal. To supervise the finetuning, the SFT trainer is enabled. The SFT trainer configuration, along with LoRa setup, where the training loss is minimal, is referred to as the optimal training configuration, as shown in Table 2.
There is evidence that GPT-2 is an ideal large language model (LLM) for text quality evaluation [18, 19, 20]. Since the Voight Kampf task evaluation is ongoing, the study employs GPT-2 to gauge the ifnetuned model’s perplexity estimation as preliminary results. Out of 22 Voight-Kampf test questions, seven files indicated a perplexity score above forty. In other words, approximately thirty-two percent were more like human-written, and the GPT-2 evaluator failed to detect the AI-generated content.
The Voight Kampf Task in Eloquent Lab 2025 is designed in a way that encourages participants to
build a system that fools detectors developed by the Voight Kampf AI detection Task at PAN 2025. The
PAN detectors evaluate the submissions (text generations) of the finetuned model using five metrics:
Brier, c@1, F1, F0.5u, and arithmetic mean of all the metrics [
Content: The letter is from someone claiming to be Prince Joe Eboh, Chairman of the Contract
Award Committee of the Niger Delta Development Commission (NDDC).
The sender explains that a surplus of $25 million USD from petroleum contracts needs to be discreetly transferred out of Nigeria.
Due to local laws prohibiting civil servants from holding foreign accounts, they seek a foreign partner to temporarily receive the funds.
The recipient is promised 20% of the amount for their cooperation, while 75% will go to committee members and 5% for expenses.
The sender requests personal and banking details from the recipient to initiate the transfer.
The letter emphasizes secrecy and urgency, aiming to complete the transaction in 21 working days.
The sender thanks the recipient for their cooperation and assures that God will reward your kindness and patience.
LLM response: Received via email on 26 January 2013. Original subject: Transfer of NDDC Fund. The text of the letter is a body of text within an email. There is no signature or introduction. The letter is directed at a generic recipient, as it uses a common greeting of Dear SirMadam. There is no return address. The language and style of the letter are formal. The tone is persuasive, polite, and courteous, but also suspiciously flattering and manipulative. The tone is urgent and secretive. The overall appearance of the letter indicates that it is likely a scam. The text of the letter indicates that the scam is likely an advance-fee scam and a phishing attempt. There is an intention to part the recipient from their money, likely in exchange for nothing. There is a risk of a penalty for attempting to transfer money out of the country without a proper license. There is a risk of legal prosecution for attempting to do as the sender requests because the sender’s requests are likely illegal. There is a likelihood of being added to a spam filter for a short time because of the word
JOE in the subject. There is a high risk of the message being phishing for personal or banking information. The message could contain malware. The message is likely a response to a job inquiry. The message is likely a response to a request for money. The message could be part of a pyramid scheme. The message is likely a scam .... } ] } }
The moderately high overall performance of the finetuned Llama 3 proves that the data-centric approach is fruitful in breaking the AI detectors. However, the approach’s limitations include repeating given input and compromising reasoning in the responses when attempting to generate human-like responses, as shown in the above sample output. Despite the limitations, the findings proved that the study’s approach is seen as competitive on the Eloquent leader board. The findings suggest that the companies that rely solely on detectors must evolve, and existing detectors are insuficient. The insights from this data-centric approach should for detectors to accurate labeling that reduces the risk of deception. For instance, the finetuned model, available on HuggingFace , can be used to generate more samples that can be added to the detector’s training set labeled as AI-generated so that detectors learn from their own blind spot. Additional inspection includes finding what features helped to fool the detectors and updating the detectors with new feature weights to detect these subtle human-like generations. Therefore, the study’s insights can benefit companies labeling AI-generated content to develop detectors in support of the EU’s suggestion to tackle misinformation in sensitive domains like news, education, and so on. 1,087 paragraphs of texts authored by a human 917 text samples related to real news collected from Politifact 21,417 true articles obtained from diferent legitimate news sites 10,269 statements from online platforms
The author would like to acknowledge the university’s High Performance & Scientific Computing Team for providing us access to the Nvidia H100 GPU for finetuning and evaluating the Llama 3 and GPT-2 LLMs.
During the preparation of this work, the author(s) used Grammarly in order to: Grammar and spelling check. After using these tool(s)/service(s), the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content. content, social context, and spatiotemporal information for studying fake news on social media, Big data 8 (2020) 171–188. [16] H. Ahmed, I. Traore, S. Saad, Detecting opinion spams and fake news using text classification,
Security and Privacy 1 (2018) e9. [17] H. Ahmed, I. Traore, S. Saad, Detection of online fake news using n-gram analysis and machine learning techniques, in: Intelligent, Secure, and Dependable Systems in Distributed and Cloud Environments: First International Conference, ISDDC 2017, Vancouver, BC, Canada, October 26-28, 2017, Proceedings 1, Springer, 2017, pp. 127–138. [18] Y. Liu, et al., Assessing text readability and quality with language models (2020). [19] S. Dudy, S. Bedrick, Are some words worth more than others?, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, volume 2020, 2020, p. 131. [20] A. Das, R. M. Verma, Can machines tell stories? a comparative study of deep neural language models and metrics, IEEE Access 8 (2020) 181258–181292.