The advancement of large language models (LLMs) and their increasing utilization for text generation have raised significant concerns about misinformation and potential misuse. This research focuses on developing a dependable model to diferentiate between human-authored text and LLM-generated content. By employing transformer models and ensemble learning techniques, we aim to improve the accuracy and reliability of text classification. Our results indicate that the proposed model efectively identifies and categorizes texts, providing a valuable tool for addressing the challenges posed by the widespread use of LLM-generated text.
Recent advancements in LLMs, particularly with the development of sophisticated systems like GPT-3.5
and GPT-4, have revolutionized content creation across various fields, from advertising and news
writing to education and medical research. These models are now capable of generating text that closely
mimics human writing, enhancing productivity in numerous professional workflows. However, this
rapid integration comes with significant challenges, including the spread of misinformation [
The detection of AI-generated text has become a critical area of research, driven by the need to safeguard the integrity of information across digital platforms. Traditional approaches to text verification, which rely heavily on stylistic and linguistic features, are often insuficient when faced with the sophistication of modern LLMs. These models can generate content that not only mirrors human writing but also adapts to various contexts and styles, making manual and even some automated detection methods obsolete. Consequently, more advanced techniques are required to diferentiate between human-authored and machine-generated text efectively.
In this study, initially, we fine-tuned transformer models using the training dataset. This process involved adjusting the model parameters to fit the specific characteristics of our dataset better, thereby improving the model’s performance on our particular task. We then compared their accuracy with that of a cumulative learning model. The results indicated that incorporating the cumulative learning model improved predictive accuracy. This enhancement highlights the potential of cumulative learning in refining the performance of transformer models and underscores its value in developing more reliable AI systems for text classification tasks.
In recent years, the field of AI-text generator detection has seen significant advancements, driven by the
increasing sophistication of LLMs such as GPT-3.5 and GPT-4. Early approaches focused on identifying
stylistic and lexical discrepancies between human-written and machine-generated texts. These methods
often utilized shallow machine learning techniques and handcrafted features, but their efectiveness
diminished as LLMs became more advanced. Islam et al. [
With the rapid advancement and widespread adoption of LLMs, distinguishing between human- and machine-authored texts has become increasingly challenging. Leveraging expertise in authorship verification, the Generative AI Authorship Verification Task @ PAN, in collaboration with the VoightKampf Task @ ELOQUENT Lab, aims to address this by having participants identify the human-written text from a pair of texts. This builder-breaker setup will see PAN participants develop detection systems, while ELOQUENT participants focus on creating text generation and obfuscation methods.
The dataset includes various genres such as news articles, Wikipedia intro texts, and fanfiction,
sourced from ELOQUENT participants. Additionally, a bootstrap dataset containing real and fake news
articles from 2021 U.S. headlines is provided for training purposes. To access the dataset, participants
must register on TIRA [
In this task, we focused on improving the classification of text by specifically ignoring newline characters within the data while preserving all other information. This approach allows us to maintain the integrity of the content while addressing the task as a classification problem. To achieve this, we utilized the capabilities of three powerful transformer models: BERT, Roberta, and Electra. Each of these models comes with its own tokenizer, which plays a crucial role in preparing the text for the binary classification process. These models are well-regarded in the field of natural language processing for their efectiveness in understanding and generating human-like text. Considering the inherent 512-token limit of these models, we decided to focus on the first 512 tokens of each text sample. This decision was made to ensure that our models could process the data eficiently without exceeding their maximum token capacity. For text samples that exceeded 512 tokens, only the initial 512 tokens were included in the analysis, while the remaining tokens were disregarded. This approach ensures that we work within the constraints of the models while still capturing a significant portion of the text for classification purposes.
BERT (Bidirectional Encoder Representations from Transformers), Roberta (Robustly Optimized BERT
Approach), and Electra (Eficiently Learning an Encoder that Classifies Token Replacements
Accurately) are three advanced transformer models that have significantly impacted the field of NLP. BERT,
introduced by Google, revolutionized NLP by utilizing bidirectional training, allowing the model to
understand the context of a word based on all its surroundings, leading to state-of-the-art performance
on various tasks. Roberta, a refinement of BERT by Facebook AI, enhanced the pre-training process by
using larger batches, more data, and longer training times, which resulted in improved performance
and robustness. Electra, also developed by Google, introduced a novel pre-training approach where
the model learns to distinguish real input tokens from "fake" ones generated by a separate generator
model. This method makes Electra more eficient and often faster to train compared to BERT, while
still achieving competitive or superior performance. Each of these models has its own tokenizer to
prepare text for analysis. These models apply to a variety of NLP tasks, ranging from text classification
tasks [
Hard Ensemble Learning is an advanced technique in machine learning that involves combining multiple models, referred to as base learners, to create a more powerful and robust predictive model. Unlike conventional ensemble methods like bagging and boosting, which mainly aim to combine diverse but weaker models, Hard Ensemble Learning incorporates several complex and high-performance models to address dificult tasks. By harnessing the collective intelligence of various models, each trained on distinct data aspects, this approach enhances overall predictive accuracy and generalization.
In this study, we developed three transformer models: BERT, RoBERTa, and Electra. Each model was ifne-tuned using the respective training dataset. Our objective with ensemble learning is to combine the strengths of these models to improve overall system performance. Detailed specifics will be provided in the following section.
In this study, we utilized Google Colaboratory’s GPU to fine-tune the BERT, Roberta, and Electra models. Due to token constraints, we limited our consideration to 512 tokens. For all models, we set the hyperparameter learning rate to 2e-5 and the epoch value to 10.
In our study, we evaluated the system using the same metrics as previous PAN authorship verification
tasks [
• The arithmetic mean of all the metrics above.
Given LLMs’ complexity and human-like text generation, traditional feature extraction methods used in tasks like Author Profiling [ 18] and Authorship Attribution [19], which previously produced acceptable results, are no longer suficient. In this section, we present our experimental results on the VoightKampf Generative AI Authorship Verification 2024 task using BERT, Roberta, Electra models, and ensemble learning, which leverage the majority vote of the three models for test data samples. The test samples included an ID, text1, and text2. Participants were required to identify which of text1 or text2 was authored by a human or a machine using numerical values in the is_human variable.
We divided the dataset into 60%, 20%, and 20% for the training, validation, and testing phases, respectively. In our approach to classifying test samples, we employed a straightforward method. If the model determined that a human authored text1, we manually assigned a value less than 0.5 to the is_human variable. Conversely, if the model determined that a human authored text2, we assigned a value greater than 0.5 to the is_human variable. In cases where the model was unable to make a decision, we assigned a value of 0.5 to the is_human variable.
Table 1 shows the performance of the approaches used. The experimental results indicate that the hard voting classifier approach is more efective in distinguishing between human-generated and LLM-generated texts. Therefore, we employed the Ensemble learning model for evaluation on the final test set. The performance of this approach on the final dataset is presented in Table 2.
In this study, we addressed the challenge of distinguishing between human-written texts and those generated by LLMs. Our approach leveraged Ensemble learning, incorporating fine-tuned versions of BERT, Roberta, and Electra models. By utilizing these advanced transformer models and applying meticulous fine-tuning on the provided dataset, we were able to enhance the accuracy and reliability of our text classification system. Our findings underscore the potential of Ensemble learning combined with transformer models in advancing the field of authorship verification and enhancing the detection of AI-generated texts. the Fifteenth International Conference of the CLEF Association (CLEF 2024), Lecture Notes in Computer Science, Springer, Berlin Heidelberg New York, 2024. [15] J. Bevendorf, M. Wiegmann, E. Stamatatos, M. Potthast, B. Stein, Overview of the Voight-Kampf Generative AI Authorship Verification Task at PAN 2024, in: G. F. N. Ferro, P. Galuščáková, A. G. S. de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, CEUR-WS.org, 2024. [16] J. Bevendorf, M. Wiegmann, J. Karlgren, L. Dürlich, E. Gogoulou, A. Talman, E. Stamatatos, M. Potthast, B. Stein, Overview of the “Voight-Kampf” Generative AI Authorship Verification Task at PAN and ELOQUENT 2024, in: G. Faggioli, N. Ferro, P. Galuščáková, A. G. S. de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings, CEUR-WS.org, 2024. [17] A. A. Ayele, N. Babakov, J. Bevendorf, X. B. Casals, B. Chulvi, D. Dementieva, A. Elnagar, D. Freitag, M. Fröbe, D. Korenčić, M. Mayerl, D. Moskovskiy, A. Mukherjee, A. Panchenko, M. Potthast, F. Rangel, N. Rizwan, P. Rosso, F. Schneider, A. Smirnova, E. Stamatatos, E. Stakovskii, B. Stein, M. Taulé, D. Ustalov, X. Wang, M. Wiegmann, S. M. Yimam, E. Zangerle, Overview of PAN 2024: Multi-Author Writing Style Analysis, Multilingual Text Detoxification, Oppositional Thinking Analysis, and Generative AI Authorship Verification, in: L. Goeuriot, P. Mulhem, G. Quénot, D. Schwab, L. Soulier, G. M. D. Nunzio, P. Galuščáková, A. G. S. de Herrera, G. Faggioli, N. Ferro (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Fifteenth International Conference of the CLEF Association (CLEF 2024), Lecture Notes in Computer Science, Springer, Berlin Heidelberg New York, 2024. [18] H. B. Giglou, M. Rahgouy, T. Rahgooy, M. K. Sheykhlan, E. Mohammadzadeh, Author profiling: Bot and gender prediction using a multi-aspect ensemble approach, in: L. Cappellato, N. Ferro, D. E. Losada, H. Müller (Eds.), Working Notes of CLEF 2019 - Conference and Labs of the Evaluation Forum, Lugano, Switzerland, September 9-12, 2019, volume 2380 of CEUR Workshop Proceedings, CEUR-WS.org, 2019. URL: https://ceur-ws.org/Vol-2380/paper_231.pdf. [19] M. Rahgouy, H. Giglou, T. Rahgooy, M. Sheykhlan, E. Mohammadzadeh, Cross-domain Authorship Attribution: Author Identification using a Multi-Aspect Ensemble Approach, in: L. Cappellato, N. Ferro, D. Losada, H. Müller (Eds.), CLEF 2019 Labs and Workshops, Notebook Papers, CEURWS.org, 2019. URL: http://ceur-ws.org/Vol-2380/.