The advancement of artificial intelligence and machine learning, particularly in the field of Natural Language Processing (NLP), has led to significant progress in automatic text generation. Models such as Generative Pre-trained Transformers (GPT) [1,2,3], Pathways Language Model (PaLM) [4], and BigScience Large Open-science Open-access Multilingual Language Model (BLOOM) [5] can produce text that closely resembles human writing in terms of coherence, style, and grammar. These capabilities have broad applications, including conversational agents, code completion, machine translation, and generating radiology reports, impacting both economic and social spheres. Despite their benefits, these models also present challenges, such as the spread of misinformation, academic fraud [6,7], and the creation of offensive or biased content [8,9]. The AuTexTification task at IberLEF 2024 [10,11] addresses these challenges by distinguishing between human-written and machine-generated text.
This research is focused on fine-tuning three advanced language models: Enhanced Representation through kNowledge Integration Multilingual (ErnieM) [12], BLOOM-560m and Multilingual Decodingenhanced BERT with disentangled attention (mDeBERTaV3) [13]. Each model was individually adjusted to optimize its performance for our specific task. To further enhance the accuracy of our predictions, we employed an ensemble learning approach using soft voting. This technique combines the strengths of multiple models by averaging their predictions, thereby improving overall performance. Our final results demonstrated that the ensemble method significantly outperformed the standalone models. This finding underscores the effectiveness of integrating multiple models to achieve higher prediction accuracy and reliability in distinguishing between human-authored and AI-generated texts.
The paper is organized as follows: First, we provide an overview of related works, highlighting previous studies on AI text generation and detection. Next, the methodology section details our approach, including the fine-tuning of ErnieM, BLOOM-560m, and mDeBERTaV3 models and the use of soft voting for ensemble learning. In the Experiments section, we present our models' performance metrics and compare the ensemble method's effectiveness against individual models. Finally, the conclusion summarizes our findings, discusses their implications, and suggests directions for future research.
Text classification in NLP involves assigning texts to predefined categories using various models and algorithms, including traditional methods like Naive Bayes and advanced deep learning models like Bidirectional Encoder Representations from Transformers (BERT) [14] and GPT. These models process textual data to identify patterns that differentiate various categories, enabling applications such as author identification [15], author attribution [16], and detecting hate or offensive content [17].
The precision and effectiveness of text classification have greatly advanced due to improvements in machine learning techniques and the availability of extensive datasets. This progress facilitates more refined and accurate classifications, fostering innovation in fields such as customer service automation and content recommendation systems. Nonetheless, challenges persist, including managing ambiguous or context-dependent texts and ensuring the models operate without bias.
Due to the high sensitivity in this field, numerous studies have been conducted in recent years to distinguish between human-written and machine-generated texts. The PAN@CLEF 2024 shared task [18] provided a binary English training dataset for this purpose. However, the test set differed from the training data, and participants were asked to identify the human-authored text from two texts of a sample. Participants were required to submit a Docker file of their approach via the TIRA platform [19].
The SemEval 2024 task 8 [20,21] consisted of three subtasks. In subtask A, participants needed to distinguish between human and machine-generated text using binary data. In subtask B, they had to predict which language model generated a given text if it was machine-written. In subtask C, the objective was to identify which part of the text had been altered.
In the first version of the AuTexTification shared task at IberLEF 2023 [22], various approaches were proposed to distinguish between human and machine-generated texts in both English and Spanish. Villegas-Trejo et al. [23] utilized traditional feature extraction algorithms and machine learning models to address both subtasks. Their findings indicated that the XGB model, when combined with the TF-IDF n-gram feature extraction method and enhanced with stylometric features, demonstrated the highest performance across their experiments. Scheibe and Mandl [24] have made significant contributions by leveraging transformer-based models for text classification tasks. Specifically, in subtask 1 of their study, they employed the DeBERTaV2 model to distinguish between human and machine-generated texts. Gritsay et al. [25] applied a fine-tuning approach to large pre-trained language encoder models, specifically XLM-RoBERTa, mDeBERTa, and MiniLM-V2. They passed the CLS token through three fully connected layers. Their observations showed that mDeBERTa achieved the best F1 score.
This section will discuss the dataset and the proposed approach in detail. We will begin by describing the characteristics and composition of the dataset used in our experiments, including the data sources, and the data preparation. We will outline our proposed approach, including the models and techniques employed. This will encompass the fine-tuning of specific language models, the feature extraction methods applied, and the ensemble learning strategies implemented to enhance prediction accuracy. By providing a comprehensive overview of both the dataset and our methodology, we aim to offer a clear understanding of the foundations and innovations of our research.
The AUtomated TEXt IdenTIFICATION on languages of the Iberian peninsula (IberAuTexTification) is an expanded version of the AuTexTification task at IberLEF 2023, focusing on more models, domains, and languages including Spanish, Catalan, Basque, Galician, Portuguese, and English. Participants in this task develop models to differentiate between human-written and automatically generated texts (Subtask 1) and identify the specific model used for text generation (Subtask 2). The training dataset encompasses five diverse domains while the testing dataset includes two additional domains. Texts are generated using a variety of models such as GPT-3.5, GPT-4, LLaMA, Coral, Command, Falcon, and MPT, sourced from platforms like OpenAI, Amazon Bedrock, Anthropic, Cohere, AI21, Google Vertex AI, and Meta. Datasets are curated using TextMachina [26], incorporating texts from controlled domains like essays, news, social media, Wikipedia, WikiHow, and uncontrolled domains sourced from OSCAR [27] and Colossal Cleaned Multilingual Common Crawl. Enthusiasts in this field can visit the Zenodo website to access the training 1 and test datasets2 .
For our study, we utilized three advanced language models: mDeBERTaV3, ErnieM, and BLOOM-560m, each with its corresponding tokenizer to preprocess the text data. We determined that a token length of 170 tokens per sample was optimal based on the average length of our text samples, ensuring a balance between computational efficiency and preserving information. Any tokens beyond this limit were discarded to maintain consistency across all samples.
Given the resource constraints on Google Colaboratory, particularly the limited GPU availability, we implemented selective sampling for model fine-tuning. For both Subtasks 1 and 2, we selected 50,000 samples from the training dataset to fine-tune the BLOOM-560m model. This approach ensured that we could manage the computational load while still providing sufficient data for the model to learn effectively.
In Subtask 1, which involved distinguishing between human and machine-generated text, we selected a sample size of 60,000 for fine-tuning the mDeBERTaV3 model. For other scenarios and models, we utilized the entire training dataset, leveraging all available data to maximize the training effectiveness. This comprehensive approach aimed to ensure that our models were well-trained and capable of performing robust text classification tasks.
BLOOM-560m, ErnieM, and mDeBERTaV3 are three state-of-the-art transformer-based models that have significantly advanced the field of NLP.
BLOOM-560m is a multilingual model developed as part of the BigScience initiative, featuring 560 million parameters. It is designed to provide open-access language processing capabilities across multiple languages, promoting inclusivity and transparency in NLP research. The model's extensive training on diverse datasets allows it to handle complex linguistic contexts, making it suitable for a wide range of large-scale NLP tasks.
ErnieM by Baidu integrates external knowledge sources, such as knowledge graphs, into its language representation learning. This integration enhances ErnieM's ability to generate contextually accurate and semantically rich text, especially in multilingual settings. Its ability to incorporate structured knowledge allows it to understand and process intricate linguistic patterns more effectively. mDeBERTaV3 builds on the BERT architecture with a focus on improving performance through a disentangled attention mechanism. This mechanism separates content and positional information, enhancing the model's understanding of language nuances. Trained on a large and diverse dataset, mDeBERTaV3 excels in tasks requiring deep contextual understanding, such as text classification and machine translation, due to its improved training efficiency and comprehensive language processing capabilities.
In our approach, we utilized the Soft Voting technique for ensemble learning to enhance the performance of our text classification models. Soft Voting involves averaging the predicted probabilities of multiple models and selecting the class with the highest average probability as the final prediction. This method leverages the strengths of each individual model, compensating for their weaknesses and leading to improved overall accuracy and robustness.
We implemented Soft Voting with our three fine-tuned models: BLOOM-560m, ErnieM, and mDe-BERTaV3. By combining the predictive power of these models, we were able to achieve superior performance compared to using any single model alone. The diversity in architecture and training methodologies of these models ensures a more comprehensive understanding of the text, thereby improving the reliability and accuracy of our predictions. Our experiments demonstrated that the Soft Voting ensemble approach significantly outperforms individual models in both Subtask 1 and Subtask 2, highlighting its effectiveness in distinguishing between human and machine-generated text across multiple languages and domains.
This section outlines the experimental setup and procedures used to evaluate our models. First, we discuss the hyperparameter tuning process to optimize the performance of BLOOM-560m, ErnieM, and mDeBERTaV3 models. We then present the results of our experiments, highlighting the effectiveness of individual models and the benefits of using a Soft Voting ensemble approach. Through detailed analysis and comparative metrics, we demonstrate the superior performance of our ensemble method in distinguishing between human and machine-generated text and in accurately attributing text to its generative model.
In this study, we used Google Collaboratory to fine-tune our models: BLOOM-560m, ErnieM, and mDeBERTaV3. The fine-tuning process was conducted using the Trainer API from the Hugging Face Transformers library [28], ensuring efficient training and evaluation. A learning rate of 5e-5 was consistently applied across all models to maintain stable and effective training. For ErnieM and mDeBERTaV3, we performed fine-tuning over 8 epochs, while BLOOM-560m was tuned for 3 epochs due to its larger size and complexity. Additional hyperparameters were uniformly set for all models to optimize performance: fp16 was enabled for mixed precision training, allowing faster computation and reduced memory usage; both per_device_train_batch_size and per_device_eval_batch_size were set to 8; a weight decay of 0.01 was applied to prevent overfitting by penalizing large weights; and gradient_accumulation_steps were set to 4 to simulate a larger batch size and stabilize training.
For both subtasks in our study, we employed the macro F1 score as the primary evaluation metric. The macro F1 score is particularly well-suited for imbalanced datasets as it calculates the F1 score for each class independently and then averages them, giving equal weight to each class regardless of its frequency. This approach ensures that the performance of our models is evaluated comprehensively across all classes, providing a balanced measure of precision and recall.
We began by partitioning the training data into 80% for training and 20% for validation. After fine-tuning our models on the training set, we evaluated their performance individually and using a Soft voting ensemble method, which combined the outputs of ErnieM, mDebertaV3, and BLOOM-560m.
For this study, we submitted two runs for each subtask. In Subtask 1, we submitted the output of mDebertaV3 as run1 and the Soft voting ensemble as run2. In Subtask 2, we submitted the output of BLOOM-560m as run1 and the Soft voting ensemble as run2. The test set results indicated that the Soft voting approach consistently outperformed the individual models in all subtasks. The results for the validation set are detailed in Table 1, while the results for the test set are provided in Table 2. We examine the confusion matrix of the best model on the test data (soft voting). For Subtask 1 (Figure 1a), the confusion matrix shows that out of 11,751 generated texts, 11,629 were correctly identified, and only 122 were misclassified as human-written. Conversely, out of 10,182 human-written texts, 9,676 were correctly identified, and 506 were misclassified as machine-generated. This high level of accuracy demonstrates the effectiveness of our approach in distinguishing between human and machine-generated texts.
In Subtask 2 (Figure 1b), the confusion matrix reveals the performance of the soft voting approach in a multi-class classification scenario. The model achieved high accuracy across different categories, with notable performance in classes D and F, where 2,364 and 1,975 instances were correctly classified, respectively. However, there were some misclassifications, such as in class E, where 132 instances were incorrectly labeled as class A. Despite these challenges, the overall results underscore the robustness and efficiency of the soft voting ensemble method in handling diverse and complex text classification tasks.
In this study, we explored the efficacy of three language models-ErnieM, mDebertaV3, and BLOOM-560m-for the tasks of distinguishing between human-written and machine-generated texts and attributing generated texts to specific models. We fine-tuned these models using a carefully partitioned training dataset and applied a Soft voting ensemble method to enhance prediction accuracy. Our experiments demonstrated that the Soft voting ensemble approach significantly outperformed individual models in both subtasks. Specifically, for Subtask 1, the combined model yielded better results compared to mDebertaV3 alone, and for Subtask 2, it outperformed BLOOM-560m. These findings underscore the potential of ensemble learning to improve the robustness and accuracy of AI text detection systems. The results from the validation and test sets clearly indicated that our ensemble method could generalize well across different domains and languages, reflecting the broader applicability of our approach. This work contributes to the ongoing effort to develop reliable detectors for distinguishing between human and AI-generated texts, highlighting the importance of ensemble methods in achieving higher accuracy and robustness. Future research could explore the integration of additional models and the application of advanced ensemble techniques to further enhance performance. Additionally, addressing the challenges of model biases and ensuring fairness in AI-generated text detection remain critical areas for further investigation.