Large language models have revolutionized the field of natural language processing. As these models become more widespread, concerns about the spread of misinformation and potential misuse have grown. Consequently, distinguishing between texts written by humans and those generated by machines has become a significant challenge. In this paper, we describe our method for addressing the AuTexTification task at IberLEF 2024, which includes two main subtasks. The first subtask is a binary classification challenge that requires distinguishing between texts written by humans and those generated by AI. The second subtask is a multi-class problem involving six text generation models (A, B, C, D, E, and F). Both subtasks are conducted in multiple languages. We selected three BERT-like models as the baseline models and then used the soft voting technique to improve accuracy. The results of the test set showed that soft voting outperformed the individual models.
The advancement of artificial intelligence and machine learning, particularly in the field of Natural
Language Processing (NLP), has led to significant progress in automatic text generation. Models such
as Generative Pre-trained Transformers (GPT) [
This research is focused on fine-tuning three advanced language models: Enhanced Representation
through kNowledge Integration Multilingual (ErnieM) [
The paper is organized as follows: First, we provide an overview of related works, highlighting previous studies on AI text generation and detection. Next, the methodology section details our approach, including the fine-tuning of ErnieM, BLOOM-560m, and mDeBERTaV3 models and the use of soft voting for ensemble learning. In the Experiments section, we present our models’ performance metrics and compare the ensemble method’s efectiveness against individual models. Finally, the conclusion summarizes our findings, discusses their implications, and suggests directions for future research.
Text classification in NLP involves assigning texts to predefined categories using various models and
algorithms, including traditional methods like Naive Bayes and advanced deep learning models like
Bidirectional Encoder Representations from Transformers (BERT) [
The precision and efectiveness of text classification have greatly advanced due to improvements in machine learning techniques and the availability of extensive datasets. This progress facilitates more refined and accurate classifications, fostering innovation in fields such as customer service automation and content recommendation systems. Nonetheless, challenges persist, including managing ambiguous or context-dependent texts and ensuring the models operate without bias.
Due to the high sensitivity in this field, numerous studies have been conducted in recent years to distinguish between human-written and machine-generated texts. The PAN@CLEF 2024 shared task [18] provided a binary English training dataset for this purpose. However, the test set difered from the training data, and participants were asked to identify the human-authored text from two texts of a sample. Participants were required to submit a Docker file of their approach via the TIRA platform [ 19].
The SemEval 2024 task 8 [20, 21] consisted of three subtasks. In subtask A, participants needed to distinguish between human and machine-generated text using binary data. In subtask B, they had to predict which language model generated a given text if it was machine-written. In subtask C, the objective was to identify which part of the text had been altered.
In the first version of the AuTexTification shared task at IberLEF 2023 [ 22], various approaches were proposed to distinguish between human and machine-generated texts in both English and Spanish. Villegas-Trejo et al. [23] utilized traditional feature extraction algorithms and machine learning models to address both subtasks. Their findings indicated that the XGB model, when combined with the TF-IDF n-gram feature extraction method and enhanced with stylometric features, demonstrated the highest performance across their experiments. Scheibe and Mandl [24] have made significant contributions by leveraging transformer-based models for text classification tasks. Specifically, in subtask 1 of their study, they employed the DeBERTaV2 model to distinguish between human and machine-generated texts. Gritsay et al. [25] applied a fine-tuning approach to large pre-trained language encoder models, specifically XLM-RoBERTa, mDeBERTa, and MiniLM-V2. They passed the CLS token through three fully connected layers. Their observations showed that mDeBERTa achieved the best F1 score.
This section will discuss the dataset and the proposed approach in detail. We will begin by describing the characteristics and composition of the dataset used in our experiments, including the data sources, and the data preparation. We will outline our proposed approach, including the models and techniques employed. This will encompass the fine-tuning of specific language models, the feature extraction methods applied, and the ensemble learning strategies implemented to enhance prediction accuracy. By providing a comprehensive overview of both the dataset and our methodology, we aim to ofer a clear understanding of the foundations and innovations of our research.
The AUtomated TEXt IdenTIFICATION on languages of the Iberian peninsula (IberAuTexTification) is an expanded version of the AuTexTification task at IberLEF 2023, focusing on more models, domains, and languages including Spanish, Catalan, Basque, Galician, Portuguese, and English. Participants in this task develop models to diferentiate between human-written and automatically generated texts (Subtask 1) and identify the specific model used for text generation (Subtask 2). The training dataset encompasses five diverse domains while the testing dataset includes two additional domains. Texts are generated using a variety of models such as GPT-3.5, GPT-4, LLaMA, Coral, Command, Falcon, and MPT, sourced from platforms like OpenAI, Amazon Bedrock, Anthropic, Cohere, AI21, Google Vertex AI, and Meta. Datasets are curated using TextMachina [26], incorporating texts from controlled domains like essays, news, social media, Wikipedia, WikiHow, and uncontrolled domains sourced from OSCAR [27] and Colossal Cleaned Multilingual Common Crawl. Enthusiasts in this field can visit the Zenodo website to access the training1 and test datasets2.
For our study, we utilized three advanced language models: mDeBERTaV3, ErnieM, and BLOOM-560m, each with its corresponding tokenizer to preprocess the text data. We determined that a token length of 170 tokens per sample was optimal based on the average length of our text samples, ensuring a balance between computational eficiency and preserving information. Any tokens beyond this limit were discarded to maintain consistency across all samples.
Given the resource constraints on Google Colaboratory, particularly the limited GPU availability, we implemented selective sampling for model fine-tuning. For both Subtasks 1 and 2, we selected 50,000 samples from the training dataset to fine-tune the BLOOM-560m model. This approach ensured that we could manage the computational load while still providing suficient data for the model to learn efectively.
In Subtask 1, which involved distinguishing between human and machine-generated text, we selected a sample size of 60,000 for fine-tuning the mDeBERTaV3 model. For other scenarios and models, we utilized the entire training dataset, leveraging all available data to maximize the training efectiveness. This comprehensive approach aimed to ensure that our models were well-trained and capable of performing robust text classification tasks.
BLOOM-560m, ErnieM, and mDeBERTaV3 are three state-of-the-art transformer-based models that have significantly advanced the field of NLP.
BLOOM-560m is a multilingual model developed as part of the BigScience initiative, featuring 560 million parameters. It is designed to provide open-access language processing capabilities across multiple languages, promoting inclusivity and transparency in NLP research. The model’s extensive training on diverse datasets allows it to handle complex linguistic contexts, making it suitable for a wide range of large-scale NLP tasks.
ErnieM by Baidu integrates external knowledge sources, such as knowledge graphs, into its language representation learning. This integration enhances ErnieM’s ability to generate contextually accurate and semantically rich text, especially in multilingual settings. Its ability to incorporate structured knowledge allows it to understand and process intricate linguistic patterns more efectively.
mDeBERTaV3 builds on the BERT architecture with a focus on improving performance through a disentangled attention mechanism. This mechanism separates content and positional information, enhancing the model’s understanding of language nuances. Trained on a large and diverse dataset, mDeBERTaV3 excels in tasks requiring deep contextual understanding, such as text classification and machine translation, due to its improved training eficiency and comprehensive language processing capabilities. 1https://zenodo.org/records/10853560 2https://zenodo.org/records/11034382
In our approach, we utilized the Soft Voting technique for ensemble learning to enhance the performance of our text classification models. Soft Voting involves averaging the predicted probabilities of multiple models and selecting the class with the highest average probability as the final prediction. This method leverages the strengths of each individual model, compensating for their weaknesses and leading to improved overall accuracy and robustness.
We implemented Soft Voting with our three fine-tuned models: BLOOM-560m, ErnieM, and mDeBERTaV3. By combining the predictive power of these models, we were able to achieve superior performance compared to using any single model alone. The diversity in architecture and training methodologies of these models ensures a more comprehensive understanding of the text, thereby improving the reliability and accuracy of our predictions. Our experiments demonstrated that the Soft Voting ensemble approach significantly outperforms individual models in both Subtask 1 and Subtask 2, highlighting its efectiveness in distinguishing between human and machine-generated text across multiple languages and domains.
This section outlines the experimental setup and procedures used to evaluate our models. First, we discuss the hyperparameter tuning process to optimize the performance of BLOOM-560m, ErnieM, and mDeBERTaV3 models. We then present the results of our experiments, highlighting the efectiveness of individual models and the benefits of using a Soft Voting ensemble approach. Through detailed analysis and comparative metrics, we demonstrate the superior performance of our ensemble method in distinguishing between human and machine-generated text and in accurately attributing text to its generative model.
In this study, we used Google Collaboratory to fine-tune our models: BLOOM-560m, ErnieM, and mDeBERTaV3. The fine-tuning process was conducted using the Trainer API from the Hugging Face Transformers library [28], ensuring eficient training and evaluation. A learning rate of 5e-5 was consistently applied across all models to maintain stable and efective training. For ErnieM and mDeBERTaV3, we performed fine-tuning over 8 epochs, while BLOOM-560m was tuned for 3 epochs due to its larger size and complexity. Additional hyperparameters were uniformly set for all models to optimize performance: fp16 was enabled for mixed precision training, allowing faster computation and reduced memory usage; both per_device_train_batch_size and per_device_eval_batch_size were set to 8; a weight decay of 0.01 was applied to prevent overfitting by penalizing large weights; and gradient_accumulation_steps were set to 4 to simulate a larger batch size and stabilize training.
For both subtasks in our study, we employed the macro F1 score as the primary evaluation metric. The macro F1 score is particularly well-suited for imbalanced datasets as it calculates the F1 score for each class independently and then averages them, giving equal weight to each class regardless of its frequency. This approach ensures that the performance of our models is evaluated comprehensively across all classes, providing a balanced measure of precision and recall.
We began by partitioning the training data into 80% for training and 20% for validation. After fine-tuning our models on the training set, we evaluated their performance individually and using a Soft voting ensemble method, which combined the outputs of ErnieM, mDebertaV3, and BLOOM-560m.
For this study, we submitted two runs for each subtask. In Subtask 1, we submitted the output of mDebertaV3 as run1 and the Soft voting ensemble as run2. In Subtask 2, we submitted the output of BLOOM-560m as run1 and the Soft voting ensemble as run2. The test set results indicated that the Soft (a) Confusion matrix for Subtask 1.
(b) Confusion matrix for Subtask 2. voting approach consistently outperformed the individual models in all subtasks. The results for the validation set are detailed in Table 1, while the results for the test set are provided in Table 2.
We examine the confusion matrix of the best model on the test data (soft voting). For Subtask 1 (Figure 1a), the confusion matrix shows that out of 11,751 generated texts, 11,629 were correctly identified, and only 122 were misclassified as human-written. Conversely, out of 10,182 human-written texts, 9,676 were correctly identified, and 506 were misclassified as machine-generated. This high level of accuracy demonstrates the efectiveness of our approach in distinguishing between human and machine-generated texts.
In Subtask 2 (Figure 1b), the confusion matrix reveals the performance of the soft voting approach in a multi-class classification scenario. The model achieved high accuracy across diferent categories, with notable performance in classes D and F, where 2,364 and 1,975 instances were correctly classified, respectively. However, there were some misclassifications, such as in class E, where 132 instances were incorrectly labeled as class A. Despite these challenges, the overall results underscore the robustness and eficiency of the soft voting ensemble method in handling diverse and complex text classification tasks.
In this study, we explored the eficacy of three language models—ErnieM, mDebertaV3, and BLOOM560m—for the tasks of distinguishing between human-written and machine-generated texts and attributing generated texts to specific models. We fine-tuned these models using a carefully partitioned training dataset and applied a Soft voting ensemble method to enhance prediction accuracy. Our experiments demonstrated that the Soft voting ensemble approach significantly outperformed individual models in both subtasks. Specifically, for Subtask 1, the combined model yielded better results compared to mDebertaV3 alone, and for Subtask 2, it outperformed BLOOM-560m. These findings underscore the potential of ensemble learning to improve the robustness and accuracy of AI text detection systems. The results from the validation and test sets clearly indicated that our ensemble method could generalize well across diferent domains and languages, reflecting the broader applicability of our approach. This work contributes to the ongoing efort to develop reliable detectors for distinguishing between human and AI-generated texts, highlighting the importance of ensemble methods in achieving higher accuracy and robustness. Future research could explore the integration of additional models and the application of advanced ensemble techniques to further enhance performance. Additionally, addressing the challenges of model biases and ensuring fairness in AI-generated text detection remain critical areas for further investigation. [15] H. B. Giglou, M. Rahgouy, T. Rahgooy, M. K. Sheykhlan, E. Mohammadzadeh, Author profiling: Bot and gender prediction using a multi-aspect ensemble approach., in: CLEF (Working Notes), 2019. [16] M. Rahgouy, H. B. Giglou, T. Rahgooy, M. K. Sheykhlan, E. Mohammadzadeh, Cross-domain authorship attribution: Author identification using a multi-aspect ensemble approach., in: CLEF (Working Notes), 2019. [17] M. K. Sheykhlan, J. Shafi, S. Kosari, Pars-hao: Hate speech and ofensive language detection on persian social media using ensemble learning, Authorea Preprints (2023). [18] J. Bevendorf, X. B. Casals, B. Chulvi, D. Dementieva, A. Elnagar, D. Freitag, M. Fröbe, D. Korenčić, M. Mayerl, A. Mukherjee, A. Panchenko, M. Potthast, F. Rangel, P. Rosso, A. Smirnova, E. Stamatatos, B. Stein, M. Taulé, D. Ustalov, M. Wiegmann, E. Zangerle, Overview of PAN 2024: Multi-Author Writing Style Analysis, Multilingual Text Detoxification, Oppositional Thinking Analysis, and Generative AI Authorship Verification, in: Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Fourteenth International Conference of the CLEF Association (CLEF 2024), Lecture Notes in Computer Science, Springer, Berlin Heidelberg New York, 2024. [19] M. Fröbe, M. Wiegmann, N. Kolyada, B. Grahm, T. Elstner, F. Loebe, M. Hagen, B. Stein, M. Potthast, Continuous Integration for Reproducible Shared Tasks with TIRA.io, in: J. Kamps, L. Goeuriot, F. Crestani, M. Maistro, H. Joho, B. Davis, C. Gurrin, U. Kruschwitz, A. Caputo (Eds.), Advances in Information Retrieval. 45th European Conference on IR Research (ECIR 2023), Lecture Notes in Computer Science, Springer, Berlin Heidelberg New York, 2023, pp. 236–241. URL: https://link. springer.com/chapter/10.1007/978-3-031-28241-6_20. doi:10.1007/978-3-031-28241-6_20. [20] Y. Wang, J. Mansurov, P. Ivanov, J. Su, A. Shelmanov, A. Tsvigun, C. Whitehouse, O. Mohammed Afzal, T. Mahmoud, T. Sasaki, T. Arnold, A. Aji, N. Habash, I. Gurevych, P. Nakov, M4: Multi-generator, multi-domain, and multi-lingual black-box machine-generated text detection, in: Y. Graham, M. Purver (Eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, St. Julian’s, Malta, 2024, pp. 1369–1407. URL: https: //aclanthology.org/2024.eacl-long.83. [21] Y. Wang, J. Mansurov, P. Ivanov, J. Su, A. Shelmanov, A. Tsvigun, O. M. Afzal, T. Mahmoud, G. Puccetti, T. Arnold, et al., Semeval-2024 task 8: Multidomain, multimodel and multilingual machine-generated text detection, arXiv preprint arXiv:2404.14183 (2024). [22] A. M. Sarvazyan, J. Á. González, M. Franco-Salvador, F. Rangel, B. Chulvi, P. Rosso, Overview of autextification at iberlef 2023: Detection and attribution of machine-generated text in multiple domains, arXiv preprint arXiv:2309.11285 (2023). [23] Z. Villegas-Trejo, H. Gómez-Adorno, S.-L. Ojeda-Trueba, Exploring text representations for detecting automatically generated text (2023). [24] T. Scheibe, T. Mandl, Univ. of hildesheim at autextification 2023: Detection of automatically generated texts (2023). [25] G. Gritsay, A. Grabovoy, A. Kildyakov, Y. Chekhovich, Automated text identification: Multilingual transformer-based models approach (2023). [26] A. M. Sarvazyan, J. Á. González, M. Franco-Salvador, Textmachina: Seamless generation of machine-generated text datasets, arXiv preprint arXiv:2401.03946 (2024). [27] J. Abadji, P. O. Suarez, L. Romary, B. Sagot, Towards a cleaner document-oriented multilingual crawled corpus, arXiv preprint arXiv:2201.06642 (2022). [28] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al., Huggingface’s transformers: State-of-the-art natural language processing, arXiv preprint arXiv:1910.03771 (2019).