Text detoxification is a text-to-text generation task that relies on available data for experiments. In recent years, this task has primarily focused on well-resourced languages while neglecting lower-resource languages. This work explores various approaches to building a multilingual solution for diferent languages, with an emphasis on 9 languages in the Multilingual Text Detoxification Task at PAN 2024. Throughout the experiments, we consider not only diferent model types but also employ fine-tuning on various combinations of datasets. As a result, we achieve third place in human evaluation and show promising progress towards developing a multilingual solution for the text detoxification task using large language models such as mT0 and XGLM. We also observe that fine-tuning on combinations of relatively similar languages is a promising direction-especially when real data for some languages is lacking.
In earlier works, unsupervised methods like CondBERT [8] demonstrated efectiveness in the text
detoxification task by identifying and rephrasing toxic parts of the text. However, these methods were
eventually surpassed by encoder-decoder approaches [
Our work aims to address the underrepresentation of methods for diferent languages by considering a new dataset for nine languages.
Most recent high-performing text detoxification approaches rely on encoder-decoder models. We decided to start with models of this architecture and include decoder-only LLMs, which have shown impressive results in natural language processing (NLP)[13]. Our experiments consider encoder-decoder models such as mBART and mT0, while for the decoder-only approach, we focus on XGLM. Previous studies have highlighted the efectiveness of models like mBART and mT5[ 14], but we decide to also consider mT0 due to its multitask fine-tuning capability, which could be advantageous for text detoxification. Additionally, in our selection of decoder-only model, we choose XGLM because of its multilingual nature and good reported performance results. It’s worth noting that XGLM was not trained on Amharic data; thus, no results are reported in this language here. Regarding the mT0, we aim to consider diferent model sizes such as base 1, large2, and xl3. Also, we consider XGLM (7.5B)4 only.
Along with the dataset provided for the development5, we utilize both English6 and Russian7 ParaDetox [11] datasets during the experiments. Our objective is to utilize these monolingual datasets to tailor models for text detoxification before fine-tuning on the provided multilingual dataset. As previously stated in Section 1, potential discrepancies exist in the collected datasets. Therefore, we do not consider merging such datasets for fine-tuning and decide to utilize them separately.
Though we consider both encoder-decoder and decoder-based models, the fine-tuning process in general is not necessarily diferent. We examine two types of datasets during fine-tuning: utilizing English and Russian ParaDetox datasets as well as the MultilingualParaDetox dataset for nine languages. With the first ones, we explore whether initial fine-tuning on English and Russian languages improves the convergence of multilingual models during final fine-tuning on the MPD dataset. Finally, we conduct ifne-tuning on a multilingual dataset using diferent language combinations: (i) monolingual fine-tuning on each language independently; (ii) multilingual fine-tuning using all available languages; (iii) finetuning on diferent combinations of languages. For the third approach, our hypothesis suggests that closely related languages significantly impact fine-tuning success more than a monolingual approach. 1bigscience/mt0-base 2bigscience/mt0-large 3bigscience/mt0-xl 4facebook/xglm-7.5B 5textdetox/multilingual_paradetox 6https://huggingface.co/datasets/s-nlp/paradetox 7https://huggingface.co/datasets/s-nlp/ru_paradetox Also, since the MPD lacks annotated data—with only 400 examples available for each language—various combinations of similar languages should be considered to augment training data and may lead to improved performance on holdout sets.
Due to the constraints of computational resources for LLMs like mT0-xl and XGLM, we choose not to fine-tune all their weights. Instead, we investigate the potential of utilizing Low Rank Adaptation (LoRA) [15] to facilitate the training of these large models.
During the experiments, prompts are not used, and we rely solely on model convergence without providing additional instructions during fine-tuning and evaluation. Additionally, early stopping [ 16] is employed. All reported results are based on metric values measured automatically on the test set.
We iteratively test our hypotheses without considering all potential scenarios within each experiment iteration, so we do not present results for every experimental setup.
First, we aim to investigate the multilingual performance of various models and consider fine-tuning them on diferent combinations of datasets, including PD and MPD. Table 1 presents the results of these experiments. Notably, utilizing the PD dataset can enhance target performance in most languages after ifne-tuning mBART and mT0-large.
For other models, we decided not to explore diferent combinations of datasets due to the large size and complexity of experiments. Thus, for them we only included results for PD and MPD together. As we can see, mT0-large achieves the maximum absolute values in target performance in five languages; mT0-xl+LoRA is the best for one language only, and XGLM+LoRA performs the best in three languages.
In Section 4.1, we found that mT0-large and XGLM+LoRA showed the most promising performance. However, conducting experiments with LLMs like XGLM requires significant computational resources and time. Since our focus in this section is to explore fine-tuning across various language combinations, we choose to conduct experiments here with mT0-large only due to the trade-of between its size and multilingual performance.
In Table 2, we present the results obtained from fine-tuning using diferent combinations of languages. Specifically, we observe that fine-tuning the model in Russian or Ukrainian separately yields poorer performance than fine-tuning their combination. Similar patterns were observed in experiments with Hindi and Amharic, where training on their combination resulted in the best performance. However, when it comes to German, English, Spanish, and Arabic, fine-tuning using their combination shows worse results than fine-tuning them separately. Nevertheless, English and Spanish still exhibit similar improvement patterns when combined.
As a result, we can approve our hypothesis that closely related languages can serve to improve results while training on their combination, enriching the training dataset.
Based on the reported results, our final submission combines the top-performing outputs from the mt0-large model for English, German, Ukrainian, Russian, and Amharic with those from XGLM+LoRA for Spanish, Arabic, and Hindi. Although we could not exceed the performance of the delete baseline for Chinese, we have replicated its results and included them in our final submission.
Afterward, we chose to preprocess these combined results by excluding words from a multilingual toxic lexicon dataset8 provided in the competition. Table 3 illustrates a comparison of the results before and after excluding toxic lexicon words at this stage. As we can see, the removal of such words positively impacts almost all languages, though it did not afect the results for Chinese and German.
As mentioned earlier, our top submission secured fourth place in the automatic test evaluation, yet it reached third place in the manual evaluation through human annotation (refer to Table 4). Notably, according to human evaluation, our results for such languages as Spanish, Hindi, and Arabic are the top ones, indicating that decoder-only LLMs are more efective at handling the text detoxification task and generating more human-like text.
8textdetox/multilingual_toxic_lexicon This study explored fine-tuning various models with diferent architectures for the task of text detoxification. Our experiments also investigated the use of varied combinations of datasets and languages during fine-tuning. By combining diferent approaches, we achieved fourth place in test evaluation and third place in human evaluation. In this work, we particularly demonstrated that cross-lingual transfer between languages is a promising approach, improving languages such as Ukrainian and Amharic by transferring knowledge from closely related languages such as Russian and Indian respectively. We also showed that training decoder-only LLMs can be a promising direction, yielding the best results according to human evaluation, which totally aligns with the latest advancements in the NLP sphere. M. T. Diab, V. Stoyanov, X. Li, Few-shot learning with multilingual language models, ArXiv abs/2112.10668 (2021). URL: https://api.semanticscholar.org/CorpusID:260651613. [8] D. Dale, A. Voronov, D. Dementieva, V. Logacheva, O. Kozlova, N. Semenov, A. Panchenko, Text detoxification using large pre-trained neural models, ArXiv abs/2109.08914 (2021). URL: https://api.semanticscholar.org/CorpusID:237572304. [9] L. Laugier, J. Pavlopoulos, J. S. Sorensen, L. Dixon, Civil rephrases of toxic texts with selfsupervised transformers, ArXiv abs/2102.05456 (2021). URL: https://api.semanticscholar.org/ CorpusID:231861515. [10] V. Logacheva, D. Dementieva, I. Krotova, A. Fenogenova, I. Nikishina, T. Shavrina, A. Panchenko, A study on manual and automatic evaluation for text style transfer: The case of detoxification, Proceedings of the 2nd Workshop on Human Evaluation of NLP Systems (HumEval) (2022). URL: https://api.semanticscholar.org/CorpusID:248780050. [11] V. Logacheva, D. Dementieva, S. Ustyantsev, D. Moskovskiy, D. Dale, I. V. Krotova, N. Semenov, A. Panchenko, Paradetox: Detoxification with parallel data, in: Annual Meeting of the Association for Computational Linguistics, 2022. URL: https://api.semanticscholar.org/CorpusID:248780527. [12] D. Dementieva, D. Moskovskiy, D. Dale, A. Panchenko, Exploring methods for cross-lingual text style transfer: The case of text detoxification, in: International Joint Conference on Natural Language Processing, 2023. URL: https://api.semanticscholar.org/CorpusID:265445167. [13] H. Naveed, A. U. Khan, S. Qiu, M. Saqib, S. Anwar, M. Usman, N. Barnes, A. S. Mian, A comprehensive overview of large language models, ArXiv abs/2307.06435 (2023). URL: https: //api.semanticscholar.org/CorpusID:259847443. [14] L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, C. Rafel, mt5: A massively multilingual pre-trained text-to-text transformer, in: North American Chapter of the Association for Computational Linguistics, 2020. URL: https://api.semanticscholar.org/CorpusID: 225040574. [15] J. E. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, W. Chen, Lora: Low-rank adaptation of large language models, ArXiv abs/2106.09685 (2021). URL: https://api.semanticscholar.org/CorpusID: 235458009. [16] J. Dodge, G. Ilharco, R. Schwartz, A. Farhadi, H. Hajishirzi, N. A. Smith, Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping, ArXiv abs/2002.06305 (2020). URL: https://api.semanticscholar.org/CorpusID:211132951.