Despite diferent countries and social platform regulations, digital abusive speech persists as a significant challenge. One of the way to tackle abusive, or more specifically, toxic language can be automatic text detoxification-a text style transfer task (TST) of changing register of text from toxic to more non-toxic. Thus, in this shared task, we aim to obtain text detoxification models for 9 languages: English, Spanish, German, Chinese, Arabic, Hindi, Ukrainian, Russian, and Amharic. This paper presents the Multilingual Text Detoxification (TextDetox) task, the underlying datasets, the evaluation setups, the submissions from participants, and the results obtained. Warning: This paper contains rude texts that only serve as illustrative examples.
Models (LLMs) [
Within the baselines approaches for automatic text detoxification, multiple unsupervised baselines
were created based on ideas of Delete-Retrieve-Generate [
Toxic Input
What a f**k is this about? А н**рена ты здесь это писал? Та н**уй ти мені впав, скотина ти така)))
Was für ein besch**senes Jahr
Este país se va a la m**rda ط**ارﺷ ﺎﯾ ﮫﺗزﺎﻧﺟﺑ اوﺷﻣﺗو لﯾﺗﻘﻟا اوﻠﺗﻘﺗ
卧槽，抓到了！ ये माद**द डरे हुए लग रहे है ? Detoxified Output
What is this about? А зачем ты здесь это писал? Та навіщо ти мені потрібен
Was für ein schlechtes Jahr.
Cosas van muy mal en este país ﮫﺗزﺎﻧﺟﺑ اوﺷﻣﺗو لﯾﺗﻘﻟا اوﻠﺗﻘﺗ
天啊，抓到了！ ये लोग डरे हुए लग रहे है ? አንተ ቆሻሻ በዚህ ወቅት አይንህን ማየት አልፈልግም
አንተ ጥሩ ሰው አይደለህም በዚህ ወቅት አንተን ማየት አልፈልግም
In this shared task, we explored both setups—cross-lingual and multilingual one—providing new
multilingual parallel text detoxification dataset for 9 languages [
All the resources produced from the task are listed at the shared task page 1 and are also mentioned in the corresponding sections.
Here, we provide the shared task main definitions—how we understand toxicity, text style transfer task, cross-lingual and multilingual setup, and the competition rules.
Definition of Toxicity While there can be diferent types of toxic language in conversations [ 17, 18],
i.e. sarcasm, hate speech, threats, in this work, we include samples with substrings that are commonly
referred to as vulgar or profane language [
Text Detoxification as Text Style Transfer In this work, we adapt the formal task definition of the
text style transfer described in [
Having a set of style and a corpus of texts , a text style transfer (TST) model is a function : × × → that, given a source style , a target style , and an input text , produces an output text such that: • The style of the text changes from the source style to the target style and is measured by a style classifier: () ̸= (), () = ; • The content of the source text is saved in the target text as much as required for the task and estimated by a content similarity function: (, ) ≥ ; • The fluency of the target text achieves the required level according to the fluency estimator: () ≥ , where and are the threshold values for the content preservation ( ) and fluency ( ) functions and can be adjusted to the specific task. In our task, the source style is toxic and the target style is non-toxic.
Cross-lingual Text Detoxification As parallel text detoxification corpora might not be available for any language, one of the important tasks is to explore cross-lingual text detoxification knowledge transfer. In this case, we assume that training data is available for the resource-rich language (i.e. English) and the task is to obtain a text detoxification system for a new language.
Multilingual Text Detoxification If parallel corpora available for multiple language, then both monolingual text detoxification models per language and multilingual model for all languages can be obtained.
The share task timeline was divided in to two phases—development and test.
Development Phase For the first phase, only training parallel data for English and Russian from
previous works [
For each of our 9 target languages, we prepared parallel text detoxification corpus. We asked experts and native speakers to contribute for corpora collection. Further, we describe the collection details per each language: English (Section 3.1), Russian (Section 3.2), Ukrainian (Section 3.3), Spanish (Section 3.4), German (Section 3.5), Hindi (Section 3.6), Amharic (Section 3.7), Arabic (Section 3.8), Chinese (Section 3.9). All the instructions per language are available online.3 We also opensource the obtained resources for the public usage.4
For all the data, we adapt the concept of English ParaDetox [26] collection pipeline as it was designed to automate the data collection as well as verification with crowdsourcing. The pipeline consists of three tasks: Task 1: Rewrite text in a polite way Annotators need to provide the detoxified paraphrase of the text so it becomes non-toxic and the main content is saved or to skip paraphrasing if the text is not possible to rewrite in non-toxic way; Task 2: Do these sentences mean the same? Check if the content is indeed the same between the original toxic text and its potential non-toxic paraphrase; Task 3: Is this text ofensive?
In the same manner, each language stakeholder asked the annotators to rewrite the toxic samples verifying the main three criteria: (i) the new paraphrase should be non-toxic; (ii) the content should be saved as much as possible; (iii) the resulted text should be fluent but may contain some minor mistakes (as the majority of the original toxic samples are examples from posts from social networks). The rewriting of the texts and verification of their quality could have been done either via crowdsourcing or via manual annotation. The main goal for each language was to obtain 1000 parallel pairs that were later splitted into dev and test sets.
Data Preprocessing For all languages, we maintain the length of samples as sentences of around 5-20 tokens. Also, if a text sample is from a social network, we anonymize any mentioning of usernames and links.
For English, we reused the data from English ParaDetox dataset [
The same as for English, there were previously available training and validation data from previous work [22, 28]. We reused this data and manually annotated some additional toxic examples taken from various toxicity datasets.
Input Toxicity Data The original toxic samples were taken from two binary toxicity classification Kaggle Toxic Comments datasets [29, 30]. 3.2.1. Annotation Process The training and validation sets of RuParaDetox were acquired through crowdsourcing via Toloka platform with fluent Russian speakers. Additionally, we employed annotators who are native in Russian and hold a Masters degree in Computer Science to compile additional samples to the test set.
We used the data presented in MultiParaDetox paper [28] providing the main details of data collection: Input Toxicity Data For the Ukrainian language, there was no existing binary toxicity classification corpus. Therefore, we filtered explicitly toxic samples containing obscene lexicon from the predefined list [31] within the Ukrainian Tweets Corpus [32]. 3.3.1. Annotation Process We adapt ParaDetox [26] collection pipeline and verified the data quality via crowdsourcing. We utilized the Toloka platform for crowdsourcing tasks in Ukrainian. The annotators, who were native Ukrainian speakers, underwent an examination before starting the tasks.
We used the data presented in MultiParaDetox paper [28] providing the main details of data collection: Input Toxicity Data For Spanish, we selected samples for annotation from three datasets: hate speech detection ones [33, 34] as well as filtered by keywords Spanish Tweets corpus [35]. 3.4.1. Annotation Process We adapt ParaDetox [26] collection pipeline and verified the data quality via crowdsourcing. We utilized the Toloka platform for crowdsourcing tasks in Ukrainian. The annotators, who were native Spanish speakers, underwent an examination before starting the tasks.
German ParaDetox was collected with several annotators with manual quality verification: 3.5.1. Input Data Preparation The German language source data in this work is based on three datasets containing toxic, ofensive, or hate speech comments on social media about primarily political events in Germany or the US. For the two datasets from the GermEval 2018 [36] and GermEval 2021 [37] shared tasks, we used data from both the test and the train split and filtered it as follows. For the GermEval 2018 data, we only used samples labeled with the coarse class “OFFENSE” whereas for the GermEval 2021 data – which contains diferent labels – we only used samples annotated with the “ Sub1_Toxic” class. The third dataset [38] was filtered so only samples were kept where both expert annotators classified the samples as hate speech.
The data from the three datasets was merged and deduplicated via exact string matching. Furthermore, we removed all samples that included less than 5 or more than 30 white-space separated tokens. 3.5.2. Annotation Process To create the final parallel detoxified German dataset, we hired two native German annotators. Annotator A is a female born in 1994 who holds a Master of Arts degree in Social Sciences, and Annotator B is a male born in 1992 who holds a Master of Science degree in Computer Science. The data was distributed so that each sample was transcribed by only one of the annotators. 3.6. Hindi Hindi dataset was collected manually by a native-speaker annotator gaining data from multiple sources: 3.6.1. Input Data Preparation Input Toxicity Data We used the HASOC dataset created at FIRE 2019 [39] as source for Hindi language. Contents in this dataset are relevant within Indian subcontinent which are collected from various social media platforms prevalent in India. In this dataset, hostile posts are divided into HATE SPEECH, OFFENSIVE and PROFANE. For curation, posts containing OFFENSIVE and PROFANE contents in train and test splits were used. 1455 PROFANE posts (1237 train + 218 test) and 873 OFFENSIVE posts (676 train + 197 test) were chosen to prepare detoxifiable toxic data for our task.
Input Preprocessing On a total of 2328 samples, we first performed deduplication via exact string matching. Mentions, links and emojis were also removed as part of this step. 3.6.2. Annotation Process Annotation Task(s) The posts after input preprocessing were manually verified. Those with less than 5 white-space separated tokens were removed and which had more than 25 white-space separated tokens were re-framed to bring them down to this limit. Toxicity and meaning of the posts were unchanged during this re-framing. These posts were then bifurcated into detoxifiable and non-detoxifiable labels. The manual re-framing and bifurcation were carried by a NLP researcher with working experience on hate/toxic speech.
Out of 2328 samples, 1007 samples were marked as detoxifiable. From these detoxifiable samples, we carefully sampled 24 data points and detoxified them. These detoxified samples were evaluated by two experts who are native Hindi language speakers to provide precise samples to the annotators for detoxifying the whole dataset. Annotators were guided based on expert prepared samples and were asked to re-write toxic pairs in a non-toxic manner, keeping the meaning of the original post unchanged. Detoxification was carried out by two annotators and we provide their details in the corresponding subsection.
Annotators One male NLP researcher working in the field of hate/toxic speech and another female student enrolled in Bachelor’s Degree and having working knowledge in Machine Learning, were employed to carry out the detoxification of whole dataset. Both annotators are Indian, native Hindi speakers and are well versed with the topicality covered in the dataset.
We compiled new Amharic ParaDetox datasets with the following annotation details, based on prior
studies of hate and ofensive language:
3.7.1. Input Data Preparation
The Amharic ParaDetox dataset is derived from merging two pre-existing studies conducted on the
X/Twitter datasets [
Input Toxicity Data The input toxicity data is entirely sourced from the two previous studies, namely
Ayele et al. [
We conducted pilot annotation of 125 sample items with three native Amharic speaker annotators and evaluated the annotation quality with experts and annotators together in a group meeting to improve the understandings of annotators for the main task. The main annotation task comprises of 2,995 tweets, each annotated by one annotator. Annotators were asked to classify each tweet in to two broad categories, detoxifiable and non-detoxifiable. For the detoxifiable category, annotators are asked to detoxify and re-write the text. For non-detoxifiable tweets, annotators choose non-detoxifiable and select reason as; it is hate speech, it is normal speech or indeterminate to decide the label. Annotators Annotators have previous hate speech annotation experiences and already holds Masters degree in Computer Science. Only two of the annotators were evolved in the main annotation task, where both of them are university lecturers and have basic knowledge of natural language processing tasks. One of the annotators is from Adama Scinece and Technology University with experience of 15 years of teaching Computer Science, who is female. The other annotator is a male, who has been teaching Computer science over 12 years in Kotebe University of Education.
3.8.1. Input Data Preparation Arabic ParaDetox was collected with several annotators with manual quality verification: The Arabic ParaDetox dataset was created by combining parts of several existing datasets along with the Arabic-translated version of the Jigsaw dataset [27]. It includes the Levantine Twitter Dataset for Hate Speech and Abusive Language (L-HSAB) [41], which focuses on Levantine dialects, and the Tunisian Hate and Abusive Speech (T-HSAB) dataset [42], which targets Tunisian dialects. It also incorporates the OSACT dataset [43] and the Arabic Levantine Twitter Dataset for Misogynistic Language (LeT-Mi) [44], which specifically addresses gender-based abuse. These resources combine to form the Arabic ParaDetox dataset, aimed at aiding the development of toxicity classifiers capable of handling Arabic content across various dialects and contexts. 3.8.2. Annotation Process Annotators The detoxification process was conducted by three annotators, each with a PhD. The team includes two males and one female, all of whom have a strong interest in computational linguistics. These native Arabic speakers possess a deep understanding of the subjects encompassed within the dataset. Each text sample was transcribed by two of the annotators to ensure accuracy and consistency in the data.
3.9.1. Input Data Preparation We collected new Chinese ParaDetox datasets with the following annotation details: Input Toxicity Data The Chinese ParaDetox dataset is derived from TOXICN [45], a recently released Chinese toxic language dataset. TOXICN was compiled from social media platforms and comprises 12,011 comments addressing several sensitive topics, including gender, race, region, and LGBTQ issues. From this dataset, we extracted a subset based on multiple criteria: the number of toxic words, the ratio of toxic words in the comments, the length of comments, and the toxic scores of comments. Input Preprocessing We set thresholds for the criteria mentioned above: the number of toxic words ranged from 1 to 5, the ratio of toxic words in comments was less than 0.5, and the length of comments ranged from 3 to 50 words, ensuring suitability for annotators to rewrite them. Following these criteria, we extracted 1,516 samples from the training set and 231 samples from the test set.
Subsequently, we employed a pre-trained toxic classifier [ 45] to compute the toxic scores of the selected comments, using a threshold score of 0.978 to filter the candidates. Ultimately, we collected 1,149 samples from the training set and 231 samples from the test set, resulting in a total of 1,380 samples deemed suitable for annotation. 3.9.2. Annotation Process Annotation Tasks For data annotation and verification, we employed a specifically designed threetask pipeline: Task 1: Determine if the sentences are toxic or neutral. Annotators were required to choose one of three options: the given sentence is neutral, toxic but can be rewritten, or toxic and cannot be rewritten. The last option was included based on the observation that some toxic texts are impossible to rewrite in a non-toxic manner.
Task 2: Rewrite sentences in a non-toxic style. Annotators were instructed to create detoxified versions of the toxic sentences identified in Task 1. They were advised to retain the main content of the original sentences and rewrite the toxic words in a polite manner.
Task 3: Cross-check verification. The rewritten sentences from Task 2 were cross-distributed to diferent annotators for verification. The goal was to ensure the rewritten sentences were nontoxic and adhered to our guidelines. If annotators selected the “No” option, indicating the sentence did not meet the criteria, a further meta-rewrite process was initiated.
From the 1,380 toxic samples, 1,031 samples were successfully detoxified and verified, with 861 from the training set and 170 from the test set. The remaining 349 samples were either considered non-toxic or toxic but could not be rewritten.
Annotators For the detoxification process, we hired three native Chinese annotators. Two female annotators, both aged 22, hold Bachelor’s degrees in Engineering, and a male annotator, aged 32, holds a Master’s degree in Computer Science. All annotators are native Chinese speakers residing in mainland China, ensuring they deeply understand the Chinese language and the detoxification task.
The full picture of the collected ParaDetox data for all target languages is presented in Table 1. While
the methods of collecting human annotations vary across languages—some data were gathered via
crowdsourcing, others by hiring local native speakers—the quality of the texts was uniformly verified
by experts to ensure three key attributes as introduced in [
For each language for the shared task’s phases: • During the development phase: 400 only toxic parts were available for participants to perform cross-lingual experiments. • During the test phase: (i) 400 ParaDetox instances were fully released; (ii) participants should provide their final solutions for 600 toxic parts of the test dataset.
For English and Russian during all phases, additional training parallel datasets were available released
from previous work [
We provide four baselines for our shared task: (i) trivial Duplicate baseline; (ii) a rule-based Delete approach; (iii) Backtranslation pipeline that reduces the task to the monolingual one; (iv) finally, finetuned for the downstream task on the dev dataset mT5 instance. The code for all the baselines is available online.9
Duplicate Trivial baseline: the output sentence is a copy-paste of the input sentence. This baseline has 1.0 (or 100%) SIM score by definition.
Delete For the first unsupervised baseline, we perform an elimination of obscene and toxic substrings
from a text according to the predefined lists of keywords. For the shared task, we collected and compiled
together the lists of such toxic keywords for all target languages based on openly available sources
(see Table 2). The amount of toxic keywords per language difers which displays the diversity of
morphological forms and variations of toxicity expressions across languages. For participants and
further public usage, we release our compiled list online.10
Backtranslation As for a more sophisticated unsupervised baseline, we perform translation of
nonEnglish texts in English with NLLB [19] instance11 and then perform detoxification with the fine-tuned
on English ParaDetox train part BART [
We adopt the monolingual evaluation pipelines from [
Content Similarity (SIM) is the cosine similarity between LaBSE17 embeddings [53] of the source texts and the generated texts.
Fluency (ChrF1) is used to estimate the proximity of the detoxified texts to human references. While
in several previous work language acceptability classifiers based on CoLa-like corpora were utilized for
lfuency estimation [
Joint score (J) is the aggregation of the three above metrics. The metrics STA, SIM and ChrF1 are subsequently combined into the final J score used for the final ranking of approaches. Given an input toxic text and its output detoxified version , for a test set of samples:
J = 1 ∑=︀1 STA() · SIM(, ) · ChrF1(, ),
where STA(), SIM(, ), ChrF1(, ) ∈ [
For the test set, we performed the human evaluation to obtain final judgements on the participants’ systems. The details and instructions of the annotation setups are available for the public usage.18
We used Toloka19 crowdsourcing platform for manual evaluation of automatic detoxification. For each project, the annotators who indicated the knowledge of the necessary language could access the tasks. Quality Control First, the annotators underwent a rigorous selection process, starting with a training pool of 20 tasks with predefined answers and instructional feedback. Following this, annotators faced an examination pool comprising 10 tasks with known answers. Only those achieving a minimum accuracy of 65% in both the training and examination pools were permitted to proceed to the main evaluation tasks.
During the execution of the real tasks, annotators’ performance was meticulously monitored using several measures. Response speed was a critical factor; annotators who responded rapidly (less than 15 seconds per page) to three consecutive task pages were permanently banned. Additionally, we evaluated the answers of the annotators to mixed tasks with known true answers. Those who demonstrated low performance, were defined as achieving less than 65% accuracy, were required to undergo re-examination. Furthermore, annotators who skipped three task pages were also banned from the study. Evaluation Dataset We selected 100 random original toxic samples per each language from the test part of our dataset. Each pair was annotated within a separate project dedicated to the respective language. These projects were categorized into three distinct types, which will be described in detail in the following section. 16https://huggingface.co/textdetox/xlmr-large-toxicity-classifier 17huggingface.co/sentence-transformers/LaBSE 18https://github.com/textdetox/textdetox_clef_2024/tree/main/instructions/human_evaluation 19https://toloka.ai 17 1 Text 1 Interested Tolokers 2 Text 2 3
None
In5g/e3n1e/r2al4,,th1e1c:o1n7cePpMt of the human evaluation mirrored the approach used inTothloekaau:toDmaattaicseovlaultuiaotniotno. drive AI Each project type focused on assessing one of the three key qualities of detoxification; style transfer accuracTy,rcyonoteuntt stimasilakristy,aPorsroflujaeecntTcsyo. lBUoeslkoewers,rwaeSnkpirldlosvrideePvraoidfeielewtailreMedesdsesusaclgrteispstionDoofcsea&chRepsrooujercctetsype and its Lspoeockificfofropcoussswibiltehpinrotbhlee mevsatlhuaattiToonlopkreorscemssig.ht run into and fix them for best results. Check the instructions and th Projects Which texdteisvimceorsewofhfeennsicvoe?m-pgloeltdinengptaaisrsks
popular scWrheiechn tseixzteis.8m3o%re oofffeunsseivres?use mobile Style Transfer Accuracy To measure style transfer accuracy, we employed a pairwise comparison betweeGnentehreal otarsigksinal toxic text and the generated detoxified text. Annotators were tasked with determining which text was more toxic: the left text, the right text, or neither. An illustration of this task can be WfouhndiicnFhigutree2.xt is more offensive? - golden pairs To ePnuhbalniccdeersecarliipsmtio,nw: Ienrdaicnadtoemwihziecdhttheextsiesquence of original and detoxified texts. Annotators’ votes more offensive were tPhreivnatceocnovmermteednitn: ttooxincu_pmaeirrwicisael values using the following logic: if the original toxic text was deemedWmhicohretetxotxisicm,tohree ovfofetenswivea?s recorded as 1; if the detoxified text was considered more toxic, the vote was recorded as 0; if “none of the texts” was selected, the vote was recorded as 0.5.
Which of the texts is more offensive?
Labeling completed! 1)
StaIrtadmateN: Manaycy10a,2n0d24Il9ik:3e4tAoMruin people's fun.
Accepted 100% (38 of 38) 2) I am Nancy and I like to fuck up people's fun.
17 Tolokers submitted task 0 Skipped task suites 0 Expir Overview Efficiency Statistics Settings
Figure 2: The English interface example of crowdsourcing Task 1: pairwise toxicity comparison.
Which of the texts is more offensive?
Some issues need your attention Content Sim1C)ihlaecrkitoyur reFcoormtmheencdoantiotensnttosiimmpirloavreiteyffimcieentcryic, we presented pairs of texts in a fixed order (the toxic phrase follotgwheiesttdisnabgymkteichkmeedidsettabokyxegifieardegpaphirnarac.stiec)eatnedstasskmeaddaensnuorteattohrast tionienvdeircamteakwehether the sense of the two texts was similar. Annotators’ responses were binary, with “yes” indicating that the sense was similar and “no” indicating otherwise. The responses were mapped to numerical values, with a “yes” respons2e)beTiansgkgrseecttoinrdgemdyasa1ssankidckae"dnob"yrgersepopnrascetibceeintegsrtescmoraddeed sausr0e. tAhanAt iuilldnueisevtrenarctieon of this task can be found in Figurmea3k. e this same mistake again .
Speed/quality balance Top 90% of Tolokers wh Fluency F3o8rthe fluency metric, sente10nc0es were evaluated individually rLaathnegrutahgaensin pairs. AnnoEtantgolrissh (exam-based) were asked Twashke1stuhieterseTeacxht1text was inGteelnliegraibltlaesaksnd correct, with three possible responses: yes, partially, and no. Each response was mapped to a numeric value: “yes” was scoredWahsic1h, “tpeaxrttiisamllyo”reas 0.5, agnrde“anteor”than 75 as 0. The fluency scoreTefoxrt 2a pair of texts was determined by comparing tohfefesncsoirvees?o-fetxhaemdetoxified text 2 to the origin1a0l 0t3ext. IfNtohneedetoxified tex4t5received a higher or equal fluencyWshcoicrhe,tethxet ipsamirowreas scoreMdisassing 1; otherwiseC,ointtwrolatsasskcsored as 0. An iTllrauisntinragttiaosnksof this task can be founodffeinnsFiivgeu?r-ew4o.rk greater than or equal to
Skip Joint Score The final scores for each pair within each project were calculated by weighting them according to the skill of the crowdsourcing annotators. This skill was dynamically assigned based on the quality of their responses to both the tasks with known answers and theQreuaalltiatyskcso.Tnhtreollogic for calculating the JPoirnictescaornedinotvheerlmaapnual evaluation mirrors that of the automatic evaluation. For each Price per task suite $0.05 Overlap 3 Tolokers Price per 1 item $0.026 Fast responses
Minimum time per task suite — 2 Public description: Check if the meanings of the two sentences are close or not.
Private comment: content 5/31/D2o4th,e1se1s:e3n6tenPceMs mean the same? Toloka: Data solution TryDo otheusettwtoasesntkenscesaPmrseoajnaetchetTssoamleUotshkienger?sr aSnkildls rePvroifeilew rMeesssualgtess
Labeling completed! Docs & Resources Look for pSotsarstdibatlee: Mparyo2b0l,e2m02s4 3t:h0a7PtMTolokers might run into and fix them for best results. Check the
Dear Greenman, As usual you have proven to be an arse hole popular screen size. 83% of users use mobile devices when completing tasks Projects Is this sentence grammatical? Is this sentence grammatical? - golden pairs Is t17his sentence16grammatical0? - golden pa3irs Public description: Check if the sentence is understandable and grammatically correct.
Inter1ested TYoeloskers2
No
Tolokers submitted task
Skipped task suites
Expired Private comment: fluency Is this sentence grammatical? No way number bricks fit in a car
Lsatilblsetalnindigngcuop mforpyoleurtelyidng! crazy leader who loves himself and Sntaorbtoddaytee:lsMe .
ay 14, 2024 10:04 AM Is the sentence intelligible and correct? y AccepteYdESst,iltlh0deerefeanrdeinngoothfmeiisrtalykinegs aonrdmsineolfrismhisletaakdeesr.(punctuation, casing)
10 % (39 39) p39 Task1suites n
PARTIALLY, mistak1e0s0do not hamper understanding the text
Languages
No General tasks
Yes 2 NO, mistakes make it difficult to understand the text Client
Audience Figu2rIne7t4e8:reTshteeEdnTgolilsohkeinrtserface1e2x0ample of crowdTsooluorkceinrsg sTuasbkm3i:ttfemludeeantancsyk theevaslaumateio?n-.
Training tasks 11
Do these sentences WORK Speed/quality balance
Top 90% of Tolokers who English (exam-based) Toloka for mobile, Toloka M0issing gSrkeipatpeerdthtaansk65su; ites
DGo othtehsse ttweonsdenttoenlcisestemneatnothGeostahmiec trhoincgk?, rather than bDaobtyhepsoeps.enIfteynocuesdesirgere,ayteoruthcaann65 accuracy, content simHilaerditeys,earnvde sflueenvceyry) ubseiantgingthaensdamraepefohremwulilal baes isnuEbthXjeeAcMateudtotom. atic evaluation. language, thleoJcoaintet sscoourercweass vdeetreifryminingedthbisy imnfuoltrimplayitniognt.he three imndeiavni dthuealsasmcoer?es- (style transfer
OvSekiprviewPriceSatantdisotivcesrlap Settings
y Is the sentence intelligible and correct?
YES, there are no mistakes or minor mistakes (punctuation, casing)
Price per task suite $0.03 We received 20 submissions for the development phase leaderboard and 31 submQiussailoitnys cfoornthtreotlest phase p PARTIALLY, mistakes do not hamper understanding the text leaderboarOdv;etrhlaepfina3l Tmoalonkuearslly evaluated leaderboard was based on 17 submissions who confirmed Each teamPisricperepseernT1t aeitdesmwkisth$0i.t0s1r6espective leaderboard name (in some cases, additionally, by aAusuedrniaemnece their participnationNinOt,hmeisctoamkepsetmitaiokne.itHdeifrfeic, uwltetobruinefldyedresstacnridbethtehteesxotluEtaironninsgosf our final paIrfteicairpneadntins.last 24 hours > 1 , t of the corresponding team member that did a submission) and approaSckhipupseedd in bracketsI:f task suites skipped in a row ≥ 1 in target langpCuoangterosP.lAtRasTkIAsLLY, mistakes doTnraoitnhinagmtpaesrkusnderstanding the text Submitted Team cake, Submission d1n910 (few-shot Kimi.AI) [55] The participants achieved the resulting
S3a9me situation is going o1n0in0the pnw too. score with a few-short LLM inference by using a two stage process: Cfirsotn,t4ro0l0tassakms pLlaesngfruoRamecgeEenNtscoanntrdol and training tEasnk greli
Task suites General tasks RU provided datasets were used to be detoxified by a proprietary LLM—Kimi.AI [ 56] which is a large
Is the sentence intelligible and correct? Is thisIf sneumntbeernocf etask responsesgr≥e a1 language model chatbot developed by Moonshot AI, a Beijing-based startup. In the second step, the grammatical? - exam participantsyemploYyEeSd,nthewerleyadreetnooximfiedisstaakmepsleosr mtoincoornmstirsutackteasp(rpoumncpttuawtihoenr,ecathseinygw)ere iDnocltuhedseedsaenstences mean the sa examples of t6he0desired behavior and the m6o0del Kimi.AI, thus, was prompted to perfIsortmhidsestoexnifictaetniocne Miss grammIfsautbimcaittle?d-tawskosrukites ≥ 20 , t grea sentencSeskiwp as detPoxriifiecdethaenmdosotavpeprrolaprpiate way.
n
SINAI employed the Tree-of-Thought prompting strategy based on thFeaOstpreenspAoIn’ssemsodel GMiPnTim-u3m.5t[im58e]p.er task suite — 15 Given a toxic sentence, the model was prompted to output three options of potential detoxified sentences. Then the model was asked to decide in terms of ofensiveness, content, and fluency which oneIfoQnuutmutbhaeerlsioetfyfast task suite respon control responses assignments Speed/quality balance
Top Price per task suite $0.044
Fast responses Optional settings
Minimum ti If number Team MarSanAI, Submission maryam.najafi (Mistral-7b with PPO) [59] This team ofered a solution only for two languages: English and Russian. A reinforcement learning method was utilized to fine-tune an LLM—Mistral-7b [ 60]—coupled with a Proximal Policy Optimization (PPO) [61] using the implementation from HuggingFace TRL [62]; the reward was obtained using the provided toxicity classifier.
Team Linguistic_Hygienist, Submission gangopsa (T5 & BART) [63] The solution consisted of two components: i) the supervised solution for the English and Russian languages; ii) the unsupervised solution for the other 7 languages. The supervised solution used T5 [64] and BART [65] as base models; the exponentially weighted moving average and ROUGE scores were used as loss functions for Russian and English, respectively. The unsupervised solution utilized hashing techniques, log odds ratio, and grammatical rules to identify and conceal toxic words across other 7 languages; additionally, it incorporated a mask prediction model to maintain the original sentences meaning intact.
Team VitalyProtasov (mT0-large) [66] In the proposed solution, the team used a text-to-text model—mT0-large [67]—which was trained on diferent combinations of languages. In addition, before training, certain filtering techniques were applied to the data.
Team nikita.sushko (mT0-XL) [68] The participant used the text-to-text mT0-XL [67] model that was fine-tuned in two stages. In the first stage, a model was fine-tuned on the parallel data of all languages; this model was used to generate synthetic parallel data from non-parallel samples. The resulting data was cleaned and filtered using a cosine distance between LaBSE embeddings and the toxicity scores by the provided classification models followed by a modification with improved delete approach. At the end, the synthetic and filtered “golden” data were merged into new training set to ifne-tune a new instance of the text-to-text multilingual model.
Team SmurfCat, Submission adugeen (mT0-XL) [69] Multilingual model mT0-XL [67] was as well used by this team. First, the model was fine-tuned for text generation using a combination of parallel and translated datasets. The model was further aligned with the Odds Ratio Preference Optimization (ORPO) [70]. During the inference stage, the best candidate generated by the model was chosen by calculating scores from STA and SIM models.
Team gleb.shnshn (zero-shot LLaMa-3) This solution was based on a modern open-source LLM— LlaMa3-70B [71]. The model was prompted using the zero-shot prompting method for the detoxification task.
Team memu_pro_kotow, Submission SomethingAwful (few-shot LLaMa-3 & mT0-XL) [72] In this solution, “uncensored” LLaMa3 [71] was introduced and initialized for every target language except Amharic. Using the recent alignment jailbreaking method by identifying “refusal” directions and subtracting them from model weights [73], they used LLaMa3-70B to get predictions using a few-shot prompting strategy. So, the model received 10 examples of detoxification via starting prompt. For the Amharic language, the text-to-text mT0-XL [67] model was used: the model was fine-tuned on the Amharic parallel dataset.
Team Magnifying_Glass, Submission ZhongyuLuo (Translation & BART-detox, ruT5-detox &
Postprocessing) [74] The team used a combination of diferent methods and models depending on
the language. For the majority of languages, the participant used a text-to-text encoder-decoder NLLB
translation model [19] to translate data from various languages into English. Then, the translated data
was detoxified using the English BART-detox model [
Team NaiveNeuron, Submission erehulka (few-shot LLaMa-3) [77] The team used a text-to-text Llama3 [71] which was prompted using a few-shot method.
Team team0, Submission dkenco (few-shot Cotype-7b) In this case, the team put a stress solely on the English and Russian languages. Two language-specific approaches were used based on Cotype-7b model [78]. For English, there was employed a zero-shot prompting technique where the prompt included brief instructions for the text detoxification task. For the Russian language, the team used a few-shot approach: the system prompt included brief instructions for the task to be performed as well as five randomly picked samples from the parallel development set. During inference, for both languages, there were applied regular expressions intended as filters.
Team NLPunks, Submission bmmikheev (few-shot LlaMa-3) This team used a text-to-text Llama3-70B [71] by with a few-shot prompting method. For English and Russian, the generated output was evaluated manually. For other languages, GPT-3.5 [58] was used to evaluate outputs. For all languages, the system prompt was formulated in English.
Team Iron Autobots, Submission razvor (few-shot LlaMa-3) The participant as well used a text-to-text Llama3-70b [71] with a few-shot prompting method.
Team MBZUAI-UnbabelDetox, Submission mkrisnai (few-shot GPT-3.5) In this team, a twostep prompting approach was utilized. At the first step, GPT-3.5 [ 58] was prompted with a few-shot method to produce synthetic detoxification data. Then, the resulting data was employed in the prompt to GPT-3.5 to perform detoxification.
Team Yekaterina29 (mT5-XL) The participant fine-tuned mT5-XL instances [ 51] on the provided development multilingual parallel dataset.
Almost all of the participant used the current state-of-the-art Large Language Models (LLMs), among which are GPT-3.5 [58] and LLaMa-3 [71] models. To enhance the model’s performance on the task of detoxification, most participants used the few-shot prompting method. Among smaller models, mT5 [51] and mT0 [67] were utilized: usually, these models were fine-tuned using ad hoc filtering and data augmentation techniques, for instance, as RAG and backtranslation. Additionally, region-specific LLMs were also employed—Cotype [78] and Kimi.AI [56].
Here, we provide the final results of the final test phase, of our tasks. The full detailed tables of results per each language and per each metric can be found in Appendix A.
We received 20 submissions for the development phase automatic leaderboard and 31 submissions for the test phase automatic leaderboard. Automatic evaluation leaderboards are publicly available online.20 The final leaderboard from the test automatic phase evaluation is presented in Table 3. 20https://codalab.lisn.upsaclay.fr/competitions/18243#results
The leading solutions were consistent across most languages, except for Spanish, Chinese, and Hindi. However, with the automatic evaluation leaderboard publicly available to all participants, some teams focused on optimizing their models specifically for the evaluation metrics, leading to potential overfitting.
Most solutions surpassed the baseline for at least one language, and in some cases, participant systems approached the performance of human references. However, except for Hindi, no participant solution outperformed human references in the automatic evaluation across any language. Although the automatic evaluation scores for human references across most languages hovered around a J score of 0.7, the results for Chinese were notably poor, with the highest participant score being 0.178 and the best human reference score at 0.201. The results leads to a further investigations of the robustness of the automatic evaluation metrics.
The top three teams across the majority of the languages generally employed a similar strategy, ifne-tuning the mT0-XL text-to-text model. Team SmurfCat is holding the best automatic evaluation scores for all the languages, which was achieved by additionally fine-tuning mT0-XL with a recent ORPO alignment method. The majority of the submissions were multilingual, designed to cover all languages within a single model. These models demonstrated consistent score distributions across languages, with notable declines in performance for Chinese and Hindi. An exception was user ansafronov, who achieved the top score specifically for Chinese.
After participants confirmed their submissions via a form, we received 17 entries for the human evaluation phase. This evaluation was conducted on a subsample of 100 test set items through crowdsourcing. The results of the human evaluation, organized by team and language, are publicly available.21 The ifnal leaderboard based on human evaluation is presented in Table 4.
The human evaluation leaderboard saw significant changes compared to the automatic evaluation phase. Human references received high scores from the annotators, with J scores around 0.8 or higher. However, not all teams surpassed the mT5 and Delete baselines. Interestingly, the Delete baseline outperformed the mT5 text-to-text generation baseline in languages such as Arabic, Hindi, Ukrainian, Russian, and Amharic. This indicates that not all multilingual models are equally proficient in understanding and handling toxicity across diferent languages.
In the human evaluation phase, participants’ solutions closely matched the human references, even surpassing the provided references from parallel datasets in some languages. The top solution, after manual evaluation, was presented by user SomethingAwful and was based on the “uncensored” LLaMa3-70B language model. Interestingly, SomethingAwful’s solution did not achieve the highest scores across all nine languages but excelled in Spanish, German, and Russian. The leader of the automatic evaluation leaderboard, Team SmurfCat, secured second place. Participants nikita.sushko and VitalyProtasov switched places in the manual leaderboard.
Similar to the automatic leaderboard, human assessments revealed that certain models excelled in specific languages. For instance, nikita.sushko and VitalyProtasov achieved top results in Arabic and Hindi. Despite Team mkrisnai ranking 7th overall, their solution performed exceptionally well in 21https://github.com/textdetox/textdetox_clef_2024/tree/main/human_evaluation_results English, Spanish, Russian, and Ukrainian. Additionally, Team Team cake secured the highest scores specifically for English and Chinese.
From the detailed results in Appendix A, it is evident that the solutions surpassed human references in English, Spanish, and German, often achieving near-perfect fluency. However, this success does not extend to other languages. These results highlight the impressive human-like text generation capabilities of modern LLMs, though they still struggle with handling toxicity and maintaining consistent controllable generation across languages. Future work should focus on developing more challenging tasks, particularly in cross-lingual contexts.
In Multilingual Text Detoxification task at PAN 2024, participants were tasked with transforming text style from toxic to non-toxic across nine languages: English, Spanish, German, Chinese, Arabic, Hindi, Ukrainian, Russian, and Amharic. The task was divided into two phases: (i) development phase focused on cross-lingual transfer approaches; (ii) test phase utilized parallel training data for all languages and encouraged multilingual solutions. Participants’ submissions in both phases underwent evaluation using a set of automatic metrics, followed by human evaluation of the test subset to determine the final leaderboard rankings.
Participants employed modern state-of-the-art Large Language Models either by prompting them in few-shot formats or fine-tuning medium-sized models. For certain languages with suficient training data, these models approached or even exceeded human reference provided in the shared task. However, this was primarily observed for resource-rich European languages. Opportunities for enhancement remain significant for less resource-rich languages and those with limited data, highlighting the need for further exploration in cross-lingual text detoxification and knowledge transfer.
We express our deepest gratitude to Toloka platform to support our shared task. Crowdsourced data collection and human evaluation were made possible through the provided research grant. icencio (Eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, Association for Computational Linguistics, 2022, pp. 6804–6818. URL: https://doi.org/10.18653/v1/2022.acl-long.469. doi:10.18653/V1/2022.ACL-LONG.469. [27] Jigsaw, Toxic comment classification challenge, https://www.kaggle.com/c/jigsaw-toxic-commentclassification-challenge, 2017. Accessed: 2024-03-18. [28] D. Dementieva, N. Babakov, A. Panchenko, Multiparadetox: Extending text detoxification with parallel data to new languages, arXiv preprint arXiv:2404.02037 (2024). [29] A. Belchikov, Russian language toxic comments, https://www.kaggle.com/blackmoon/russianlanguage-toxic-comments, 2019. Accessed: 2023-12-14. [30] A. Semiletov, Toxic Russian Comments: Labelled comments from the popular Russian social network, https://www.kaggle.com/alexandersemiletov/toxic-russian-comments, 2020. Accessed: 2023-12-14. [31] K. Bobrovnyk, Ukrainian obscene lexicon, https://github.com/saganoren/obscene-ukr, 2019. Accessed: 2023-12-14. [32] K. Bobrovnyk, Automated building and analysis of ukrainian twitter corpus for toxic text detection, in: COLINS 2019. Volume II: Workshop, 2019. URL: https://ena.lpnu.ua:8443/server/api/core/ bitstreams/c4c645c1-f465-4895-98dd-765f862cf186/content. [33] J. C. Pereira-Kohatsu, L. Q. Sánchez, F. Liberatore, M. Camacho-Collados, Detecting and monitoring hate speech in twitter, Sensors 19 (2019) 4654. URL: https://doi.org/10.3390/s19214654. doi:10. 3390/S19214654. [34] M. Taulé, M. Nofre, V. Bargiela, X. Bonet, Newscom-tox: a corpus of comments on news articles annotated for toxicity in spanish, Language Resources and Evaluation (2024) 1–41. [35] J. M. Pérez, D. A. Furman, L. Alonso Alemany, F. M. Luque, RoBERTuito: a pre-trained language model for social media text in Spanish, in: Proceedings of the Thirteenth Language Resources and Evaluation Conference, European Language Resources Association, Marseille, France, 2022, pp. 7235–7243. URL: https://aclanthology.org/2022.lrec-1.785. [36] M. Wiegand, M. Siegel, J. Ruppenhofer, Overview of the GermEval 2018 Shared Task on the
Identification of Ofensive Language (2018). [37] J. Risch, A. Stoll, L. Wilms, M. Wiegand, Overview of the GermEval 2021 Shared Task on the Identification of Toxic, Engaging, and Fact-Claiming Comments, in: Proceedings of the GermEval 2021 Shared Task on the Identification of Toxic, Engaging, and Fact-Claiming Comments, Duesseldorf, Germany, 2021, pp. 1–12. [38] B. Ross, M. Rist, G. Carbonell, B. Cabrera, N. Kurowsky, M. Wojatzki, Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis, in: M. Beißwenger, M. Wojatzki, T. Zesch (Eds.), Proceedings of NLP4CMC III: 3rd Workshop on Natural Language Processing for Computer-Mediated Communication, volume 17 of Bochumer Linguistische Arbeitsberichte, Bochum, Germany, 2016, pp. 6–.9. [39] T. Mandl, S. Modha, P. Majumder, D. Patel, M. Dave, C. Mandlia, A. Patel, Overview of the hasoc track at fire 2019: Hate speech and ofensive content identification in indo-european languages, in: Proceedings of the 11th Annual Meeting of the Forum for Information Retrieval Evaluation, FIRE ’19, Association for Computing Machinery, New York, NY, USA, 2019, p. 14–17.
URL: https://doi.org/10.1145/3368567.3368584. doi:10.1145/3368567.3368584. [40] A. A. Ayele, S. Dinter, T. D. Belay, T. T. Asfaw, S. M. Yimam, C. Biemann, The 5Js in Ethiopia: Amharic hate speech data annotation using Toloka Crowdsourcing Platform, in: Proceedings of the 4th International Conference on Information and Communication Technology for Development for Africa (ICT4DA), Bahir Dar, Ethiopia, 2022, pp. 114–120. URL: https://ieeexplore.ieee.org/ document/9971189. [41] H. Mulki, H. Haddad, C. B. Ali, H. Alshabani, L-hsab: A levantine twitter dataset for hate speech and abusive language, in: Proceedings of the third workshop on abusive language online, 2019, pp. 111–118. [42] H. Haddad, H. Mulki, A. Oueslati, T-hsab: A tunisian hate speech and abusive dataset, in:
International conference on Arabic language processing, Springer, 2019, pp. 251–263. [43] H. Mubarak, K. Darwish, W. Magdy, T. Elsayed, H. Al-Khalifa, Overview of osact4 arabic ofensive language detection shared task, in: Proceedings of the 4th Workshop on open-source arabic corpora and processing tools, with a shared task on ofensive language detection, 2020, pp. 48–52. [44] H. Mulki, B. Ghanem, Let-mi: An Arabic Levantine Twitter dataset for misogynistic language, in: Proceedings of the Sixth Arabic Natural Language Processing Workshop, Association for Computational Linguistics, Kyiv, Ukraine (Virtual), 2021, pp. 154–163. URL: https://aclanthology. org/2021.wanlp-1.16. [45] J. Lu, B. Xu, X. Zhang, C. Min, L. Yang, H. Lin, Facilitating fine-grained detection of Chinese toxic language: Hierarchical taxonomy, resources, and benchmarks, in: A. Rogers, J. Boyd-Graber, N. Okazaki (Eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023, pp. 16235–16250. URL: https://aclanthology.org/2023.acl-long.898. [46] D. Dementieva, S. Ustyantsev, D. Dale, O. Kozlova, N. Semenov, A. Panchenko, V. Logacheva, Crowdsourcing of parallel corpora: the case of style transfer for detoxification, in: D. Ustalov, F. Casati, A. Drutsa, I. Stelmakh, N. Pavlichenko, D. Baidakova (Eds.), Proceedings of the 2nd Crowd Science Workshop: Trust, Ethics, and Excellence in Crowdsourced Data Management at Scale colocated with 47th International Conference on Very Large Data Bases (VLDB 2021), Copenhagen, Denmark, August 20, 2021, volume 2932 of CEUR Workshop Proceedings, CEUR-WS.org, 2021, pp. 35–49. URL: https://ceur-ws.org/Vol-2932/paper2.pdf. [47] R. J. Gabriel, English full list of bad words and top swear words banned by google, https://github.com/cofee-and-fun/google-profanity-words/blob/main/data/en.txt, 2023. Accessed: 2023-12-12. [48] K. Bobrovnyk, The dictionary of ukrainian obscene words, https://github.com/saganoren/obsceneukr, 2019. Accessed: 2023-12-12. [49] I. Shutterstock, List of dirty, naughty, obscene, and otherwise bad words, https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words, 2020.
Accessed: 2023-12-12. [50] A. Jiang, X. Yang, Y. Liu, A. Zubiaga, SWSR: A chinese dataset and lexicon for online sexism detection, Online Soc. Networks Media 27 (2022) 100182. URL: https://doi.org/10.1016/j.osnem. 2021.100182. doi:10.1016/J.OSNEM.2021.100182. [51] L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, C. Rafel, mt5: A massively multilingual pre-trained text-to-text transformer, in: K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tür, I. Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, Y. Zhou (Eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, Association for Computational Linguistics, 2021, pp. 483–498. URL: https://doi.org/10.18653/ v1/2021.naacl-main.41. doi:10.18653/V1/2021.NAACL-MAIN.41. [52] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, V. Stoyanov, Unsupervised cross-lingual representation learning at scale, in: D. Jurafsky, J. Chai, N. Schluter, J. R. Tetreault (Eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, Association for Computational Linguistics, 2020, pp. 8440–8451. URL: https://doi.org/10.18653/v1/2020.acl-main. 747. doi:10.18653/V1/2020.ACL-MAIN.747. [53] F. Feng, Y. Yang, D. Cer, N. Arivazhagan, W. Wang, Language-agnostic BERT sentence embedding, in: S. Muresan, P. Nakov, A. Villavicencio (Eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, Association for Computational Linguistics, 2022, pp. 878–891. URL: https://doi. org/10.18653/v1/2022.acl-long.62. doi:10.18653/V1/2022.ACL-LONG.62. [54] M. Post, A call for clarity in reporting BLEU scores, in: O. Bojar, R. Chatterjee, C. Federmann, M. Fishel, Y. Graham, B. Haddow, M. Huck, A. Jimeno-Yepes, P. Koehn, C. Monz, M. Negri, A. Névéol, M. L. Neves, M. Post, L. Specia, M. Turchi, K. Verspoor (Eds.), Proceedings of the Third Conference on Machine Translation: Research Papers, WMT 2018, Belgium, Brussels, October 31 - November 1, 2018, Association for Computational Linguistics, 2018, pp. 186–191. URL: https://doi.org/10.18653/v1/w18-6319. doi:10.18653/V1/W18-6319. [55] J. Peng, Z. Han, H. Zhang, J. Ye, C. Liu, B. Liu, M. Guo, H. Chen, Z. Lin, Y. Tang, A Multilingual Text Detoxification Method Based on Few-shot Learning and CO-STAR Framework, in: G. Faggioli, N. Ferro, P. Galuščáková, A. G. S. de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, CEUR-WS.org, 2024. [56] M. AI, Kimi chatbot, 2024. URL: https://kimi.moonshot.cn, accessed: 2024-05-31. [57] M. Vallecillo-Rodríguez, A. M. Martín-Valdivia, SINAI at PAN 2024 TextDetox: Application of Tree of Thought Strategy in Large Language Models for Multilingual Text Detoxification, in: G. Faggioli, N. Ferro, P. Galuščáková, A. G. S. de Herrera (Eds.), Working Notes of CLEF 2024 Conference and Labs of the Evaluation Forum, CEUR-WS.org, 2024. [58] OpenAI, Chatgpt: Optimizing language models for dialogue, 2022. URL: https://openai.com/blog/ chatgpt, accessed: 2024-05-31. [59] M. Najafi, E. Tavan, S. Colreavy, Marsan at PAN 2024 TextDetox: ToxiCleanse RL and Paving the Way for Toxicity-Free Online Discourse, in: G. Faggioli, N. Ferro, P. Galuščáková, A. G. S. de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, CEUR-WS.org, 2024. [60] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de Las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, W. E. Sayed, Mistral 7b, CoRR abs/2310.06825 (2023). URL: https://doi.org/10.48550/ arXiv.2310.06825. doi:10.48550/ARXIV.2310.06825. arXiv:2310.06825. [61] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal policy optimization algorithms,
CoRR abs/1707.06347 (2017). URL: http://arxiv.org/abs/1707.06347. arXiv:1707.06347. [62] L. von Werra, Y. Belkada, L. Tunstall, E. Beeching, T. Thrush, N. Lambert, S. Huang, Trl: Transformer reinforcement learning, https://github.com/huggingface/trl, 2020. [63] S. Gangopadhyay, M. Khan, H. Jabeen, HybridDetox: Combining Supervised and Unsupervised Methods for Efective Multilingual Text Detoxification, in: G. Faggioli, N. Ferro, P. Galuščáková, A. G. S. de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, CEUR-WS.org, 2024. [64] C. Rafel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, Exploring the limits of transfer learning with a unified text-to-text transformer, J. Mach. Learn. Res. 21 (2020) 140:1–140:67. URL: http://jmlr.org/papers/v21/20-074.html. [65] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, L. Zettlemoyer, BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension, in: D. Jurafsky, J. Chai, N. Schluter, J. R. Tetreault (Eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, Association for Computational Linguistics, 2020, pp. 7871–7880. URL: https: //doi.org/10.18653/v1/2020.acl-main.703. doi:10.18653/V1/2020.ACL-MAIN.703. [66] V. Protasov, PAN 2024 Multilingual TextDetox: Exploring Cross-lingual Transfer in Case of Large Language Models, in: G. Faggioli, N. Ferro, P. Galuščáková, A. G. S. de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, CEUR-WS.org, 2024. [67] N. Muennighof, T. Wang, L. Sutawika, A. Roberts, S. Biderman, T. L. Scao, M. S. Bari, S. Shen, Z. X. Yong, H. Schoelkopf, X. Tang, D. Radev, A. F. Aji, K. Almubarak, S. Albanie, Z. Alyafeai, A. Webson, E. Raf, C. Rafel, Crosslingual generalization through multitask finetuning, in: A. Rogers, J. L. Boyd-Graber, N. Okazaki (Eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, Association for Computational Linguistics, 2023, pp. 15991–16111. URL: https://doi.org/10.18653/v1/2023.acl-long.891. doi:10.18653/V1/2023.ACL-LONG.891. [68] N. Sushko, PAN 2024 Multilingual TextDetox: Exploring Diferent Regimes For Synthetic Data Training For Multilingual Text Detoxification, in: G. Faggioli, N. Ferro, P. Galuščáková, A. G. S. de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, CEUR-WS.org, 2024. [69] E. Rykov, K. Zaytsev, I. Anisimov, A. Voronin, SmurfCat at PAN TexDetox 2024: Alignment of Multilingual Transformers for Text Detoxification, in: G. Faggioli, N. Ferro, P. Galuščáková, A. G. S. de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, CEUR-WS.org, 2024. [70] J. Hong, N. Lee, J. Thorne, ORPO: monolithic preference optimization without reference model, CoRR abs/2403.07691 (2024). URL: https://doi.org/10.48550/arXiv.2403.07691. doi:10.48550/ ARXIV.2403.07691. arXiv:2403.07691. [71] AI@Meta, Llama 3 model card (2024). URL: https://github.com/meta-llama/llama3/blob/main/
MODEL_CARD.md. [72] S. Pletenev, Memu_pro_kotow at PAN 2024 TextDetox: Uncensored Llama3 Helps to Censor Better, in: G. Faggioli, N. Ferro, P. Galuščáková, A. G. S. de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, CEUR-WS.org, 2024. [73] A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A. Dombrowski, S. Goel, N. Li, M. J. Byun, Z. Wang, A. Mallen, S. Basart, S. Koyejo, D. Song, M. Fredrikson, J. Z. Kolter, D. Hendrycks, Representation engineering: A top-down approach to AI transparency, CoRR abs/2310.01405 (2023). URL: https://doi.org/10.48550/arXiv.2310.01405. doi:10.48550/ARXIV.2310.01405. arXiv:2310.01405. [74] Z. Luo, M. Luo, A. Wang, Multilingual Text Detoxification Using Google Cloud Translation and Post-Processing, in: G. Faggioli, N. Ferro, P. Galuščáková, A. G. S. de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, CEUR-WS.org, 2024. [75] A. Forever, rugpt3, 2022. URL: https://huggingface.co/ai-forever, accessed: 2024-05-31. [76] V. Zinkovich, S. Karpukhin, N. Kurdiukov, P. Tikhomirov, nlp_enjoyers at Multilingual Textual Detoxification (CLEF-2024, in: G. Faggioli, N. Ferro, P. Galuščáková, A. G. S. de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, CEUR-WS.org, 2024. [77] E. Řehulka, M. Šuppa, RAG Meets Detox: Enhancing Text Detoxification Using Open-Source Large Language Models with Retrieval Augmented Generation, in: G. Faggioli, N. Ferro, P. Galuščáková, A. G. S. de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, CEUR-WS.org, 2024. [78] MTS.AI, Cotype: Generative ai solutions, 2022. URL: https://mts.ai, accessed: 2024-05-31.
Here, we provide the extended results—from both automatic and human evaluation setups—based on three evaluation parameters for all languages: English (Table 5), Spanish (Table 6), German (Table 7), Chinese (Table 8), Arabic (Table 9), Hindi (Table 10), Ukrainian (Table 11), Russian (Table 12), and Amharic (Table 13). In every table, the baselines are highlighted with gray ; Human References are highlighted with green ; the ordering is made by J score from Human Evaluation results. The automatic evaluation is based on the full test set of 600 samples per language; human evaluation was performed on 100 set of the test set per language.
J
J
J
J*