<?xml version="1.0" encoding="UTF-8"?>
<TEI xml:space="preserve" xmlns="http://www.tei-c.org/ns/1.0" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xsi:schemaLocation="http://www.tei-c.org/ns/1.0 https://raw.githubusercontent.com/kermitt2/grobid/master/grobid-home/schemas/xsd/Grobid.xsd"
 xmlns:xlink="http://www.w3.org/1999/xlink">
	<teiHeader xml:lang="en">
		<fileDesc>
			<titleStmt>
				<title level="a" type="main">Overview of the CLEF-2024 CheckThat! Lab Task 6 on Robustness of Credibility Assessment with Adversarial Examples (InCrediblAE)</title>
			</titleStmt>
			<publicationStmt>
				<publisher/>
				<availability status="unknown"><licence/></availability>
			</publicationStmt>
			<sourceDesc>
				<biblStruct>
					<analytic>
						<author>
							<persName><forename type="first">Piotr</forename><surname>Przybyła</surname></persName>
							<email>piotr.przybyla@upf.edu</email>
							<affiliation key="aff0">
								<orgName type="institution">Universitat Pompeu Fabra</orgName>
								<address>
									<settlement>Barcelona</settlement>
									<country key="ES">Spain</country>
								</address>
							</affiliation>
							<affiliation key="aff1">
								<orgName type="department">Institute of Computer Science</orgName>
								<orgName type="institution">Polish Academy of Sciences</orgName>
								<address>
									<settlement>Warsaw</settlement>
									<country key="PL">Poland</country>
								</address>
							</affiliation>
						</author>
						<author>
							<persName><forename type="first">Ben</forename><surname>Wu</surname></persName>
							<affiliation key="aff2">
								<orgName type="institution">University of Sheffield</orgName>
								<address>
									<settlement>Sheffield</settlement>
									<country key="GB">UK</country>
								</address>
							</affiliation>
						</author>
						<author>
							<persName><forename type="first">Alexander</forename><surname>Shvets</surname></persName>
							<affiliation key="aff0">
								<orgName type="institution">Universitat Pompeu Fabra</orgName>
								<address>
									<settlement>Barcelona</settlement>
									<country key="ES">Spain</country>
								</address>
							</affiliation>
						</author>
						<author>
							<persName><forename type="first">Yida</forename><surname>Mu</surname></persName>
							<email>y.mu@sheffield.ac.uk</email>
							<affiliation key="aff2">
								<orgName type="institution">University of Sheffield</orgName>
								<address>
									<settlement>Sheffield</settlement>
									<country key="GB">UK</country>
								</address>
							</affiliation>
						</author>
						<author>
							<persName><forename type="first">Kim</forename><forename type="middle">Cheng</forename><surname>Sheang</surname></persName>
							<email>kimcheng.sheang@upf.edu</email>
							<affiliation key="aff0">
								<orgName type="institution">Universitat Pompeu Fabra</orgName>
								<address>
									<settlement>Barcelona</settlement>
									<country key="ES">Spain</country>
								</address>
							</affiliation>
						</author>
						<author>
							<persName><forename type="first">Xingyi</forename><surname>Song</surname></persName>
							<email>x.song@sheffield.ac.uk</email>
							<affiliation key="aff2">
								<orgName type="institution">University of Sheffield</orgName>
								<address>
									<settlement>Sheffield</settlement>
									<country key="GB">UK</country>
								</address>
							</affiliation>
						</author>
						<author>
							<persName><forename type="first">Horacio</forename><surname>Saggion</surname></persName>
							<email>horacio.saggion@upf.edu</email>
							<affiliation key="aff2">
								<orgName type="institution">University of Sheffield</orgName>
								<address>
									<settlement>Sheffield</settlement>
									<country key="GB">UK</country>
								</address>
							</affiliation>
						</author>
						<title level="a" type="main">Overview of the CLEF-2024 CheckThat! Lab Task 6 on Robustness of Credibility Assessment with Adversarial Examples (InCrediblAE)</title>
					</analytic>
					<monogr>
						<idno type="ISSN">1613-0073</idno>
					</monogr>
					<idno type="MD5">A50E090B0F0B5003DA208327C9AA5A4C</idno>
				</biblStruct>
			</sourceDesc>
		</fileDesc>
		<encodingDesc>
			<appInfo>
				<application version="0.7.2" ident="GROBID" when="2025-04-23T17:58+0000">
					<desc>GROBID - A machine learning software for extracting information from scholarly documents</desc>
					<ref target="https://github.com/kermitt2/grobid"/>
				</application>
			</appInfo>
		</encodingDesc>
		<profileDesc>
			<textClass>
				<keywords>
					<term>adversarial examples, robustness, misinformation detection, credibility, text classification, natural language processing H. Saggion) 0000-0001-9043-6817 (P. Przybyła)</term>
					<term>0009-0002-0918-526X (B. Wu)</term>
					<term>0000-0002-8255-9435 (Y. Mu)</term>
					<term>0000-0002-4662-0358 (K. C. Sheang)</term>
					<term>0000-0002-4188-6974 (X. Song)</term>
					<term>0000-0003-0016-7807 (H. Saggion)</term>
				</keywords>
			</textClass>
			<abstract>
<div xmlns="http://www.tei-c.org/ns/1.0"><p>Task 6 at CheckThat! Lab, organised at CLEF-2024, is devoted to assessing the robustness of misinformation detection solutions implemented as text classification models. The participants of the task were provided with prediction models and data examples for several problems of credibility estimation and their goal was to come up with adversarial examples (AEs): small modifications to the provided text fragments, such that the original meaning is preserved, but the victim classifier changes its decision. The evaluation involved five domains (detection of: biased news, propaganda techniques, false claims, rumours and COVID-19 misinformation) and three classifiers (BiLSTM, BERT and adversarially fine-tuned RoBERTa). Six teams participated in the task, representing a variety of approaches and substantially outperforming previous AE generation solutions. We also performed manual evaluation, which highlighted some modification techniques that are particularly likely to pass unnoticed by human readers. Overall, the task results emphasise the need to assess the robustness of text classification solutions before implementing them in content filtering on large platforms, such as social media.</p></div>
			</abstract>
		</profileDesc>
	</teiHeader>
	<text xml:lang="en">
		<body>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="1.">Introduction</head><p>The challenges of misinformation have been taken up with great energy and vigour by the NLP and IR communities. The main reasons for such enthusiastic adoption of the new tasks are wide availability of textual data to train on and tantalisingly simple dichotomy of fake vs. real, clearly fitting the familiar task of binary classification. Among a great deal of work in the domain <ref type="bibr" target="#b0">[1,</ref><ref type="bibr" target="#b1">2]</ref>, this framework has also enabled numerous shared tasks, including detecting hyperpartisan news <ref type="bibr" target="#b2">[3]</ref>, propaganda <ref type="bibr" target="#b3">[4]</ref>, bots <ref type="bibr" target="#b4">[5]</ref>, false claims <ref type="bibr" target="#b5">[6]</ref> and more. These research results quickly found applications in content moderation for large media platforms, which increasingly rely on ML tools to support, but also to replace the human effort <ref type="bibr" target="#b6">[7]</ref>.</p><p>However, a shared task framework is far from the real-world application scenario, where the test data are not fixed, but are generated continuously by users. This means that if a malicious actor sees their non-credible content rejected by the system, they are likely to try to modify it to pass the filters, rather than simply abandon their goals. Unfortunately, the deep learning architectures, which many of the best-performing solutions use, are known for their susceptibility to adversarial examples, i.e. data instances modified with the intent of fooling a classifier <ref type="bibr" target="#b7">[8]</ref>. While discovering adversarial examples for text is more challenging than in other domains, it is definitely possible <ref type="bibr" target="#b8">[9]</ref>. Thus, investigating the robustness of text credibility assessment solutions is indispensable for making them applicable in real-world adversarial scenarios.</p><p>Here we report on the shared task on Investigating Robustness of Credibility Assessment with Adversarial Examples (InCrediblAE), which was organised as Task 6 of the CheckThat! 2024 evaluation lab <ref type="bibr" target="#b9">[10,</ref><ref type="bibr" target="#b10">11]</ref> at the CLEF 2024 conference. In InCrediblAE, participants get access to the following resources for each domain:</p><p>1. three victim classifiers, assessing the credibility of the input text and returning a score, 2. an attack dataset, including around 400 instances unseen by the classifier in training.</p><p>In the task we take into account five domains, corresponding to important challenges in the credibility assessment -see section 2 for details.</p><p>The goal of the participants is to make modification to the instances in the attack dataset, turning them into adversarial examples. Each adversarial example (AE) is evaluated on meaning preservation, i.e. how similar it is to the original; and classifier confusion, i.e. whether the output of the victim classifier is different than for the original.</p><p>The evaluation consists of two stages. The automatic evaluation follows the framework established in the field of adversarial learning, with the above factors assessed through automatic measures, i.e. BODEGA score <ref type="bibr" target="#b11">[12]</ref>, leading to a ranking list -see section 3.1. In the manual evaluation we use human judgement to assess the semantic similarity between attack sentences and their original counterparts. This process aims to highlight potential errors arising from automatic evaluation metrics, as well as to create a high-quality text similarity dataset for the future development and evaluation of metrics -see section 3.2.</p><p>The task has attracted six teams submitting various approaches (section 4.1), some of which have clearly outperformed previous solutions applied to the same problems (section 4.2), generally confirming the high vulnerability of popular text classifiers to adversarial attacks. What is more, the results of the manual evaluation (section 4.3) highlight cases where AEs might appear far from the original text if judged by automatic measures, but are in fact quite convincing to human annotators. The results of the manual annotation are made openly available for future research <ref type="foot" target="#foot_0">1</ref> . We also share the code<ref type="foot" target="#foot_1">2</ref> and data <ref type="foot" target="#foot_2">3</ref>for the automatic evaluation.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.">Task data</head><p>The shared task uses the foundation of the BODEGA framework <ref type="bibr" target="#b11">[12]</ref>, which has been created to enable systematic robustness testing in the area of misinformation detection. Within this framework, several domains are available, each organised around a credibility assessment problem, defined as a binary text classification task. Within a domain, an expert-annotated corpus of documents is used to train victim classifiers (train subset) and test the attack performance (attack subset).</p><p>The participants are provided with Python code, hosted on Google Colab, allowing to interact with the victim models in an attack scenario through OpenAttack interface <ref type="bibr" target="#b12">[13]</ref>. Their goal is to prepare a procedure that modifies the text fragments in the attack dataset to achieve a different decision of a classifier with a minimal meaning alteration. The participant submission includes the AEs, as well as the number of victim model queries needed to find them.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.1.">Datasets</head><p>Four of the domains in InCrediblAE have been prepared in BODEGA based on previously published corpora. The final one (C19) is new and was not available before.</p><p>Style-based news bias assessment (HN) is a task of verifying credibility of a news article based on its overall writing style. It relies on previous work indicating that stylistic analysis of fake news outlets can be used to distinguish them from credible sources <ref type="bibr" target="#b14">[14,</ref><ref type="bibr" target="#b15">15]</ref>. The corpus contains news bias annotations assigned at the level of the source (whole website) by journalists from BuzzFeed and MediaBiasFactCheck.com 4 . For the purpose of BODEGA, 10% of the training instances were used and the non-credibility label was assigned to articles from sources marked as hyperpartisan, both left-and right-wing. Propaganda detection (PR) is focused on recognising specific manipulation techniques appealing to emotions <ref type="bibr" target="#b16">[16]</ref>, for example name-calling, flag-waving or straw-man fallacy. This approach has the advantage of being fine-grained by highlighting manipulative fragments in text, akin to the NER (Named Entity Recognition) tasks. We rely on the token-level annotations of 14 techniques, marked by professional annotators for the SemEval 2020 Task 11 (Detection of Propaganda Techniques in News Articles) <ref type="bibr" target="#b3">[4]</ref>, for which the training set is public 5 . In order to cast the task as binary text classification for BODEGA, the corpus was split into sentences and those including some tokens marked as propaganda were labelled as non-credible.</p><p>Fact checking (FC) is an approach to misinformation detection based on extracting claims made in a piece of text and verifying them with respect to a trusted knowledge base <ref type="bibr" target="#b17">[17]</ref>. In order to represent the problem as binary classification, we focus on the final stage of the workflow, when a claim is compared to relevant evidence from the knowledge base, which either confirms its validity or refutes it. In BODEGA, the data from the FEVER shared task <ref type="bibr" target="#b18">[18]</ref> is used, consisting of claims that were paired with relevant passages from Wikipedia articles. The instances where a claim is supported by the evidence were labelled as credible, and those when it is refuted as non-credible.</p><p>Rumour detection (RD) is aimed at detecting information spreading widely over social media despite it not coming from a credible source. Rumours can be detected using many indicators <ref type="bibr" target="#b19">[19]</ref>, but here we focus on the textual content of a social media post, as well as the reactions of other users. In BODEGA, this is achieved thanks to the augmented dataset of rumours and non-rumours for rumour detection <ref type="bibr" target="#b20">[20]</ref>, created from Twitter threads relevant to six real-world events, labelled by experts according to the source reliability of the initial post. One of the events (Charlie Hebdo shooting) was set aside as the attack dataset.</p><p>COVID-19 misinformation detection (C19) focuses on binary classification 6 of misinformation related to COVID-19 <ref type="bibr" target="#b21">[21,</ref><ref type="bibr" target="#b22">22]</ref>. Given a known false claim about the disease, the task is to determine whether a user's tweet supports that false claim. If so, the tweet is classified as COVID-19 misinformation (positive class). Alternative responses, such as contradicting, questioning, commenting, or irrelevance regarding the false claim are reserved for the negative class. Appendix B provides examples from the dataset.</p><p>Table <ref type="table" target="#tab_0">1</ref> summarises the information on datasets, including the sizes of subsets: training (for training victim classifiers), development (reserved for future use) and attack (to be modified into AEs), as well as the percentage of positive (non-credible) instances.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.2.">Victim classifiers</head><p>Training datasets were used to prepare victim models, representing popular approaches to text classification. Two of the models (BiLSTM and BERT) were trained as in BODEGA framework, but the surprise classifier was trained specifically for the shared task and it was revealed to the participants in the test phase, one week before the submissions.</p><p>BiLSTM classifier consists of an embedding layer (token representations of size 32), two LSTM <ref type="bibr" target="#b23">[23]</ref> layers (forwards and backwards, hidden representation of size 128) and a dense linear layer converting the text fragment representation (of size 256) into two-class probability, normalised using softmax.</p><p>BERT classifier is a bert-base-uncased model <ref type="bibr" target="#b24">[24]</ref> from the HuggingFace Transformers library <ref type="bibr" target="#b25">[25]</ref>, fine-tuned for sequence classification using Adam optimiser with linear weight decay <ref type="bibr" target="#b26">[26]</ref> for 5 epochs.</p><p>Surprise classifier is a RoBERTa model <ref type="bibr" target="#b27">[27]</ref>, i.e. roberta-base from HuggingFace Transformers, adversarially-trained to be more robust to adversarial attacks. We use data augmentation to improve robustness: First, the model is fine-tuned for one epoch on the train dataset, then adversarial examples are generated from the entire train dataset using BERT-ATTACK <ref type="bibr" target="#b28">[28]</ref>, and then the model is fine-tuned for one epoch on a combination of the train dataset and the successful adversarial examples. We train with constant learning rate 2 × 10 −5 and the Adam optimiser. We use batch of size 32 for all tasks except PR, which uses a batch of size 64. Due to computational constraints, for HN we only generate adversarial examples from a subset (6000 samples) of the training data.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.">Evaluation</head><p>The evaluation procedure consists of two stages. Firstly, the BODEGA framework is used to automatically assess the attack effectiveness of each participant in 15 scenarios (5 domains × 3 victims). The average BODEGA score is used to create the leaderboard, expressing the overall performance. Secondly, the task most challenging for automatic evaluation (fact-checking) is used to perform manual annotation of meaning preservation in selected instances.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.1.">Automatic evaluation</head><p>In automatic evaluation, when an example 𝑥 𝑖 is modified into AE 𝑥 * 𝑖 , the quality of the transformation is assessed through BODEGA score defined as follows <ref type="bibr" target="#b11">[12]</ref>:</p><formula xml:id="formula_0">BODEGA_score(𝑥 𝑖 , 𝑥 * 𝑖 ) = Con_score(𝑥 𝑖 , 𝑥 * 𝑖 ) × Sem_score(𝑥 𝑖 , 𝑥 * 𝑖 ) × Char_score(𝑥 𝑖 , 𝑥 * 𝑖 ),</formula><p>where:</p><p>• Con_score, i.e. confusion score, takes value of 1 when the attacked classifier predicts a different class for 𝑥 * 𝑖 than it did for 𝑥 𝑖 , and 0 otherwise. • Sem_score, i.e. semantic similarity score, is a measure of meaning preservation between 𝑥 𝑖 and 𝑥 * 𝑖 , computed using the BLEURT <ref type="bibr" target="#b29">[29]</ref> evaluation measure (BLEURT-20 variant), clipped to the (0-1) range.</p><p>• Char_score, i.e. character similarity score, is a measure of similarity of 𝑥 𝑖 and 𝑥 * 𝑖 as character sequences, computed through Levenshtein distance <ref type="bibr" target="#b30">[30]</ref>, scaled to (0-1) similarity.</p><p>We can see that an AE will be ranked highly if it changes the output of the classifier (Con_score(𝑥 𝑖 , 𝑥 * 𝑖 ) = 1), but at the same time preserves both the meaning (Sem_score(𝑥 𝑖 , 𝑥 * 𝑖 ) ≈ 1) and the appearance (Char_score(𝑥 𝑖 , 𝑥 * 𝑖 ) ≈ 1) of the original text. To measure the overall attack success in a particular scenario, the BODEGA score averaged over all instances in the attack set is employed. However, the constituent scores, also averaged over the dataset, can be used to understand the results. We also report the number of queries that is needed (on average) for a single AE to be found.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Table 2</head><p>Category definitions for the manual evaluation.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Category Definitions</head></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Preserve the Semantic Meaning</head><p>This label is used when the semantic content of the attack sample closely aligns with that of the original sample. Participants should use this label if the meaning, context, and intent of the compared texts remain essentially unchanged, indicating that the attack sample has effectively maintained the core message of the original.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Change the Semantic Meaning</head><p>Participants should apply this label when there is a noticeable alteration in the semantic content between the original and the attack sample. This label indicates that while the attack sample may be related or similar in some aspects to the original, it diverges enough in meaning or intent to be considered distinct or modified. For example, name entities (including Year, Name, Location, etc.) are changed in the modified text.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>No sense</head><p>The content of the attack sample does not make any sense.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Table 3</head><p>Confidence scores and definitions used by Mu et al. <ref type="bibr" target="#b31">[31]</ref>.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Confidence Definitions 1</head><p>Extremely unconfident about the annotation (I'm really unsure about the annotation. It may belong to another category as well, you may wish to discard this instance from the training.)</p><formula xml:id="formula_1">2</formula><p>Not confident about the annotation (I'm not sure about the annotation, it seems it also belongs to other categories, but you can still include this instance as a "silver standard instance" in training.)</p><formula xml:id="formula_2">3</formula><p>Pretty confident about the annotation (I'm pretty sure about the annotation, but might be in high chance other annotators may label it in a different category.)</p><formula xml:id="formula_3">4</formula><p>Fairly confident about the annotation (I'm confident about the annotation, but might be in small chance other annotators may label it in a different category.) 5</p><p>Extremely confident about the annotation (I'm certain about the annotation without a doubt.)</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.2.">Manual evaluation</head><p>The goal of the manual evaluation is to highlight the cases where the automatic evaluation measures, especially regarding semantic similarity, might not be an accurate representation of human reception of the adversarially modified content. To that end, we have selected the samples from the fact-checking domain and the surprise victim, where even small changes in text can alter the meaning and, consequently, the credibility label.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Task Description</head><p>We aim to gather assessments regarding the semantic similarities between attack samples and the original samples. Participants in the shared task are requested to dedicate approximately 60 minutes to this manual evaluation (i.e., 100 samples per participant), which are conducted using an open-source, collaborative annotation platform, i.e., GATE Teamware 2 <ref type="bibr" target="#b32">[32]</ref>. Judges rate the sample pairs based on the following scale: (a). Preserve the Semantic Meaning, (b). Change the Semantic Meaning, and (c). No sense. Table <ref type="table">2</ref> demonstrates categories and descriptions. Similar to the work of Mu et al. <ref type="bibr" target="#b31">[31]</ref>, annotators are required to indicate the confidence level (see Table <ref type="table">3</ref>) of their assigned class in the 'Confidence Row'.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Data Sampling</head><p>We randomly select 100 paired samples (i.e., original and modified texts) from each submission, resulting in a total of 600 paired samples. Note that only the successful attacks are considered.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Annotator Training</head><p>We train the annotators by providing a training document detailing the annotation pipeline, which includes (i) a step-by-step tutorial for using the GATE Teamware platform, (ii) a user information sheet to inform about any potential issues and risks that may occur during data annotation, and (iii) a user consent sheet as required by the ethical approval from the University of Sheffield, where the annotation was performed. A total of 12 annotators (i.e., 6 participants and 6 organisers) were recruited to manually annotate the paired samples. These 12 annotators were further divided into 6 separate groups (i.e., two annotators per group). In each group, 100 tweets were assigned to each annotator. Finally, this process yielded 100 double-annotated paired samples from each group, resulting in 600 double-annotated samples in total.</p><p>Annotation Methodology and Quality Assurance All samples are double-annotated by the shared task organizers and participants. Briefly, each paired sample is annotated by one participant and one shared task organiser. A third annotator from the shared-task organisers is used to resolve any conflicts. Given that there are three categories in total, the annotation with the highest confidence score will be considered in the case of three differing annotations.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.">Results</head><p>Here we outline the results of the InCrediblAE shared task in three steps: first, we describe the solutions submitted by the participants (section 4.1), then we present the results of the automatic (section 4.2) and manual evaluation (section 4.3).</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.1.">Participating solutions</head><p>The SINAI team <ref type="bibr" target="#b33">[33]</ref> proposes a method for adversarial attacks based on the substitution of characters by homoglyphs (e.g. characters which resemble the target such as l ≈ 1). The method uses exhaustive search with two variants: with memory and without memory. They ground their approach in the fact that homoglyphs could deceive the human eye while at the same time provoke a Large Language Model classifier to reverse its prediction due to the presence of an unexpected token. According to the official leader board, the approach is sub-optimal, obtaining the last rank in the task according to the official evaluation metrics. However, human evaluation of content preservation is ranked high.</p><p>The MMU_NLP participation <ref type="bibr" target="#b34">[34]</ref> features a system to attack classifiers based on lexical substitution and character replacement. The proposed method searches for candidate words to attack followed by a word replacement mechanism. The word search mechanism masks words to check their vulnerability with those words having a high impact on the classifier performance retained for the attack. The replacement step uses homoglyphs for character replacement or lexical substitution. Character replacement is tested in two different conditions: random character attack or begin/end of word attack. The lexical replacement attack uses a large language model to retrieve a word similar to the target word. Overall result fall short compared to other participants, however the proposed methods improves over the baseline in several settings. The character attack method seems more effective than the word replacement approach.</p><p>The Palöri team <ref type="bibr" target="#b35">[35]</ref> proposes an approach to identify vulnerable words by computing (relying on a masked language model) the difference between the probability distribution of the original sentence and the sentence with a word masked. These differences are used as scores to rank words by their "vulnerability" according to the model. The ranked words (most to least vulnerable) are then replaced using a word from a list of substitutes proposed by the language model. The sentence with the replaced word is used to attack the victim classifier. In case of success, the new sentence is returned, otherwise the method loops using the sentence with the "best" possible substitution (i.e. one which reduced the victim's confidence the most). The method produces successful attacks which however do not preserve the original sentence's meaning. To address this problem a "synonym" dictionary is created, using GloVe embeddings and the aclImbd dataset, to draw substitutes from. The new method only contributes minor improvements over the masked language model. The solution of the OpenFact team <ref type="bibr" target="#b36">[36]</ref> consists in a coupling of various word-substitution approaches in an ensemble in such a way that if the first approach does not succeed in changing the classifier's decision, then the second one is called. In particular, a modification of BERT-ATTACK <ref type="bibr" target="#b28">[28]</ref> was proposed (it features a change in parameter values, an alternative selection of a replacement position by an exhaustive search of candidates that provide the largest difference in probabilities for a predicted class, and an iterative replacement from 1 to 7 words, including punctuation and digits at the latter, until the success of an attack) and backed up by a genetic algorithm <ref type="bibr" target="#b37">[37]</ref> realised in the OpenAttack framework <ref type="bibr" target="#b12">[13]</ref>. Another ensemble was compiled of a proposed greedy search by word swap with synonyms in the word embedding space (prebuilt "counter-fitted" GloVe embeddings <ref type="bibr" target="#b38">[38]</ref>) and another model available in the OpenAttack, TextFooler <ref type="bibr" target="#b39">[39]</ref>, which unlike other similar approaches replaces words in agreement with the syntax of the attacked text. Apart from ensemble models, approaches from the TextAttack framework <ref type="bibr" target="#b40">[40]</ref> were used. They demonstrated superior performance over the baseline methods in automatic scores. In particular, CLARE <ref type="bibr" target="#b41">[41]</ref> -a model that implements a special mask-then-infill procedure that incorporates replace/insert/merge operations allowing for outcomes varied in length -was applied for PR, FC and C19 tasks and consistently yielded better results in all automatic scores. Overall, this solution gained the best automatic scores in most of the domains for most of the victims, which made it the first in the leaderboard created by averaging the scores across all scenarios. However, it was ranked very low within the human evaluation, as in the majority of the cases the meaning of the text was changed.</p><p>The TurQUaz <ref type="bibr" target="#b42">[42]</ref> team leverages a genetic algorithm to look for a combination of character modifications. The modifications that are introduced using a mutation operator include homoglyph replacement, three options of word splitting (random, favouring existing words in the subword outcome, and special heuristic-wise), insertion or removal of individual random letters, and shuffling the order of the letters within the word. The search is carried out until the first flip is found. Only the use of homoglyphs and word splits proved to be efficient. Apart from the genetic algorithm, the team experimented with attacks made by utilizing large language models (LLMs) such as Llama 3 <ref type="foot" target="#foot_3">7</ref> and Mistral<ref type="foot" target="#foot_4">8</ref> . Three approaches have been tried: (i) prompting a model for text paraphrasing, (ii) leveraging a model for identifying words to be changed, and (iii) generating adversarial examples with one LLM and verifying whether an attack is going to be successful with another LLM. None of the approaches outperformed the genetic algorithm, however, the team believes in the potential of LLMs and suggests fine-tuning the models specifically for this task to improve their performance in the future. The primary solution has been ranked third based on the BODEGA score.</p><p>The TextTrojaners team <ref type="bibr">[43]</ref> introduces a BeamAttack method that makes attacks at a word level using RoBERTa <ref type="bibr" target="#b27">[27]</ref> and a beam search as a backbone to produce contextually appropriate word substitutions. Beam search algorithm adapted by enabling operations of replacing, skipping, or removing words allows for generating and evaluating multiple alternative word replacement combinations in a single run. For the identification of the most vulnerable words to be replaced, the team experiments with two ranking approaches that use the explainable AI framework LIME <ref type="bibr" target="#b44">[44]</ref> and logit-based importance scores, as proposed in <ref type="bibr" target="#b28">[28]</ref>. In a series of ablation studies, they show that the choice of a ranking method varies across victims and depends on the dataset. The solution achieved the second-best result on the BODEGA evaluation metric but gained rather low manual evaluation scores.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.2.">Automatic evaluation</head><p>Table <ref type="table">4</ref> includes the results of the evaluation against the BiLSTM victim. Generally, we can see that different approaches dominate in different scenarios. However, in every domain, the best BODEGA score (in boldface) is achieved by a solution submitted to InCrediblAE, rather than a reference solution from previous work (BERT-ATTACK or DeepWordBug). HN appears to be the easiest domain, with the leading solution (TextTrojaners) achieving BODEGA score of 0.91 through 100% confusion with 91% semantic similarity and 99% character similarity. This result, closely followed by OpenFact, is especially impressive when compared to the scores of BERT-ATTACK (BODEGA score of 0.64). The TextTrojaners approach also leads in PR, but we need to note its high amount of queries needed -in this case, 593 compared to TurQUaz achieving almost the same result (0.68 instead of 0.70) with six times less queries. The C19 appears to be the domain most challenging for attacks, although even here we note a vast improvement over the reference method (OpenFact: 0.72 vs BERT-ATTACK: 0.50).</p><p>In the attacks against the BERT victim, evaluated in table 5, the OpenFact method dominates, ceding only in FC to homoglyph-based SINAI. We can also note that the best BODEGA score is either equivalent (for FC, HN and C19) or significantly lower (for PR and RD) than for BiLSTM, indicating higher difficulty of attacking a Transformer-based classifier. This also results in a higher number of queries necessary to</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Table 4</head><p>Results of the automatic evaluation of attacks against the BiLSTM victims, for each of the five domains, expressed through BODEGA score and its constituents, as well as the number of queries (except for OpenFact, whose submission did not include this value). find an AE, e.g. while the leading method for HN domain needed 937 queries to obtain the score of 0.91 with BiLSTM, attacking BERT with the same approach requires 4328 queries on average. The results for the surprise victim (table <ref type="table" target="#tab_3">6</ref>) show that the adversarially-trained classifier indeed is more challenging to attack, leading to lower scores in PR and RD domains. However, the general view remains similar, with OpenFact again dominating with the exception of FC. Their approach clearly works very well with Transformer-based victims, just with an exception of this single task. The increasing level of difficulty is again reflected in a rising numbers of queries, reaching a record value of over 15000 for TextTrojaners attacking in the RD domain.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Domain</head><p>Finally, table <ref type="table" target="#tab_4">7</ref> shows the final leaderboard obtained by averaging the BODEGA scores across victims and domains. We can see that all solutions submitted to the task have beaten the DeepWordBug reference and most have also outperformed BERT-ATTACK, which is a strong reference point. OpenFact and TextTrojaners are clear leaders, with the former slightly better-performing, especially against Transformers victim. However, we need to emphasise that the averaged ranking does not show the whole picture and various methods work best for various scenarios. For example, SINAI is the best approach for the BERT-FC combination. </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.3.">Manual Evaluation</head><p>We randomly selected 100 successful adversarial samples from each team's submission to the scenario including fact checking and surprise victim. During the evaluation, both the original and adversarial samples were presented to the annotators, with differences highlighted. The annotators were asked to categorise each sample pair into one of the three categories described in Section 3.2. They also provided a confidence score for each annotation (5: very confident, 1: not confident). Each sample pair was judged by at least two annotators. The annotation agreement of the initial annotators was 0.52 (Cohen's Kappa). A third annotator was invited if there was a conflict between the two initial annotators. We determined the ranking of participants by the number of samples that fell into the 'Preserve the Semantic Meaning' category, adhering to the principle that a higher count indicates better performance. This ranking method was used because the task demands that the adversarial samples maintain their original meaning. Table <ref type="table">8</ref> presents the final manual evaluation results for all participants.</p><p>In general, we observe we observe a discrepancy between the manual and automatic evaluations (see Table <ref type="table" target="#tab_4">7</ref>). This may be because the manual evaluation task is a fact-checking task. Even a slight replacement of named entities (such as changing the year from 1990 to 1991) could result in a change of meaning.</p><p>The leading team (SINAI) in manual evaluation uses an adversarial attack method mainly based on the substitution of characters with homoglyphs and achieves a 99% score. This suggests that the use of homoglyphs can successfully deceive the annotator's eye. Similarly, team MMU employs similar attack approaches, such as lexical substitution and character replacement, which also achieve a high manual evaluation score (96%).</p><p>Team TurQUaz proposes a method of inserting white space to split English words, which achieved third place on the leaderboard (62%). However, this method may sometimes change the meaning of the original fact-checking document, resulting in a lower manual evaluation score compared to the methods proposed by teams SINAI and MMU. Besides, by inserting white space in the original text, the modified text may become non-interpretable by humans, resulting in a high proportion of third category submissions by Team TurQUaz, i.e., "The sentence does not make any sense. "</p><p>We observe that OpenFact (11%) and TextTrojaners (7%) obtain lower manual evaluation scores. By manually investigating text modified by Team OpenFact, we notice that some key information, such as time and location, has been changed. Note that such named entities play a vital role in the context of fact-checking downstream tasks. Therefore, changes to these named entities result in a higher number of samples labelled as "Change the Semantic Meaning. " As for the solution of TextTrojaners, the choice of alternative words solely depends on the context rather than the word to be replaced. This can significantly deviate the meaning. In addition, the operation of word removal without further word agreement adjustment may lead to nonsensical sentences.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.">Discussion</head><p>The first conclusion from the results obtained is clear: the state of the art in AE generation for misinformation detection, established by previous solutions, has advanced considerably. Various solutions were submitted to the shared task, but they are based on the established lines of research in the area: word replacements (preserving meaning similarity) or character replacements (preserving visual similarity).</p><p>The word-level solutions (esp. TextTrojaners and OpenFact) were performing the best in most scenarios, but not all of them. Fact checking is a clear outlier due to its nature: every word matters, making it hard to perform any change without drastically affecting the meaning. This opens the avenue for character-level modifications and, indeed, such solution (SINAI) provided the best results in manual evaluation.</p><p>We also need to acknowledge the limitations of the evaluation setup. It aims to predict the likelihood of an AE fooling the victim classifier and transmitting the intended message, as encoded through BODEGA score. However, human readers are the intended recipients of misinformation, and they are also able to refuse to engage with a message that seems suspicious, artificial or distorted, e.g. due to use of letter with non-standard shapes. Thus, the success of AEs will also depend on the visual appearance of the manipulated content, which is not directly evaluated in the current setup. Quantifying this effect would be challenging in manual evaluation and even more so in an automatic setup.</p><p>In any case, the results that we do have leave one thing clear: popular architectures for text classification are very vulnerable to attack with AEs. While the adversarially-trained model posed slightly harder challenge, ultimately AEs were found for nearly all cases in these scenarios as well.</p><p>How can we protect the real-world deployments of text classifiers against such attack? The first barrier can be established by limiting the access to the victim model. We can see from the results that the advance over previous state of the art was accompanied by a raise in the number of queries sent, well into hundreds and thousands for each generated AE sample. Nevertheless, all machine-learning-based solutions for content filtering should be only deployed after a thorough analysis of their adversarial robustness.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="6.">Conclusion</head><p>In the InCrediblAE shared task, six teams participated with various solutions, both operating with the word-level and character-level changes. The participants' approaches (and two reference solutions) were evaluated using five misinformation-detection scenarios and three victim models.</p><p>In total, 53,544 text modifications were considered and automatically assessed in terms of classifier confusion, meaning preservation and character similarity. The submitted solutions easily outperformed previous work in all of the tested scenarios. The manual evaluation highlighted the special role of the fact-checking tasks and the efficacy of character replacement in performing modification imperceptible to humans.</p><p>We hope that the combined effort of the participants and organisers of the InCrediblAE shared task will succeed in both highlighting the importance of robustness testing and showcasing the best solutions. To facilitate this outcome, the code and resources necessary for performing the automatic evaluation remain openly available. <ref type="foot" target="#foot_5">9</ref>authority can be held responsible for them. We also acknowledge support from Departament de Recerca i Universitats de la Generalitat de Catalunya (ajuts SGR-Cat 2021) and from Maria de Maeztu Units of Excellence Programme CEX2021-001195-M, funded by MCIN/AEI /10.13039/501100011033.</p></div><figure xmlns="http://www.tei-c.org/ns/1.0" type="table" xml:id="tab_0"><head>Table 1</head><label>1</label><figDesc>The sizes of subsets in each domain and the percentage of cases labelled as positive.</figDesc><table><row><cell cols="5">Task Training Attack Development Positive</cell></row><row><cell>HN</cell><cell>60,235</cell><cell>400</cell><cell>3,600</cell><cell>50.00%</cell></row><row><cell>PR</cell><cell>12,675</cell><cell>416</cell><cell>3,320</cell><cell>29.42%</cell></row><row><cell>FC</cell><cell>172,763</cell><cell>405</cell><cell>19,010</cell><cell>51.27%</cell></row><row><cell>RD</cell><cell>8,694</cell><cell>415</cell><cell>2,070</cell><cell>32.68%</cell></row><row><cell>C19</cell><cell>1,130</cell><cell>595</cell><cell>0</cell><cell>42.55%</cell></row></table></figure>
<figure xmlns="http://www.tei-c.org/ns/1.0" type="table" xml:id="tab_2"><head>Table 5</head><label>5</label><figDesc>Results of the automatic evaluation of attacks against the BERT victims, for each of the five domains, expressed through BODEGA score and its constituents, as well as the number of queries (except for OpenFact, whose submission did not include this value).</figDesc><table><row><cell>Domain</cell><cell cols="6">Method BODEGA Confusion Semantic Character Queries</cell></row><row><cell>PR</cell><cell>MMU_NLP</cell><cell>0.33</cell><cell>0.47</cell><cell>0.75</cell><cell>0.95</cell><cell>438.26</cell></row><row><cell></cell><cell>OpenFact</cell><cell>0.68</cell><cell>0.97</cell><cell>0.77</cell><cell>0.89</cell><cell>-</cell></row><row><cell></cell><cell>Palöri</cell><cell>0.56</cell><cell>0.97</cell><cell>0.64</cell><cell>0.88</cell><cell>65.37</cell></row><row><cell></cell><cell>SINAI</cell><cell>0.38</cell><cell>0.43</cell><cell>0.92</cell><cell>0.98</cell><cell>288.73</cell></row><row><cell></cell><cell>TextTrojaners</cell><cell>0.62</cell><cell>0.99</cell><cell>0.71</cell><cell>0.86</cell><cell>4,097.37</cell></row><row><cell></cell><cell>TurQUaz</cell><cell>0.46</cell><cell>0.68</cell><cell>0.72</cell><cell>0.94</cell><cell>254.75</cell></row><row><cell></cell><cell>BERT-ATTACK</cell><cell>0.43</cell><cell>0.70</cell><cell>0.68</cell><cell>0.90</cell><cell>80.16</cell></row><row><cell></cell><cell>DeepWordBug</cell><cell>0.28</cell><cell>0.36</cell><cell>0.79</cell><cell>0.96</cell><cell>27.43</cell></row><row><cell>FC</cell><cell>MMU_NLP</cell><cell>0.55</cell><cell>0.73</cell><cell>0.78</cell><cell>0.96</cell><cell>710.29</cell></row><row><cell></cell><cell>OpenFact</cell><cell>0.80</cell><cell>1.00</cell><cell>0.83</cell><cell>0.97</cell><cell>-</cell></row><row><cell></cell><cell>Palöri</cell><cell>0.62</cell><cell>0.98</cell><cell>0.66</cell><cell>0.96</cell><cell>102.52</cell></row><row><cell></cell><cell>SINAI</cell><cell>0.82</cell><cell>0.97</cell><cell>0.86</cell><cell>0.98</cell><cell>250.74</cell></row><row><cell></cell><cell>TextTrojaners</cell><cell>0.79</cell><cell>1.00</cell><cell>0.83</cell><cell>0.96</cell><cell>1,390.83</cell></row><row><cell></cell><cell>TurQUaz</cell><cell>0.74</cell><cell>1.00</cell><cell>0.78</cell><cell>0.95</cell><cell>70.70</cell></row><row><cell></cell><cell>BERT-ATTACK</cell><cell>0.53</cell><cell>0.77</cell><cell>0.73</cell><cell>0.95</cell><cell>146.73</cell></row><row><cell></cell><cell>DeepWordBug</cell><cell>0.44</cell><cell>0.53</cell><cell>0.84</cell><cell>0.98</cell><cell>54.32</cell></row><row><cell>RD</cell><cell>MMU_NLP</cell><cell>0.15</cell><cell>0.37</cell><cell>0.42</cell><cell>0.93</cell><cell>986.24</cell></row><row><cell></cell><cell>OpenFact</cell><cell>0.65</cell><cell>0.78</cell><cell>0.86</cell><cell>0.95</cell><cell>-</cell></row><row><cell></cell><cell>Palöri</cell><cell>0.18</cell><cell>0.45</cell><cell>0.42</cell><cell>0.94</cell><cell>1,077.24</cell></row><row><cell></cell><cell>SINAI</cell><cell>0.12</cell><cell>0.14</cell><cell>0.87</cell><cell>1.00</cell><cell>143.06</cell></row><row><cell></cell><cell>TextTrojaners</cell><cell>0.59</cell><cell>0.80</cell><cell>0.79</cell><cell cols="2">0.91 10,618.93</cell></row><row><cell></cell><cell>TurQUaz</cell><cell>0.22</cell><cell>0.38</cell><cell>0.61</cell><cell>0.95</cell><cell>417.75</cell></row><row><cell></cell><cell>BERT-ATTACK</cell><cell>0.18</cell><cell>0.44</cell><cell>0.43</cell><cell>0.96</cell><cell>774.31</cell></row><row><cell></cell><cell>DeepWordBug</cell><cell>0.16</cell><cell>0.23</cell><cell>0.70</cell><cell>0.99</cell><cell>232.74</cell></row><row><cell>HN</cell><cell>MMU_NLP</cell><cell>0.47</cell><cell>0.86</cell><cell>0.55</cell><cell>0.97</cell><cell>806.60</cell></row><row><cell></cell><cell>OpenFact</cell><cell>0.91</cell><cell>1.00</cell><cell>0.92</cell><cell>0.99</cell><cell>-</cell></row><row><cell></cell><cell>Palöri</cell><cell>0.60</cell><cell>0.96</cell><cell>0.64</cell><cell>0.98</cell><cell>502.03</cell></row><row><cell></cell><cell>SINAI</cell><cell>0.24</cell><cell>0.27</cell><cell>0.87</cell><cell>1.00</cell><cell>245.10</cell></row><row><cell></cell><cell>TextTrojaners</cell><cell>0.85</cell><cell>1.00</cell><cell>0.87</cell><cell>0.97</cell><cell>4,327.67</cell></row><row><cell></cell><cell>TurQUaz</cell><cell>0.46</cell><cell>0.84</cell><cell>0.59</cell><cell>0.93</cell><cell>223.54</cell></row><row><cell></cell><cell>BERT-ATTACK</cell><cell>0.60</cell><cell>0.96</cell><cell>0.64</cell><cell>0.97</cell><cell>648.41</cell></row><row><cell></cell><cell>DeepWordBug</cell><cell>0.22</cell><cell>0.29</cell><cell>0.78</cell><cell>1.00</cell><cell>395.94</cell></row><row><cell>C19</cell><cell>MMU_NLP</cell><cell>0.45</cell><cell>0.82</cell><cell>0.58</cell><cell>0.95</cell><cell>142.18</cell></row><row><cell></cell><cell>OpenFact</cell><cell>0.72</cell><cell>0.91</cell><cell>0.82</cell><cell>0.96</cell><cell>-</cell></row><row><cell></cell><cell>Palöri</cell><cell>0.52</cell><cell>0.96</cell><cell>0.57</cell><cell>0.93</cell><cell>201.01</cell></row><row><cell></cell><cell>SINAI</cell><cell>0.41</cell><cell>0.47</cell><cell>0.89</cell><cell>1.00</cell><cell>32.16</cell></row><row><cell></cell><cell>TextTrojaners</cell><cell>0.71</cell><cell>0.98</cell><cell>0.78</cell><cell>0.92</cell><cell>2,628.90</cell></row><row><cell></cell><cell>TurQUaz</cell><cell>0.57</cell><cell>0.96</cell><cell>0.62</cell><cell>0.95</cell><cell>102.92</cell></row><row><cell></cell><cell>BERT-ATTACK</cell><cell>0.42</cell><cell>0.74</cell><cell>0.60</cell><cell>0.95</cell><cell>161.70</cell></row><row><cell></cell><cell>DeepWordBug</cell><cell>0.27</cell><cell>0.39</cell><cell>0.71</cell><cell>0.99</cell><cell>61.06</cell></row></table></figure>
<figure xmlns="http://www.tei-c.org/ns/1.0" type="table" xml:id="tab_3"><head>Table 6</head><label>6</label><figDesc>Results of the automatic evaluation of attacks against the Surprise victims, for each of the five domains, expressed through BODEGA score and its constituents, as well as the number of queries (except for OpenFact, whose submission did not include this value).</figDesc><table><row><cell>Domain</cell><cell cols="6">Method BODEGA Confusion Semantic Character Queries</cell></row><row><cell>PR</cell><cell>MMU_NLP</cell><cell>0.28</cell><cell>0.40</cell><cell>0.76</cell><cell>0.94</cell><cell>525.66</cell></row><row><cell></cell><cell>OpenFact</cell><cell>0.62</cell><cell>0.93</cell><cell>0.75</cell><cell>0.87</cell><cell>-</cell></row><row><cell></cell><cell>Palöri</cell><cell>0.25</cell><cell>0.54</cell><cell>0.55</cell><cell>0.83</cell><cell>482.21</cell></row><row><cell></cell><cell>SINAI</cell><cell>0.26</cell><cell>0.31</cell><cell>0.89</cell><cell>0.97</cell><cell>374.20</cell></row><row><cell></cell><cell>TextTrojaners</cell><cell>0.45</cell><cell>0.97</cell><cell>0.55</cell><cell cols="2">0.79 10,286.82</cell></row><row><cell></cell><cell>TurQUaz</cell><cell>0.20</cell><cell>0.26</cell><cell>0.78</cell><cell>0.95</cell><cell>471.40</cell></row><row><cell></cell><cell>BERT-ATTACK</cell><cell>0.20</cell><cell>0.32</cell><cell>0.69</cell><cell>0.91</cell><cell>117.64</cell></row><row><cell></cell><cell>DeepWordBug</cell><cell>0.13</cell><cell>0.17</cell><cell>0.81</cell><cell>0.96</cell><cell>26.87</cell></row><row><cell>FC</cell><cell>MMU_NLP</cell><cell>0.51</cell><cell>0.68</cell><cell>0.78</cell><cell>0.96</cell><cell>201.15</cell></row><row><cell></cell><cell>OpenFact</cell><cell>0.80</cell><cell>1.00</cell><cell>0.82</cell><cell>0.97</cell><cell>-</cell></row><row><cell></cell><cell>Palöri</cell><cell>0.66</cell><cell>1.00</cell><cell>0.68</cell><cell>0.97</cell><cell>117.77</cell></row><row><cell></cell><cell>SINAI</cell><cell>0.44</cell><cell>0.50</cell><cell>0.89</cell><cell>0.99</cell><cell>43.14</cell></row><row><cell></cell><cell>TextTrojaners</cell><cell>0.82</cell><cell>1.00</cell><cell>0.84</cell><cell>0.97</cell><cell>498.93</cell></row><row><cell></cell><cell>TurQUaz</cell><cell>0.71</cell><cell>1.00</cell><cell>0.75</cell><cell>0.94</cell><cell>90.55</cell></row><row><cell></cell><cell>BERT-ATTACK</cell><cell>0.56</cell><cell>0.79</cell><cell>0.73</cell><cell>0.96</cell><cell>164.07</cell></row><row><cell></cell><cell>DeepWordBug</cell><cell>0.37</cell><cell>0.46</cell><cell>0.83</cell><cell>0.98</cell><cell>53.39</cell></row><row><cell>RD</cell><cell>MMU_NLP</cell><cell>0.16</cell><cell>0.35</cell><cell>0.46</cell><cell>0.97</cell><cell>2,894.29</cell></row><row><cell></cell><cell>OpenFact</cell><cell>0.55</cell><cell>0.71</cell><cell>0.82</cell><cell>0.93</cell><cell>-</cell></row><row><cell></cell><cell>Palöri</cell><cell>0.19</cell><cell>0.47</cell><cell>0.43</cell><cell>0.93</cell><cell>1,513.25</cell></row><row><cell></cell><cell>SINAI</cell><cell>0.09</cell><cell>0.10</cell><cell>0.85</cell><cell>1.00</cell><cell>149.12</cell></row><row><cell></cell><cell>TextTrojaners</cell><cell>0.54</cell><cell>0.87</cell><cell>0.69</cell><cell cols="2">0.84 15,458.12</cell></row><row><cell></cell><cell>TurQUaz</cell><cell>0.17</cell><cell>0.28</cell><cell>0.63</cell><cell>0.96</cell><cell>466.13</cell></row><row><cell></cell><cell>BERT-ATTACK</cell><cell>0.17</cell><cell>0.41</cell><cell>0.42</cell><cell>0.95</cell><cell>951.87</cell></row><row><cell></cell><cell>DeepWordBug</cell><cell>0.12</cell><cell>0.18</cell><cell>0.69</cell><cell>0.99</cell><cell>229.56</cell></row><row><cell>HN</cell><cell>MMU_NLP</cell><cell>0.47</cell><cell>0.77</cell><cell>0.62</cell><cell>0.97</cell><cell>713.89</cell></row><row><cell></cell><cell>OpenFact</cell><cell>0.83</cell><cell>0.99</cell><cell>0.86</cell><cell>0.97</cell><cell>-</cell></row><row><cell></cell><cell>Palöri</cell><cell>0.34</cell><cell>0.57</cell><cell>0.62</cell><cell>0.98</cell><cell>1,453.38</cell></row><row><cell></cell><cell>SINAI</cell><cell>0.36</cell><cell>0.41</cell><cell>0.88</cell><cell>1.00</cell><cell>202.67</cell></row><row><cell></cell><cell>TextTrojaners</cell><cell>0.67</cell><cell>1.00</cell><cell>0.72</cell><cell>0.92</cell><cell>4,596.62</cell></row><row><cell></cell><cell>TurQUaz</cell><cell>0.28</cell><cell>0.47</cell><cell>0.61</cell><cell>0.94</cell><cell>376.77</cell></row><row><cell></cell><cell>BERT-ATTACK</cell><cell>0.38</cell><cell>0.67</cell><cell>0.60</cell><cell>0.95</cell><cell>1,781.97</cell></row><row><cell></cell><cell>DeepWordBug</cell><cell>0.16</cell><cell>0.21</cell><cell>0.76</cell><cell>1.00</cell><cell>384.34</cell></row><row><cell>C19</cell><cell>MMU_NLP</cell><cell>0.42</cell><cell>0.76</cell><cell>0.58</cell><cell>0.94</cell><cell>155.45</cell></row><row><cell></cell><cell>OpenFact</cell><cell>0.72</cell><cell>0.99</cell><cell>0.78</cell><cell>0.93</cell><cell>-</cell></row><row><cell></cell><cell>Palöri</cell><cell>0.46</cell><cell>0.99</cell><cell>0.51</cell><cell>0.89</cell><cell>299.37</cell></row><row><cell></cell><cell>SINAI</cell><cell>0.17</cell><cell>0.18</cell><cell>0.92</cell><cell>1.00</cell><cell>37.97</cell></row><row><cell></cell><cell>TextTrojaners</cell><cell>0.65</cell><cell>1.00</cell><cell>0.71</cell><cell>0.91</cell><cell>6,491.39</cell></row><row><cell></cell><cell>TurQUaz</cell><cell>0.41</cell><cell>0.75</cell><cell>0.59</cell><cell>0.92</cell><cell>253.78</cell></row><row><cell></cell><cell>BERT-ATTACK</cell><cell>0.37</cell><cell>0.68</cell><cell>0.58</cell><cell>0.93</cell><cell>198.26</cell></row><row><cell></cell><cell>DeepWordBug</cell><cell>0.20</cell><cell>0.28</cell><cell>0.72</cell><cell>0.98</cell><cell>60.94</cell></row></table></figure>
<figure xmlns="http://www.tei-c.org/ns/1.0" type="table" xml:id="tab_4"><head>Table 7</head><label>7</label><figDesc>Final leaderboard, created by averaging BODEGA scores across all scenarios (five domains and three victims).</figDesc><table><row><cell>#</cell><cell></cell><cell cols="2">Method BODEGA avg.</cell></row><row><cell>1.</cell><cell cols="2">OpenFact</cell><cell>0.7458</cell></row><row><cell>2.</cell><cell cols="2">TextTrojaners</cell><cell>0.7074</cell></row><row><cell>3.</cell><cell></cell><cell>TurQUaz</cell><cell>0.4859</cell></row><row><cell>4.</cell><cell></cell><cell>Palöri</cell><cell>0.4776</cell></row><row><cell>5.</cell><cell cols="2">MMU_NLP</cell><cell>0.3848</cell></row><row><cell>6.</cell><cell></cell><cell>SINAI</cell><cell>0.3507</cell></row><row><cell cols="3">-BERT-ATTACK</cell><cell>0.4261</cell></row><row><cell cols="3">-DeepWordBug</cell><cell>0.2682</cell></row><row><cell>Table 8</cell><cell></cell><cell></cell></row><row><cell>Manual evaluation results.</cell><cell></cell><cell></cell></row><row><cell cols="2">Team</cell><cell cols="2">% of Preserve the meaning</cell></row><row><cell cols="2">SINAI</cell><cell>99%</cell></row><row><cell cols="2">MMU_NLP</cell><cell>96%</cell></row><row><cell cols="2">TurQUaz</cell><cell>62%</cell></row><row><cell cols="2">Plagori</cell><cell>14%</cell></row><row><cell cols="2">OpenFact</cell><cell>11%</cell></row><row><cell cols="2">TextTrojaners</cell><cell>7%</cell></row></table></figure>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="1" xml:id="foot_0">https://github.com/GateNLP/CLEF2024_InCrediblAE_Manual_Evaluation_Dataset</note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="2" xml:id="foot_1">https://github.com/piotrmp/BODEGA</note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="3" xml:id="foot_2">https://gitlab.com/checkthat_lab/clef2024-checkthat-lab/-/tree/main/task6</note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="7" xml:id="foot_3">https://huggingface.co/docs/transformers/model_doc/llama3</note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="8" xml:id="foot_4">https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2</note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="9" xml:id="foot_5">https://github.com/piotrmp/BODEGA</note>
		</body>
		<back>

			<div type="acknowledgement">
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Acknowledgments</head><p>The work of P. Przybyła is part of the ERINIA project, which has received funding from the European Union's Horizon Europe research and innovation programme under grant agreement No 101060930. This work has been also partially funded by the European Commission under contract numbers HE-101070278 and ISF-101080090. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the funders. Neither the European Union nor the granting</p></div>
			</div>

			<div type="annex">
<div xmlns="http://www.tei-c.org/ns/1.0"><head>A. Annotation Guidelines</head><p>InCrediblAE Shared Task Annotation All participants who submit entries will have their submissions manually evaluated for semantic similarity. To aid in this human evaluation process, participants in the shared tasks will be required to manually assess a selection of the submitted examples.</p><p>Manual Scoring We will gather assessments regarding the semantic similarities between attack samples and the original samples. Participants in the shared task are requested to dedicate approximately 60 mins to this manual evaluation (i.e., 100 samples per participant), which will be conducted using this online annotation tool. Judges will rate the sample pairs based on the following scale: (a). Preserve the Semantic Meaning, (b). Change the Semantic Meaning, (c). No sense. <ref type="bibr">(a)</ref>. Preserve the Semantic Meaning: This label is used when the semantic content of the attack sample closely aligns with that of the original sample. Participants should use this label if the meaning, context, and intent of the compared texts remain essentially unchanged, indicating that the attack sample has effectively maintained the core message of the original. For example:</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Labels and Definitions</head><p>Text Original: middle-earth. middle-earth is the fictional setting of much of british writer j. r. r. tolkien 's legendarium . middle-earth is a setting.</p><p>Text Modified: middle-earth. middle-earth is the fictional background of much of british writer j. r. r. tolkien 's legendarium . middle-earth is a setting. (b). Change the Semantic Meaning: Participants should apply this label when there is a noticeable alteration in the semantic content between the original and the attack sample. This label indicates that while the attack sample may be related or similar in some aspects to the original, it diverges enough in meaning or intent to be considered distinct or modified. For example, name entities (including Year, Name, Location, etc.) are changed in the modified text. For example:</p><p>Text Original: carol danvers. the character has also been known as binary , warbird , and captain marvel at various points in her history , and has been featured in other marvel licensed products including video games , animated television series , and merchandise such as trading cards . carol danvers is a real person.</p><p>Text Modified: carol danvers. the character has also been known as binary , warbird , and captain marvel at various points in her history , and has been featured in other marvel licensed products including video games , animated television series , and merchandise such as trading cards . carol danvers is a fictional person.</p><p>(c). No sense: The sentence does not make any sense.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Confidence Scores Please indicate how confident you are in your assigned class in the Confidence</head><p>Row. The confidence scores range from 1 to 5.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>A.1. Information Sheet for InCrediblAE Participants</head><p>You are invited to participate in this research by contributing to the evaluation of the semantic similarity of adversarial examples. It is important for you to understand the goals of the task and what your participation will involve. Please take the time to read the following information. Please, ask us if there is anything that is not clear or if you would like more information. Thank you very much for your time.</p><p>What is the task's purpose? The aim of this research is to evaluate the robustness of the text classifier in adversarial attacks (a detailed description of the task can be found here: https://checkthat. gitlab.io/clef2024/task6/). In this manual evaluation, we will gather assessments regarding the semantic similarities between attack samples and the original samples. Participants in the shared task are requested to dedicate approximately 8 hours to this manual evaluation, which will be conducted using an online tool. Judges will rate the sample pairs based on the following scale: 3 (Preserve the Semantic Meaning), 2 (Change the Semantic Meaning), and 1 (No sense).</p><p>Your participation, what it involves and why we are grateful It is up to you to decide whether or not you want to participate in this annotation task. If you do decide to support us in the project, you will be given this information sheet to keep (and be asked to sign a separate consent form). You can still withdraw at any time without any consequences and without giving any reason. The entire data collected from withdrawn participants will be destroyed immediately, and no personal information will be kept. If participants submitted a solution to the shared task that also means their submission to the shared tasks will not be manually scored. If you wish to withdraw, please contact Dr Xingyi Song (details in section 10). You will be asked to annotate adversarial examples into 4 categories using the GATE Teamware Platform (https://annotate.gate.ac.uk/).</p><p>What are the possible advantages, disadvantages and risks of being involved? Participating in this evaluation will support our research on the development of a more accurate assessment of the robustness of text classifiers and effectiveness of the adversarial attack methods. No major disadvantages or risks are foreseen, however, it is worth mentioning that the content being annotated may case distress. You will be in charge of selecting the content to be annotated, therefore, you are free to only select content that you are comfortable with. If, at any part of the experiment you feel uncomfortable with the content you are accessing, please talk to one of the responsible researchers.</p><p>Will my involvement be kept confidential? All the information that we collect from you and about you during the course of the research will be kept strictly confidential and will only be accessible to members of the research team. You will not be able to be identified in any reports or publications unless you have given your explicit consent for this on your participant consent form.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>What is the legal basis for processing my personal data?</head><p>According to data protection legislation, we are required to inform you that the legal basis we are applying in order to process your personal data is that 'processing is necessary for the performance of a task carried out in the public interest' (Article 6(1)(e)). Further information can be found in the University of Sheffield's Privacy Notice, which is available online under https://www.sheffield.ac.uk/govern/data-protection/privacy/general.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>What will happen to the data collected in this study?</head><p>The data gathered during this scoring task will be used to assess the effectiveness of adversarial examples generation. Since this data will also benefit other researchers, we plan to release a version of the annotated dataset. You will not be identified and your scoring will be aggregated with multiple other annotators. In the case that you provided us with your e-mail address at the beginning of the scoring task, we will destroy it after the scoring task is finished. The University of Sheffield will act as the Data Controller for this study.</p><p>Who is organising and funding the research? This study is organised jointly by Universitat Pompeu Fabra and the University of Sheffield at the CheckThat! Lab at CLEF 2024. Universitat Pompeu Fabra is funded by the European Union's Horizon Europe research and innovation programme under grant agreement No 101060930. The University of Sheffield is funded by the the UK's innovation agency (Innovate UK) grant 10039055 (approved under the Horizon Europe Programme as vera.ai EU grant agreement 101070093).</p><p>Who has ethically reviewed this study? This project has been ethically approved via the University of Sheffield's Ethics Review Procedure, as administered by the Computer Science Department.</p><p>What if something goes wrong and I wish to complain about the research? If you have any complaints, either from the researcher or something occurring during or following your participation in the project (e.g. a reportable serious adverse event), please contact Dr. Xingyi Song (contact details in section 10). Should you feel your complaint has not been handled to your satisfaction, you can also contact the Head of Department at the University of Sheffield, Professor Heidi Christensen (heidi.christensen@sheffield.ac.uk) who will then escalate the complaint through the appropriate channels. If the complaint relates to how your personal data has been handled, information about how to raise a complaint can be found in the University's Privacy Notice: https://www.sheffield.ac.uk/ govern/data-protection/privacy/general.  </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Contact for further information</head></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>B. COVID-19 Misinformation Dataset Examples</head></div>			</div>
			<div type="references">

				<listBibl>

<biblStruct xml:id="b0">
	<monogr>
		<author>
			<persName><forename type="first">J</forename><forename type="middle">A</forename><surname>Tucker</surname></persName>
		</author>
		<author>
			<persName><forename type="first">A</forename><surname>Guess</surname></persName>
		</author>
		<author>
			<persName><forename type="first">P</forename><surname>Barberá</surname></persName>
		</author>
		<author>
			<persName><forename type="first">C</forename><surname>Vaccari</surname></persName>
		</author>
		<author>
			<persName><forename type="first">A</forename><surname>Siegel</surname></persName>
		</author>
		<author>
			<persName><forename type="first">S</forename><surname>Sanovich</surname></persName>
		</author>
		<author>
			<persName><forename type="first">D</forename><surname>Stukal</surname></persName>
		</author>
		<author>
			<persName><forename type="first">B</forename><surname>Nyhan</surname></persName>
		</author>
		<ptr target="https://hewlett.org/library/social-media-political-polarization-political-disinformation-review-scientific-literature/" />
		<title level="m">Social Media, Political Polarization, and Political Disinformation: A Review of the Scientific Literature</title>
				<imprint>
			<date type="published" when="2018">2018</date>
		</imprint>
		<respStmt>
			<orgName>Hewlett Foundation</orgName>
		</respStmt>
	</monogr>
	<note type="report_type">Technical Report</note>
</biblStruct>

<biblStruct xml:id="b1">
	<analytic>
		<title level="a" type="main">Misinformation: susceptibility, spread, and interventions to immunize the public</title>
		<author>
			<persName><forename type="first">S</forename><surname>Van Der Linden</surname></persName>
		</author>
		<idno type="DOI">10.1038/s41591-022-01713-6</idno>
		<ptr target="https://www.nature.com/articles/s41591-022-01713-6.doi:10.1038/s41591-022-01713-6" />
	</analytic>
	<monogr>
		<title level="j">Nature Medicine</title>
		<imprint>
			<biblScope unit="volume">28</biblScope>
			<biblScope unit="page" from="460" to="467" />
			<date type="published" when="2022">2022. 2022</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b2">
	<analytic>
		<title level="a" type="main">SemEval-2019 Task 4: Hyperpartisan News Detection</title>
		<author>
			<persName><forename type="first">J</forename><surname>Kiesel</surname></persName>
		</author>
		<author>
			<persName><forename type="first">M</forename><surname>Mestre</surname></persName>
		</author>
		<author>
			<persName><forename type="first">R</forename><surname>Shukla</surname></persName>
		</author>
		<author>
			<persName><forename type="first">E</forename><surname>Vincent</surname></persName>
		</author>
		<author>
			<persName><forename type="first">P</forename><surname>Adineh</surname></persName>
		</author>
		<author>
			<persName><forename type="first">D</forename><surname>Corney</surname></persName>
		</author>
		<author>
			<persName><forename type="first">B</forename><surname>Stein</surname></persName>
		</author>
		<author>
			<persName><forename type="first">M</forename><surname>Potthast</surname></persName>
		</author>
		<idno type="DOI">10.18653/v1/S19-2145</idno>
		<ptr target="https://aclanthology.org/S19-2145.doi:10.18653/v1/S19-2145" />
	</analytic>
	<monogr>
		<title level="m">Proceedings of the 13th International Workshop on Semantic Evaluation, Association for Computational Linguistics</title>
				<meeting>the 13th International Workshop on Semantic Evaluation, Association for Computational Linguistics<address><addrLine>Minneapolis, Minnesota, USA</addrLine></address></meeting>
		<imprint>
			<date type="published" when="2019">2019</date>
			<biblScope unit="page" from="829" to="839" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b3">
	<analytic>
		<title level="a" type="main">Task 11: Detection of Propaganda Techniques in News Articles</title>
		<author>
			<persName><forename type="first">G</forename><surname>Da San Martino</surname></persName>
		</author>
		<author>
			<persName><forename type="first">A</forename><surname>Barrón-Cedeño</surname></persName>
		</author>
		<author>
			<persName><forename type="first">H</forename><surname>Wachsmuth</surname></persName>
		</author>
		<author>
			<persName><forename type="first">R</forename><surname>Petrov</surname></persName>
		</author>
		<author>
			<persName><forename type="first">P</forename><surname>Nakov</surname></persName>
		</author>
		<ptr target="http://propaganda.qcri.org/annotations/definitions.htmlhttp://arxiv.org/abs/2009.02696.arXiv:2009.02696" />
	</analytic>
	<monogr>
		<title level="m">Proceedings of the Fourteenth Workshop on Semantic Evaluation</title>
				<meeting>the Fourteenth Workshop on Semantic Evaluation<address><addrLine>SemEval-</addrLine></address></meeting>
		<imprint>
			<date type="published" when="2020">2020. 2020. 2020</date>
			<biblScope unit="page" from="1377" to="1414" />
		</imprint>
	</monogr>
	<note>SemEval-</note>
</biblStruct>

<biblStruct xml:id="b4">
	<analytic>
		<title level="a" type="main">Overview of the 7th Author Profiling Task at PAN 2019: Bots and Gender Profiling</title>
		<author>
			<persName><forename type="first">F</forename><surname>Rangel</surname></persName>
		</author>
		<author>
			<persName><forename type="first">P</forename><surname>Rosso</surname></persName>
		</author>
		<ptr target=".org" />
	</analytic>
	<monogr>
		<title level="m">CLEF 2019 Labs and Workshops</title>
		<title level="s">Notebook Papers. CEUR Workshop Proceedings</title>
		<editor>
			<persName><forename type="first">L</forename><surname>Cappellato</surname></persName>
		</editor>
		<editor>
			<persName><forename type="first">N</forename><surname>Ferro</surname></persName>
		</editor>
		<editor>
			<persName><forename type="first">D</forename><forename type="middle">E</forename><surname>Losada</surname></persName>
		</editor>
		<editor>
			<persName><forename type="first">H</forename><surname>Müller</surname></persName>
		</editor>
		<imprint>
			<publisher>CEUR-WS</publisher>
			<date type="published" when="2019">2019</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b5">
	<analytic>
		<title level="a" type="main">The FEVER2.0 Shared Task</title>
		<author>
			<persName><forename type="first">J</forename><surname>Thorne</surname></persName>
		</author>
		<author>
			<persName><forename type="first">A</forename><surname>Vlachos</surname></persName>
		</author>
		<author>
			<persName><forename type="first">O</forename><surname>Cocarascu</surname></persName>
		</author>
		<author>
			<persName><forename type="first">C</forename><surname>Christodoulopoulos</surname></persName>
		</author>
		<author>
			<persName><forename type="first">A</forename><surname>Mittal</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER)</title>
				<meeting>the Second Workshop on Fact Extraction and VERification (FEVER)</meeting>
		<imprint>
			<date type="published" when="2018">2018</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b6">
	<analytic>
		<title level="a" type="main">SoK: Content Moderation in Social Media, from Guidelines to Enforcement, and Research to Practice</title>
		<author>
			<persName><forename type="first">M</forename><surname>Singhal</surname></persName>
		</author>
		<author>
			<persName><forename type="first">C</forename><surname>Ling</surname></persName>
		</author>
		<author>
			<persName><forename type="first">P</forename><surname>Paudel</surname></persName>
		</author>
		<author>
			<persName><forename type="first">P</forename><surname>Thota</surname></persName>
		</author>
		<author>
			<persName><forename type="first">N</forename><surname>Kumarswamy</surname></persName>
		</author>
		<author>
			<persName><forename type="first">G</forename><surname>Stringhini</surname></persName>
		</author>
		<author>
			<persName><forename type="first">S</forename><surname>Nilizadeh</surname></persName>
		</author>
		<idno type="DOI">10.48550/arxiv.2206.14855</idno>
		<idno type="arXiv">arXiv:2206.14855</idno>
		<ptr target="https://arxiv.org/abs/2206.14855v2.doi:10.48550/arxiv.2206.14855" />
	</analytic>
	<monogr>
		<title level="m">The 8th IEEE European Symposium on Security and Privacy (EuroS&amp;P 2023)</title>
				<imprint>
			<publisher>IEEE</publisher>
			<date type="published" when="2022">2022</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b7">
	<monogr>
		<author>
			<persName><forename type="first">C</forename><surname>Szegedy</surname></persName>
		</author>
		<author>
			<persName><forename type="first">W</forename><surname>Zaremba</surname></persName>
		</author>
		<author>
			<persName><forename type="first">I</forename><surname>Sutskever</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><surname>Bruna</surname></persName>
		</author>
		<author>
			<persName><forename type="first">D</forename><surname>Erhan</surname></persName>
		</author>
		<author>
			<persName><forename type="first">I</forename><surname>Goodfellow</surname></persName>
		</author>
		<author>
			<persName><forename type="first">R</forename><surname>Fergus</surname></persName>
		</author>
		<idno type="DOI">10.48550/arxiv.1312.6199</idno>
		<idno type="arXiv">arXiv:1312.6199</idno>
		<idno>arXiv:1312.6199</idno>
		<ptr target="https://arxiv.org/abs/1312.6199v4.doi:10.48550/arxiv.1312.6199" />
		<title level="m">Intriguing properties of neural networks</title>
				<imprint>
			<date type="published" when="2013">2013</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b8">
	<analytic>
		<title level="a" type="main">Adversarial Attacks on Deep-learning Models in Natural Language Processing</title>
		<author>
			<persName><forename type="first">W</forename><forename type="middle">E</forename><surname>Zhang</surname></persName>
		</author>
		<author>
			<persName><forename type="first">Q</forename><forename type="middle">Z</forename><surname>Sheng</surname></persName>
		</author>
		<author>
			<persName><forename type="first">A</forename><surname>Alhazmi</surname></persName>
		</author>
		<author>
			<persName><forename type="first">C</forename><surname>Li</surname></persName>
		</author>
		<idno type="DOI">10.1145/3374217</idno>
		<ptr target="https://dl.acm.org/doi/10.1145/3374217.doi:10.1145/3374217" />
	</analytic>
	<monogr>
		<title level="j">ACM Transactions on Intelligent Systems and Technology (TIST)</title>
		<imprint>
			<biblScope unit="volume">11</biblScope>
			<date type="published" when="2020">2020</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b9">
	<analytic>
		<title level="a" type="main">The clef-2024 checkthat! lab: Check-worthiness, subjectivity, persuasion, roles, authorities, and adversarial robustness</title>
		<author>
			<persName><forename type="first">A</forename><surname>Barrón-Cedeño</surname></persName>
		</author>
		<author>
			<persName><forename type="first">F</forename><surname>Alam</surname></persName>
		</author>
		<author>
			<persName><forename type="first">T</forename><surname>Chakraborty</surname></persName>
		</author>
		<author>
			<persName><forename type="first">T</forename><surname>Elsayed</surname></persName>
		</author>
		<author>
			<persName><forename type="first">P</forename><surname>Nakov</surname></persName>
		</author>
		<author>
			<persName><forename type="first">P</forename><surname>Przybyła</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><forename type="middle">M</forename><surname>Struß</surname></persName>
		</author>
		<author>
			<persName><forename type="first">F</forename><surname>Haouari</surname></persName>
		</author>
		<author>
			<persName><forename type="first">M</forename><surname>Hasanain</surname></persName>
		</author>
		<author>
			<persName><forename type="first">F</forename><surname>Ruggeri</surname></persName>
		</author>
		<author>
			<persName><forename type="first">X</forename><surname>Song</surname></persName>
		</author>
		<author>
			<persName><forename type="first">R</forename><surname>Suwaileh</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">Advances in Information Retrieval</title>
				<editor>
			<persName><forename type="first">N</forename><surname>Goharian</surname></persName>
		</editor>
		<editor>
			<persName><forename type="first">N</forename><surname>Tonellotto</surname></persName>
		</editor>
		<editor>
			<persName><forename type="first">Y</forename><surname>He</surname></persName>
		</editor>
		<editor>
			<persName><forename type="first">A</forename><surname>Lipani</surname></persName>
		</editor>
		<editor>
			<persName><forename type="first">G</forename><surname>Mcdonald</surname></persName>
		</editor>
		<editor>
			<persName><forename type="first">C</forename><surname>Macdonald</surname></persName>
		</editor>
		<editor>
			<persName><forename type="first">I</forename><surname>Ounis</surname></persName>
		</editor>
		<meeting><address><addrLine>Nature Switzerland, Cham</addrLine></address></meeting>
		<imprint>
			<publisher>Springer</publisher>
			<date type="published" when="2024">2024</date>
			<biblScope unit="page" from="449" to="458" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b10">
	<analytic>
		<title level="a" type="main">Overview of the CLEF-2024 CheckThat! Lab: Check-worthiness, subjectivity, persuasion, roles, authorities and adversarial robustness</title>
		<author>
			<persName><forename type="first">A</forename><surname>Barrón-Cedeño</surname></persName>
		</author>
		<author>
			<persName><forename type="first">F</forename><surname>Alam</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><forename type="middle">M</forename><surname>Struß</surname></persName>
		</author>
		<author>
			<persName><forename type="first">P</forename><surname>Nakov</surname></persName>
		</author>
		<author>
			<persName><forename type="first">T</forename><surname>Chakraborty</surname></persName>
		</author>
		<author>
			<persName><forename type="first">T</forename><surname>Elsayed</surname></persName>
		</author>
		<author>
			<persName><forename type="first">P</forename><surname>Przybyła</surname></persName>
		</author>
		<author>
			<persName><forename type="first">T</forename><surname>Caselli</surname></persName>
		</author>
		<author>
			<persName><forename type="first">G</forename><surname>Da San Martino</surname></persName>
		</author>
		<author>
			<persName><forename type="first">F</forename><surname>Haouari</surname></persName>
		</author>
		<author>
			<persName><forename type="first">C</forename><surname>Li</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><surname>Piskorski</surname></persName>
		</author>
		<author>
			<persName><forename type="first">F</forename><surname>Ruggeri</surname></persName>
		</author>
		<author>
			<persName><forename type="first">X</forename><surname>Song</surname></persName>
		</author>
		<author>
			<persName><forename type="first">R</forename><surname>Suwaileh</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Fifteenth International Conference of the CLEF Association</title>
				<editor>
			<persName><forename type="first">L</forename><surname>Goeuriot</surname></persName>
		</editor>
		<editor>
			<persName><forename type="first">P</forename><surname>Mulhem</surname></persName>
		</editor>
		<editor>
			<persName><forename type="first">G</forename><surname>Quénot</surname></persName>
		</editor>
		<editor>
			<persName><forename type="first">D</forename><surname>Schwab</surname></persName>
		</editor>
		<editor>
			<persName><forename type="first">L</forename><surname>Soulier</surname></persName>
		</editor>
		<editor>
			<persName><forename type="first">G</forename><forename type="middle">M</forename><surname>Di Nunzio</surname></persName>
		</editor>
		<editor>
			<persName><forename type="first">P</forename><surname>Galuščáková</surname></persName>
		</editor>
		<editor>
			<persName><forename type="first">A</forename><surname>García Seco De Herrera</surname></persName>
		</editor>
		<editor>
			<persName><forename type="first">G</forename><surname>Faggioli</surname></persName>
		</editor>
		<editor>
			<persName><forename type="first">N</forename><surname>Ferro</surname></persName>
		</editor>
		<meeting><address><addrLine>CLEF</addrLine></address></meeting>
		<imprint>
			<date type="published" when="2024">2024. 2024</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b11">
	<monogr>
		<author>
			<persName><forename type="first">P</forename><surname>Przybyła</surname></persName>
		</author>
		<author>
			<persName><forename type="first">A</forename><surname>Shvets</surname></persName>
		</author>
		<author>
			<persName><forename type="first">H</forename><surname>Saggion</surname></persName>
		</author>
		<idno type="arXiv">arXiv:2303.08032</idno>
		<ptr target="https://arxiv.org/abs/2303.08032v1.arXiv:2303.08032" />
		<title level="m">Verifying the Robustness of Automatic Credibility Assessment</title>
				<imprint>
			<date type="published" when="2023">2023</date>
		</imprint>
	</monogr>
	<note type="report_type">arXiv preprint</note>
</biblStruct>

<biblStruct xml:id="b12">
	<analytic>
		<title level="a" type="main">OpenAttack: An Open-source Textual Adversarial Attack Toolkit</title>
		<author>
			<persName><forename type="first">G</forename><surname>Zeng</surname></persName>
		</author>
		<author>
			<persName><forename type="first">F</forename><surname>Qi</surname></persName>
		</author>
		<author>
			<persName><forename type="first">Q</forename><surname>Zhou</surname></persName>
		</author>
		<author>
			<persName><forename type="first">T</forename><surname>Zhang</surname></persName>
		</author>
		<author>
			<persName><forename type="first">Z</forename><surname>Ma</surname></persName>
		</author>
		<author>
			<persName><forename type="first">B</forename><surname>Hou</surname></persName>
		</author>
		<author>
			<persName><forename type="first">Y</forename><surname>Zang</surname></persName>
		</author>
		<author>
			<persName><forename type="first">Z</forename><surname>Liu</surname></persName>
		</author>
		<author>
			<persName><forename type="first">M</forename><surname>Sun</surname></persName>
		</author>
		<ptr target="https://aclanthology.org/2021.acl-demo" />
	</analytic>
	<monogr>
		<title level="m">ACL-IJCNLP 2021 -59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Proceedings of the System Demonstrations, Association for Computational Linguistics (ACL)</title>
				<imprint>
			<date type="published" when="2021">2021</date>
			<biblScope unit="page" from="363" to="371" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b13">
	<monogr>
		<title/>
		<idno type="DOI">10.18653/V1/2021.ACL-DEMO.43</idno>
		<idno type="arXiv">arXiv:2009.09191</idno>
		<imprint/>
	</monogr>
</biblStruct>

<biblStruct xml:id="b14">
	<analytic>
		<title level="a" type="main">This Just In: Fake News Packs a Lot in Title, Uses Simpler, Repetitive Content in Text Body, More Similar to Satire than Real News</title>
		<author>
			<persName><forename type="first">B</forename><forename type="middle">D</forename><surname>Horne</surname></persName>
		</author>
		<author>
			<persName><forename type="first">S</forename><surname>Adali</surname></persName>
		</author>
		<ptr target="http://arxiv.org/abs/1703.09398.arXiv:1703.09398" />
	</analytic>
	<monogr>
		<title level="m">Proceedings of the 2nd International Workshop on News and Public Opinion at ICWSM, Association for the Advancement of Artificial Intelligence</title>
				<meeting>the 2nd International Workshop on News and Public Opinion at ICWSM, Association for the Advancement of Artificial Intelligence</meeting>
		<imprint>
			<date type="published" when="2017">2017</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b15">
	<analytic>
		<title level="a" type="main">Capturing the Style of Fake News</title>
		<author>
			<persName><forename type="first">P</forename><surname>Przybyła</surname></persName>
		</author>
		<idno type="DOI">10.1609/aaai.v34i01.5386</idno>
		<ptr target="https://aaai.org/ojs/index.php/AAAI/article/view/5386.doi:10.1609/aaai.v34i01.5386" />
	</analytic>
	<monogr>
		<title level="m">Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20)</title>
				<meeting>the Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20)<address><addrLine>New York, USA</addrLine></address></meeting>
		<imprint>
			<publisher>AAAI Press</publisher>
			<date type="published" when="2020">2020</date>
			<biblScope unit="volume">34</biblScope>
			<biblScope unit="page" from="490" to="497" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b16">
	<monogr>
		<title level="m" type="main">Propaganda: A Pluralistic Perspective</title>
		<author>
			<persName><forename type="first">T</forename><forename type="middle">J</forename><surname>Smith</surname></persName>
		</author>
		<imprint>
			<date type="published" when="1989">1989</date>
			<publisher>Praeger</publisher>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b17">
	<monogr>
		<title level="m" type="main">Understanding the Promise and Limits of Automated Fact-Checking</title>
		<author>
			<persName><forename type="first">L</forename><surname>Graves</surname></persName>
		</author>
		<ptr target="https://reutersinstitute.politics.ox.ac.uk/sites/default/files/2018-02/graves{_}factsheet{_}180226FINAL.pdf.arXiv:arXiv:1011.1669v3" />
		<imprint>
			<date type="published" when="2018">2018</date>
		</imprint>
		<respStmt>
			<orgName>Reuters Institute, University of Oxford</orgName>
		</respStmt>
	</monogr>
	<note type="report_type">Technical Report</note>
</biblStruct>

<biblStruct xml:id="b18">
	<analytic>
		<title level="a" type="main">The Fact Extraction and VERification (FEVER) Shared Task</title>
		<author>
			<persName><forename type="first">J</forename><surname>Thorne</surname></persName>
		</author>
		<author>
			<persName><forename type="first">A</forename><surname>Vlachos</surname></persName>
		</author>
		<author>
			<persName><forename type="first">O</forename><surname>Cocarascu</surname></persName>
		</author>
		<author>
			<persName><forename type="first">C</forename><surname>Christodoulopoulos</surname></persName>
		</author>
		<author>
			<persName><forename type="first">A</forename><surname>Mittal</surname></persName>
		</author>
		<idno type="arXiv">arXiv:1811.10971v1</idno>
	</analytic>
	<monogr>
		<title level="m">Proceedings of the First Workshop on Fact Extraction and VERification (FEVER)</title>
				<meeting>the First Workshop on Fact Extraction and VERification (FEVER)</meeting>
		<imprint>
			<date type="published" when="2018">2018</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b19">
	<analytic>
		<title level="a" type="main">Deep learning-based rumor detection on microblogging platforms: A systematic review</title>
		<author>
			<persName><forename type="first">M</forename><surname>Al-Sarem</surname></persName>
		</author>
		<author>
			<persName><forename type="first">W</forename><surname>Boulila</surname></persName>
		</author>
		<author>
			<persName><forename type="first">M</forename><surname>Al-Harby</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><surname>Qadir</surname></persName>
		</author>
		<author>
			<persName><forename type="first">A</forename><surname>Alsaeedi</surname></persName>
		</author>
		<idno type="DOI">10.1109/ACCESS.2019.2947855</idno>
	</analytic>
	<monogr>
		<title level="j">IEEE Access</title>
		<imprint>
			<biblScope unit="volume">7</biblScope>
			<biblScope unit="page" from="152788" to="152812" />
			<date type="published" when="2019">2019</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b20">
	<analytic>
		<title level="a" type="main">Neural language model based training data augmentation for weakly supervised early rumor detection</title>
		<author>
			<persName><forename type="first">S</forename><surname>Han</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><surname>Gao</surname></persName>
		</author>
		<author>
			<persName><forename type="first">F</forename><surname>Ciravegna</surname></persName>
		</author>
		<idno type="DOI">10.1145/3341161.3342892</idno>
		<idno type="arXiv">arXiv:1907.07033</idno>
		<ptr target="10.1145/3341161.3342892" />
	</analytic>
	<monogr>
		<title level="m">Proceedings of the 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2019</title>
				<meeting>the 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2019</meeting>
		<imprint>
			<publisher>Association for Computing Machinery, Inc</publisher>
			<date type="published" when="2019">2019</date>
			<biblScope unit="page" from="105" to="112" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b21">
	<analytic>
		<title level="a" type="main">Categorising fine-to-coarse grained misinformation: An empirical study of the covid-19 infodemic</title>
		<author>
			<persName><forename type="first">Y</forename><surname>Jiang</surname></persName>
		</author>
		<author>
			<persName><forename type="first">X</forename><surname>Song</surname></persName>
		</author>
		<author>
			<persName><forename type="first">C</forename><surname>Scarton</surname></persName>
		</author>
		<author>
			<persName><forename type="first">I</forename><surname>Singh</surname></persName>
		</author>
		<author>
			<persName><forename type="first">A</forename><surname>Aker</surname></persName>
		</author>
		<author>
			<persName><forename type="first">K</forename><surname>Bontcheva</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing</title>
				<meeting>the 14th International Conference on Recent Advances in Natural Language Processing</meeting>
		<imprint>
			<date type="published" when="2023">2023</date>
			<biblScope unit="page" from="556" to="567" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b22">
	<monogr>
		<author>
			<persName><forename type="first">Y</forename><surname>Mu</surname></persName>
		</author>
		<author>
			<persName><forename type="first">Y</forename><surname>Jiang</surname></persName>
		</author>
		<author>
			<persName><forename type="first">F</forename><surname>Heppell</surname></persName>
		</author>
		<author>
			<persName><forename type="first">I</forename><surname>Singh</surname></persName>
		</author>
		<author>
			<persName><forename type="first">C</forename><surname>Scarton</surname></persName>
		</author>
		<author>
			<persName><forename type="first">K</forename><surname>Bontcheva</surname></persName>
		</author>
		<author>
			<persName><forename type="first">X</forename><surname>Song</surname></persName>
		</author>
		<idno type="arXiv">arXiv:2304.04811</idno>
		<title level="m">A large-scale comparative study of accurate covid-19 information versus misinformation</title>
				<imprint>
			<date type="published" when="2023">2023</date>
		</imprint>
	</monogr>
	<note type="report_type">arXiv preprint</note>
</biblStruct>

<biblStruct xml:id="b23">
	<analytic>
		<title level="a" type="main">Long Short-Term Memory</title>
		<author>
			<persName><forename type="first">S</forename><surname>Hochreiter</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><surname>Schmidhuber</surname></persName>
		</author>
		<idno type="DOI">10.1162/neco.1997.9.8.1735</idno>
	</analytic>
	<monogr>
		<title level="j">Neural Computation</title>
		<imprint>
			<biblScope unit="volume">9</biblScope>
			<biblScope unit="page" from="1735" to="1780" />
			<date type="published" when="1997">1997</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b24">
	<analytic>
		<title level="a" type="main">BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding</title>
		<author>
			<persName><forename type="first">J</forename><surname>Devlin</surname></persName>
		</author>
		<author>
			<persName><forename type="first">M.-W</forename><surname>Chang</surname></persName>
		</author>
		<author>
			<persName><forename type="first">K</forename><surname>Lee</surname></persName>
		</author>
		<author>
			<persName><forename type="first">K</forename><surname>Toutanova</surname></persName>
		</author>
		<ptr target="http://arxiv.org/abs/1810.04805.arXiv:1810.04805" />
	</analytic>
	<monogr>
		<title level="m">Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics</title>
				<meeting>the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics</meeting>
		<imprint>
			<date type="published" when="2018">2018</date>
			<biblScope unit="page" from="4171" to="4186" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b25">
	<analytic>
		<title level="a" type="main">Transformers: State-of-the-Art Natural Language Processing</title>
		<author>
			<persName><forename type="first">T</forename><surname>Wolf</surname></persName>
		</author>
		<author>
			<persName><forename type="first">L</forename><surname>Debut</surname></persName>
		</author>
		<author>
			<persName><forename type="first">V</forename><surname>Sanh</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><surname>Chaumond</surname></persName>
		</author>
		<author>
			<persName><forename type="first">C</forename><surname>Delangue</surname></persName>
		</author>
		<author>
			<persName><forename type="first">A</forename><surname>Moi</surname></persName>
		</author>
		<author>
			<persName><forename type="first">P</forename><surname>Cistac</surname></persName>
		</author>
		<author>
			<persName><forename type="first">T</forename><surname>Rault</surname></persName>
		</author>
		<author>
			<persName><forename type="first">R</forename><surname>Louf</surname></persName>
		</author>
		<author>
			<persName><forename type="first">M</forename><surname>Funtowicz</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><surname>Davison</surname></persName>
		</author>
		<author>
			<persName><forename type="first">S</forename><surname>Shleifer</surname></persName>
		</author>
		<author>
			<persName><forename type="first">P</forename><surname>Von Platen</surname></persName>
		</author>
		<author>
			<persName><forename type="first">C</forename><surname>Ma</surname></persName>
		</author>
		<author>
			<persName><forename type="first">Y</forename><surname>Jernite</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><surname>Plu</surname></persName>
		</author>
		<author>
			<persName><forename type="first">C</forename><surname>Xu</surname></persName>
		</author>
		<author>
			<persName><forename type="first">T</forename><forename type="middle">L</forename><surname>Scao</surname></persName>
		</author>
		<author>
			<persName><forename type="first">S</forename><surname>Gugger</surname></persName>
		</author>
		<author>
			<persName><forename type="first">M</forename><surname>Drame</surname></persName>
		</author>
		<author>
			<persName><forename type="first">Q</forename><surname>Lhoest</surname></persName>
		</author>
		<author>
			<persName><forename type="first">A</forename><forename type="middle">M</forename><surname>Rush</surname></persName>
		</author>
		<ptr target="https://www.aclweb.org/anthology/2020.emnlp-demos.6" />
	</analytic>
	<monogr>
		<title level="m">Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics</title>
				<meeting>the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics</meeting>
		<imprint>
			<date type="published" when="2020">2020</date>
			<biblScope unit="page" from="38" to="45" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b26">
	<analytic>
		<title level="a" type="main">Decoupled Weight Decay Regularization</title>
		<author>
			<persName><forename type="first">I</forename><surname>Loshchilov</surname></persName>
		</author>
		<author>
			<persName><forename type="first">F</forename><surname>Hutter</surname></persName>
		</author>
		<ptr target="https://openreview.net/forum?id=Bkg6RiCqY7.arXiv:1711.05101v3" />
	</analytic>
	<monogr>
		<title level="m">7th International Conference on Learning Representations, ICLR 2019</title>
				<meeting><address><addrLine>New Orleans, LA, USA</addrLine></address></meeting>
		<imprint>
			<date type="published" when="2019">2019</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b27">
	<monogr>
		<title level="m" type="main">RoBERTa: A Robustly Optimized BERT Pretraining Approach</title>
		<author>
			<persName><forename type="first">Y</forename><surname>Liu</surname></persName>
		</author>
		<author>
			<persName><forename type="first">M</forename><surname>Ott</surname></persName>
		</author>
		<author>
			<persName><forename type="first">N</forename><surname>Goyal</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><surname>Du</surname></persName>
		</author>
		<author>
			<persName><forename type="first">M</forename><surname>Joshi</surname></persName>
		</author>
		<author>
			<persName><forename type="first">D</forename><surname>Chen</surname></persName>
		</author>
		<author>
			<persName><forename type="first">O</forename><surname>Levy</surname></persName>
		</author>
		<author>
			<persName><forename type="first">M</forename><surname>Lewis</surname></persName>
		</author>
		<author>
			<persName><forename type="first">L</forename><surname>Zettlemoyer</surname></persName>
		</author>
		<author>
			<persName><forename type="first">V</forename><surname>Stoyanov</surname></persName>
		</author>
		<author>
			<persName><forename type="first">P</forename><forename type="middle">G</forename><surname>Allen</surname></persName>
		</author>
		<idno type="DOI">10.48550/arxiv.1907.11692</idno>
		<idno type="arXiv">arXiv:1907.11692</idno>
		<ptr target="https://arxiv.org/abs/1907.11692v1.doi:10.48550/arxiv.1907.11692" />
		<imprint>
			<date type="published" when="2019">2019</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b28">
	<analytic>
		<title level="a" type="main">BERT-ATTACK: Adversarial Attack Against BERT Using BERT</title>
		<author>
			<persName><forename type="first">L</forename><surname>Li</surname></persName>
		</author>
		<author>
			<persName><forename type="first">R</forename><surname>Ma</surname></persName>
		</author>
		<author>
			<persName><forename type="first">Q</forename><surname>Guo</surname></persName>
		</author>
		<author>
			<persName><forename type="first">X</forename><surname>Xue</surname></persName>
		</author>
		<author>
			<persName><forename type="first">X</forename><surname>Qiu</surname></persName>
		</author>
		<idno type="DOI">10.18653/v1/2020.emnlp-main.500</idno>
		<ptr target="https://aclanthology.org/2020.emnlp-main.500.doi:10.18653/v1/2020.emnlp-main.500" />
	</analytic>
	<monogr>
		<title level="m">Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics</title>
				<meeting>the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics</meeting>
		<imprint>
			<date type="published" when="2020">2020</date>
			<biblScope unit="page" from="6193" to="6202" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b29">
	<analytic>
		<title level="a" type="main">BLEURT: Learning Robust Metrics for Text Generation</title>
		<author>
			<persName><forename type="first">T</forename><surname>Sellam</surname></persName>
		</author>
		<author>
			<persName><forename type="first">D</forename><surname>Das</surname></persName>
		</author>
		<author>
			<persName><forename type="first">A</forename><surname>Parikh</surname></persName>
		</author>
		<idno type="DOI">10.18653/v1/2020.acl-main.704</idno>
		<ptr target="https://aclanthology.org/2020.acl-main.704.doi:10.18653/v1/2020.acl-main.704" />
	</analytic>
	<monogr>
		<title level="m">Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics</title>
				<meeting>the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics</meeting>
		<imprint>
			<date type="published" when="2020">2020</date>
			<biblScope unit="page" from="7881" to="7892" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b30">
	<analytic>
		<title level="a" type="main">Binary codes capable of correcting deletions, insertions, and reversals</title>
		<author>
			<persName><forename type="first">V</forename><forename type="middle">I</forename><surname>Levenshtein</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="j">Soviet Physics Doklady</title>
		<imprint>
			<biblScope unit="volume">10</biblScope>
			<biblScope unit="page" from="707" to="710" />
			<date type="published" when="1966">1966</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b31">
	<analytic>
		<title level="a" type="main">Vaxxhesitancy: A dataset for studying hesitancy towards covid-19 vaccination on twitter</title>
		<author>
			<persName><forename type="first">Y</forename><surname>Mu</surname></persName>
		</author>
		<author>
			<persName><forename type="first">M</forename><surname>Jin</surname></persName>
		</author>
		<author>
			<persName><forename type="first">C</forename><surname>Grimshaw</surname></persName>
		</author>
		<author>
			<persName><forename type="first">C</forename><surname>Scarton</surname></persName>
		</author>
		<author>
			<persName><forename type="first">K</forename><surname>Bontcheva</surname></persName>
		</author>
		<author>
			<persName><forename type="first">X</forename><surname>Song</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">Proceedings of the International AAAI Conference on Web and Social Media</title>
				<meeting>the International AAAI Conference on Web and Social Media</meeting>
		<imprint>
			<date type="published" when="2023">2023</date>
			<biblScope unit="volume">17</biblScope>
			<biblScope unit="page" from="1052" to="1062" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b32">
	<analytic>
		<title level="a" type="main">GATE Teamware 2: An opensource tool for collaborative document classification annotation</title>
		<author>
			<persName><forename type="first">D</forename><surname>Wilby</surname></persName>
		</author>
		<author>
			<persName><forename type="first">T</forename><surname>Karmakharm</surname></persName>
		</author>
		<author>
			<persName><forename type="first">I</forename><surname>Roberts</surname></persName>
		</author>
		<author>
			<persName><forename type="first">X</forename><surname>Song</surname></persName>
		</author>
		<author>
			<persName><forename type="first">K</forename><surname>Bontcheva</surname></persName>
		</author>
		<idno type="DOI">10.18653/v1/2023.eacl-demo.17</idno>
		<ptr target="https://aclanthology.org/2023.eacl-demo.17.doi:10.18653/v1/2023.eacl-demo.17" />
	</analytic>
	<monogr>
		<title level="m">Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, Association for Computational Linguistics</title>
				<editor>
			<persName><forename type="first">D</forename><surname>Croce</surname></persName>
		</editor>
		<editor>
			<persName><forename type="first">L</forename><surname>Soldaini</surname></persName>
		</editor>
		<meeting>the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, Association for Computational Linguistics<address><addrLine>Dubrovnik, Croatia</addrLine></address></meeting>
		<imprint>
			<date type="published" when="2023">2023</date>
			<biblScope unit="page" from="145" to="151" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b33">
	<monogr>
		<title level="m" type="main">SINAI at CheckThat! 2024: Stealthy character-level adversarial attacks using homoglyphs and search, iterative</title>
		<author>
			<persName><forename type="first">J</forename><surname>Valle Aguilera</surname></persName>
		</author>
		<author>
			<persName><forename type="first">A</forename><forename type="middle">J</forename><surname>Gutiérrez Megías</surname></persName>
		</author>
		<author>
			<persName><forename type="first">S</forename><forename type="middle">M</forename><surname>Jiménez Zafra</surname></persName>
		</author>
		<author>
			<persName><forename type="first">L</forename><forename type="middle">A</forename><surname>Ureña López</surname></persName>
		</author>
		<author>
			<persName><forename type="first">E</forename><forename type="middle">Martínez</forename><surname>Cámara</surname></persName>
		</author>
		<imprint>
			<date type="published" when="2024">2024</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b34">
	<analytic>
		<title level="a" type="main">MMU NLP at CheckThat!</title>
		<author>
			<persName><forename type="first">C</forename><surname>Roadhouse</surname></persName>
		</author>
		<author>
			<persName><forename type="first">M</forename><surname>Shardlow</surname></persName>
		</author>
		<author>
			<persName><forename type="first">A</forename><surname>Williams</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">Homoglyphs are adversarial attacks</title>
				<imprint>
			<date type="published" when="2024">2024. 2024</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b35">
	<monogr>
		<title level="m" type="main">Palöri at CheckThat! 2024 shared task 6: Glota -combining glove embeddings with roberta for adversarial attack</title>
		<author>
			<persName><forename type="first">H</forename><surname>He</surname></persName>
		</author>
		<author>
			<persName><forename type="first">Y</forename><surname>Song</surname></persName>
		</author>
		<author>
			<persName><forename type="first">D</forename><surname>Massey</surname></persName>
		</author>
		<imprint>
			<date type="published" when="2024">2024</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b36">
	<monogr>
		<title level="m" type="main">OpenFact at CheckThat! 2024: Combining multiple attack methods for effective adversarial text generation</title>
		<author>
			<persName><forename type="first">W</forename><surname>Lewoniewski</surname></persName>
		</author>
		<author>
			<persName><forename type="first">P</forename><surname>Stolarski</surname></persName>
		</author>
		<author>
			<persName><forename type="first">M</forename><surname>Stróżyna</surname></persName>
		</author>
		<author>
			<persName><forename type="first">E</forename><surname>Lewańska</surname></persName>
		</author>
		<author>
			<persName><forename type="first">A</forename><surname>Wojewoda</surname></persName>
		</author>
		<author>
			<persName><forename type="first">E</forename><surname>Księżniak</surname></persName>
		</author>
		<author>
			<persName><forename type="first">M</forename><surname>Sawiński</surname></persName>
		</author>
		<imprint>
			<date type="published" when="2024">2024</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b37">
	<analytic>
		<title level="a" type="main">Generating Natural Language Adversarial Examples</title>
		<author>
			<persName><forename type="first">M</forename><surname>Alzantot</surname></persName>
		</author>
		<author>
			<persName><forename type="first">Y</forename><surname>Sharma</surname></persName>
		</author>
		<author>
			<persName><forename type="first">A</forename><surname>Elgohary</surname></persName>
		</author>
		<author>
			<persName><forename type="first">B.-J</forename><surname>Ho</surname></persName>
		</author>
		<author>
			<persName><forename type="first">M</forename><surname>Srivastava</surname></persName>
		</author>
		<author>
			<persName><forename type="first">K.-W</forename><surname>Chang</surname></persName>
		</author>
		<idno type="DOI">10.18653/v1/D18-1316</idno>
		<ptr target="https://aclanthology.org/D18-1316.doi:10.18653/v1/D18-1316" />
	</analytic>
	<monogr>
		<title level="m">Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics</title>
				<meeting>the 2018 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics<address><addrLine>Brussels, Belgium</addrLine></address></meeting>
		<imprint>
			<date type="published" when="2018">2018</date>
			<biblScope unit="page" from="2890" to="2896" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b38">
	<analytic>
		<title level="a" type="main">Counter-fitting word vectors to linguistic constraints</title>
		<author>
			<persName><forename type="first">N</forename><surname>Mrkšić</surname></persName>
		</author>
		<author>
			<persName><forename type="first">D</forename><forename type="middle">Ó</forename><surname>Séaghdha</surname></persName>
		</author>
		<author>
			<persName><forename type="first">B</forename><surname>Thomson</surname></persName>
		</author>
		<author>
			<persName><forename type="first">M</forename><surname>Gasic</surname></persName>
		</author>
		<author>
			<persName><forename type="first">L</forename><forename type="middle">M R</forename><surname>Barahona</surname></persName>
		</author>
		<author>
			<persName><forename type="first">P.-H</forename><surname>Su</surname></persName>
		</author>
		<author>
			<persName><forename type="first">D</forename><surname>Vandyke</surname></persName>
		</author>
		<author>
			<persName><forename type="first">T.-H</forename><surname>Wen</surname></persName>
		</author>
		<author>
			<persName><forename type="first">S</forename><surname>Young</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies</title>
				<meeting>the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies</meeting>
		<imprint>
			<date type="published" when="2016">2016</date>
			<biblScope unit="page" from="142" to="148" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b39">
	<analytic>
		<title level="a" type="main">Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment</title>
		<author>
			<persName><forename type="first">D</forename><surname>Jin</surname></persName>
		</author>
		<author>
			<persName><forename type="first">Z</forename><surname>Jin</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><forename type="middle">T</forename><surname>Zhou</surname></persName>
		</author>
		<author>
			<persName><forename type="first">P</forename><surname>Szolovits</surname></persName>
		</author>
		<ptr target="https://ojs.aaai.org/index.php/AAAI/article/view/6311" />
	</analytic>
	<monogr>
		<title level="m">The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020</title>
				<imprint>
			<publisher>AAAI Press</publisher>
			<date type="published" when="2020">2020</date>
			<biblScope unit="page" from="8018" to="8025" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b40">
	<analytic>
		<title level="a" type="main">TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP</title>
		<author>
			<persName><forename type="first">J</forename><surname>Morris</surname></persName>
		</author>
		<author>
			<persName><forename type="first">E</forename><surname>Lifland</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><forename type="middle">Y</forename><surname>Yoo</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><surname>Grigsby</surname></persName>
		</author>
		<author>
			<persName><forename type="first">D</forename><surname>Jin</surname></persName>
		</author>
		<author>
			<persName><forename type="first">Y</forename><surname>Qi</surname></persName>
		</author>
		<idno type="DOI">10.18653/v1/2020.emnlp-demos.16</idno>
		<ptr target="https://aclanthology.org/2020.emnlp-demos.16.doi:10.18653/v1/2020.emnlp-demos.16" />
	</analytic>
	<monogr>
		<title level="m">Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics</title>
				<meeting>the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics</meeting>
		<imprint>
			<date type="published" when="2020">2020</date>
			<biblScope unit="page" from="119" to="126" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b41">
	<analytic>
		<title level="a" type="main">Contextualized perturbation for textual adversarial attack</title>
		<author>
			<persName><forename type="first">D</forename><surname>Li</surname></persName>
		</author>
		<author>
			<persName><forename type="first">Y</forename><surname>Zhang</surname></persName>
		</author>
		<author>
			<persName><forename type="first">H</forename><surname>Peng</surname></persName>
		</author>
		<author>
			<persName><forename type="first">L</forename><surname>Chen</surname></persName>
		</author>
		<author>
			<persName><forename type="first">C</forename><surname>Brockett</surname></persName>
		</author>
		<author>
			<persName><forename type="first">M.-T</forename><surname>Sun</surname></persName>
		</author>
		<author>
			<persName><forename type="first">W</forename><forename type="middle">B</forename><surname>Dolan</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies</title>
				<meeting>the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies</meeting>
		<imprint>
			<date type="published" when="2021">2021</date>
			<biblScope unit="page" from="5053" to="5069" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b42">
	<monogr>
		<title level="m" type="main">TurQUaz at CheckThat! 2024: Creating adversarial examples using genetic algorithm</title>
		<author>
			<persName><forename type="first">B</forename><surname>Demirok</surname></persName>
		</author>
		<author>
			<persName><forename type="first">M</forename><surname>Kutlu</surname></persName>
		</author>
		<author>
			<persName><forename type="first">S</forename><surname>Mergen</surname></persName>
		</author>
		<author>
			<persName><forename type="first">B</forename><surname>Oz</surname></persName>
		</author>
		<imprint>
			<date type="published" when="2024">2024</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b43">
	<monogr>
		<title level="m" type="main">TextTrojaners at CheckThat! 2024: Robustness of credibility assessment with adversarial examples through beamattack</title>
		<author>
			<persName><forename type="first">D</forename><surname>Guzman Piedrahita</surname></persName>
		</author>
		<author>
			<persName><forename type="first">A</forename><surname>Fazla</surname></persName>
		</author>
		<author>
			<persName><forename type="first">L</forename><surname>Krauter</surname></persName>
		</author>
		<imprint>
			<date type="published" when="2024">2024</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b44">
	<analytic>
		<title level="a" type="main">why should i trust you?&quot; explaining the predictions of any classifier</title>
		<author>
			<persName><forename type="first">M</forename><forename type="middle">T</forename><surname>Ribeiro</surname></persName>
		</author>
		<author>
			<persName><forename type="first">S</forename><surname>Singh</surname></persName>
		</author>
		<author>
			<persName><forename type="first">C</forename><surname>Guestrin</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining</title>
				<meeting>the 22nd ACM SIGKDD international conference on knowledge discovery and data mining</meeting>
		<imprint>
			<date type="published" when="2016">2016</date>
			<biblScope unit="page" from="1135" to="1144" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b45">
	<monogr>
		<title level="m">Working Notes of CLEF 2024 -Conference and Labs of the Evaluation Forum, CLEF 2024</title>
				<editor>
			<persName><forename type="first">G</forename><surname>Faggioli</surname></persName>
		</editor>
		<editor>
			<persName><forename type="first">N</forename><surname>Ferro</surname></persName>
		</editor>
		<editor>
			<persName><forename type="first">P</forename><surname>Galuščáková</surname></persName>
		</editor>
		<editor>
			<persName><forename type="first">A</forename><surname>García Seco De Herrera</surname></persName>
		</editor>
		<meeting><address><addrLine>Grenoble, France</addrLine></address></meeting>
		<imprint>
			<date type="published" when="2024">2024</date>
		</imprint>
	</monogr>
</biblStruct>

				</listBibl>
			</div>
		</back>
	</text>
</TEI>
