Task 6 at CheckThat! Lab, organised at CLEF-2024, is devoted to assessing the robustness of misinformation detection solutions implemented as text classification models. The participants of the task were provided with prediction models and data examples for several problems of credibility estimation and their goal was to come up with adversarial examples (AEs): small modifications to the provided text fragments, such that the original meaning is preserved, but the victim classifier changes its decision. The evaluation involved five domains (detection of: biased news, propaganda techniques, false claims, rumours and COVID-19 misinformation) and three classifiers (BiLSTM, BERT and adversarially fine-tuned RoBERTa). Six teams participated in the task, representing a variety of approaches and substantially outperforming previous AE generation solutions. We also performed manual evaluation, which highlighted some modification techniques that are particularly likely to pass unnoticed by human readers. Overall, the task results emphasise the need to assess the robustness of text classification solutions before implementing them in content filtering on large platforms, such as social media.
The challenges of misinformation have been taken up with great energy and vigour by the NLP and IR
communities. The main reasons for such enthusiastic adoption of the new tasks are wide availability of
textual data to train on and tantalisingly simple dichotomy of fake vs. real, clearly fitting the familiar
task of binary classification. Among a great deal of work in the domain [
However, a shared task framework is far from the real-world application scenario, where the test
data are not fixed, but are generated continuously by users. This means that if a malicious actor sees
their non-credible content rejected by the system, they are likely to try to modify it to pass the filters,
rather than simply abandon their goals. Unfortunately, the deep learning architectures, which many of
the best-performing solutions use, are known for their susceptibility to adversarial examples, i.e. data
instances modified with the intent of fooling a classifier [
Here we report on the shared task on Investigating Robustness of Credibility Assessment with
Adversarial Examples (InCrediblAE), which was organised as Task 6 of the CheckThat! 2024 evaluation
lab [
The goal of the participants is to make modification to the instances in the attack dataset, turning them into adversarial examples. Each adversarial example (AE) is evaluated on meaning preservation, i.e. how similar it is to the original; and classifier confusion , i.e. whether the output of the victim classifier is diferent than for the original.
The evaluation consists of two stages. The automatic evaluation follows the framework established
in the field of adversarial learning, with the above factors assessed through automatic measures, i.e.
BODEGA score [
The task has attracted six teams submitting various approaches (section 4.1), some of which have clearly outperformed previous solutions applied to the same problems (section 4.2), generally confirming the high vulnerability of popular text classifiers to adversarial attacks. What is more, the results of the manual evaluation (section 4.3) highlight cases where AEs might appear far from the original text if judged by automatic measures, but are in fact quite convincing to human annotators. The results of the manual annotation are made openly available for future research1. We also share the code2 and data3 for the automatic evaluation.
The shared task uses the foundation of the BODEGA framework [
The participants are provided with Python code, hosted on Google Colab, allowing to interact with
the victim models in an attack scenario through OpenAttack interface [
Four of the domains in InCrediblAE have been prepared in BODEGA based on previously published corpora. The final one (C19) is new and was not available before.
Style-based news bias assessment (HN) is a task of verifying credibility of a news article based on its overall writing style. It relies on previous work indicating that stylistic analysis of fake news
outlets can be used to distinguish them from credible sources [
Propaganda detection (PR) is focused on recognising specific manipulation techniques appealing
to emotions [
Fact checking (FC) is an approach to misinformation detection based on extracting claims made in
a piece of text and verifying them with respect to a trusted knowledge base [
Rumour detection (RD) is aimed at detecting information spreading widely over social media
despite it not coming from a credible source. Rumours can be detected using many indicators [
COVID-19 misinformation detection (C19) focuses on binary classification 6 of misinformation
related to COVID-19 [
Table 1 summarises the information on datasets, including the sizes of subsets: training (for training victim classifiers), development (reserved for future use) and attack (to be modified into AEs), as well as the percentage of positive (non-credible) instances.
Training datasets were used to prepare victim models, representing popular approaches to text classification. Two of the models (BiLSTM and BERT) were trained as in BODEGA framework, but the surprise classifier was trained specifically for the shared task and it was revealed to the participants in the test phase, one week before the submissions.
BiLSTM classifier consists of an embedding layer (token representations of size 32), two LSTM [
BERT classifier is a bert-base-uncased model [
Surprise classifier is a RoBERTa model [
The evaluation procedure consists of two stages. Firstly, the BODEGA framework is used to automatically assess the attack efectiveness of each participant in 15 scenarios (5 domains × 3 victims). The average BODEGA score is used to create the leaderboard, expressing the overall performance. Secondly, the task most challenging for automatic evaluation (fact-checking) is used to perform manual annotation of meaning preservation in selected instances.
In automatic evaluation, when an example is modified into AE * , the quality of the transformation
is assessed through BODEGA score defined as follows [
BODEGA_score(, * ) = Con_score(, * ) × Sem_score(, * ) × Char_score(, * ), where: • Con_score, i.e. confusion score, takes value of 1 when the attacked classifier predicts a diferent class for * than it did for , and 0 otherwise. • Sem_score, i.e. semantic similarity score, is a measure of meaning preservation between and * , computed using the BLEURT [29] evaluation measure (BLEURT-20 variant), clipped to the (0-1) range. • Char_score, i.e. character similarity score, is a measure of similarity of and * as character sequences, computed through Levenshtein distance [30], scaled to (0-1) similarity.
We can see that an AE will be ranked highly if it changes the output of the classifier (Con_score(, * ) = 1), but at the same time preserves both the meaning (Sem_score(, * ) ≈ 1) and the appearance (Char_score(, * ) ≈ 1) of the original text.
To measure the overall attack success in a particular scenario, the BODEGA score averaged over all instances in the attack set is employed. However, the constituent scores, also averaged over the dataset, can be used to understand the results. We also report the number of queries that is needed (on average) for a single AE to be found.
Definitions Extremely unconfident about the annotation (I’m really unsure about the annotation. It may belong to another category as well, you may wish to discard this instance from the training.) Not confident about the annotation (I’m not sure about the annotation, it seems it also belongs to other categories, but you can still include this instance as a “silver standard instance” in training.) Pretty confident about the annotation (I’m pretty sure about the annotation, but might be in high chance other annotators may label it in a diferent category.) Fairly confident about the annotation (I’m confident about the annotation, but might be in small chance other annotators may label it in a diferent category.)
Extremely confident about the annotation (I’m certain about the annotation without a doubt.)
The goal of the manual evaluation is to highlight the cases where the automatic evaluation measures, especially regarding semantic similarity, might not be an accurate representation of human reception of the adversarially modified content. To that end, we have selected the samples from the fact-checking domain and the surprise victim, where even small changes in text can alter the meaning and, consequently, the credibility label.
Task Description We aim to gather assessments regarding the semantic similarities between attack samples and the original samples. Participants in the shared task are requested to dedicate approximately 60 minutes to this manual evaluation (i.e., 100 samples per participant), which are conducted using an open-source, collaborative annotation platform, i.e., GATE Teamware 2 [32]. Judges rate the sample pairs based on the following scale: (a). Preserve the Semantic Meaning, (b). Change the Semantic Meaning, and (c). No sense. Table 2 demonstrates categories and descriptions. Similar to the work of Mu et al. [31], annotators are required to indicate the confidence level (see Table 3) of their assigned class in the ‘Confidence Row’.
Data Sampling We randomly select 100 paired samples (i.e., original and modified texts) from each submission, resulting in a total of 600 paired samples. Note that only the successful attacks are considered.
Annotator Training We train the annotators by providing a training document detailing the annotation pipeline, which includes (i) a step-by-step tutorial for using the GATE Teamware platform, (ii) a user information sheet to inform about any potential issues and risks that may occur during data annotation, and (iii) a user consent sheet as required by the ethical approval from the University of Shefield, where the annotation was performed.
A total of 12 annotators (i.e., 6 participants and 6 organisers) were recruited to manually annotate the paired samples. These 12 annotators were further divided into 6 separate groups (i.e., two annotators per group). In each group, 100 tweets were assigned to each annotator. Finally, this process yielded 100 double-annotated paired samples from each group, resulting in 600 double-annotated samples in total.
task organizers and participants. Briefly, each paired sample is annotated by one participant and one shared task organiser. A third annotator from the shared-task organisers is used to resolve any conflicts. Given that there are three categories in total, the annotation with the highest confidence score will be considered in the case of three difering annotations.
Here we outline the results of the InCrediblAE shared task in three steps: first, we describe the solutions submitted by the participants (section 4.1), then we present the results of the automatic (section 4.2) and manual evaluation (section 4.3).
The SINAI team [33] proposes a method for adversarial attacks based on the substitution of characters by homoglyphs (e.g. characters which resemble the target such as l ≈ 1). The method uses exhaustive search with two variants: with memory and without memory. They ground their approach in the fact that homoglyphs could deceive the human eye while at the same time provoke a Large Language Model classifier to reverse its prediction due to the presence of an unexpected token. According to the oficial leader board, the approach is sub-optimal, obtaining the last rank in the task according to the oficial evaluation metrics. However, human evaluation of content preservation is ranked high.
The MMU_NLP participation [34] features a system to attack classifiers based on lexical substitution and character replacement. The proposed method searches for candidate words to attack followed by a word replacement mechanism. The word search mechanism masks words to check their vulnerability with those words having a high impact on the classifier performance retained for the attack. The replacement step uses homoglyphs for character replacement or lexical substitution. Character replacement is tested in two diferent conditions: random character attack or begin/end of word attack. The lexical replacement attack uses a large language model to retrieve a word similar to the target word. Overall result fall short compared to other participants, however the proposed methods improves over the baseline in several settings. The character attack method seems more efective than the word replacement approach.
The Palöri team [35] proposes an approach to identify vulnerable words by computing (relying on a masked language model) the diference between the probability distribution of the original sentence and the sentence with a word masked. These diferences are used as scores to rank words by their "vulnerability" according to the model. The ranked words (most to least vulnerable) are then replaced using a word from a list of substitutes proposed by the language model. The sentence with the replaced word is used to attack the victim classifier. In case of success, the new sentence is returned, otherwise the method loops using the sentence with the "best" possible substitution (i.e. one which reduced the victim’s confidence the most). The method produces successful attacks which however do not preserve the original sentence’s meaning. To address this problem a "synonym" dictionary is created, using GloVe embeddings and the aclImbd dataset, to draw substitutes from. The new method only contributes minor improvements over the masked language model.
The solution of the OpenFact team [36] consists in a coupling of various word-substitution approaches
in an ensemble in such a way that if the first approach does not succeed in changing the classifier’s
decision, then the second one is called. In particular, a modification of BERT-ATTACK [ 28] was
proposed (it features a change in parameter values, an alternative selection of a replacement position
by an exhaustive search of candidates that provide the largest diference in probabilities for a predicted
class, and an iterative replacement from 1 to 7 words, including punctuation and digits at the latter,
until the success of an attack) and backed up by a genetic algorithm [37] realised in the OpenAttack
framework [
The TurQUaz [
The TextTrojaners team [
Table 4 includes the results of the evaluation against the BiLSTM victim. Generally, we can see that diferent approaches dominate in diferent scenarios. However, in every domain, the best BODEGA score (in boldface) is achieved by a solution submitted to InCrediblAE, rather than a reference solution from previous work (BERT-ATTACK or DeepWordBug). HN appears to be the easiest domain, with the leading solution (TextTrojaners) achieving BODEGA score of 0.91 through 100% confusion with 91% semantic similarity and 99% character similarity. This result, closely followed by OpenFact, is especially impressive when compared to the scores of BERT-ATTACK (BODEGA score of 0.64). The TextTrojaners approach also leads in PR, but we need to note its high amount of queries needed – in this case, 593 compared to TurQUaz achieving almost the same result (0.68 instead of 0.70) with six times less queries. The C19 appears to be the domain most challenging for attacks, although even here we note a vast improvement over the reference method (OpenFact: 0.72 vs BERT-ATTACK: 0.50).
In the attacks against the BERT victim, evaluated in table 5, the OpenFact method dominates, ceding only in FC to homoglyph-based SINAI. We can also note that the best BODEGA score is either equivalent (for FC, HN and C19) or significantly lower (for PR and RD) than for BiLSTM, indicating higher dificulty of attacking a Transformer-based classifier. This also results in a higher number of queries necessary to
FC RD HN ifnd an AE, e.g. while the leading method for HN domain needed 937 queries to obtain the score of 0.91 with BiLSTM, attacking BERT with the same approach requires 4328 queries on average.
The results for the surprise victim (table 6) show that the adversarially-trained classifier indeed is more challenging to attack, leading to lower scores in PR and RD domains. However, the general view remains similar, with OpenFact again dominating with the exception of FC. Their approach clearly works very well with Transformer-based victims, just with an exception of this single task. The increasing level of dificulty is again reflected in a rising numbers of queries, reaching a record value of over 15000 for TextTrojaners attacking in the RD domain.
Finally, table 7 shows the final leaderboard obtained by averaging the BODEGA scores across FC RD HN victims and domains. We can see that all solutions submitted to the task have beaten the DeepWordBug reference and most have also outperformed BERT-ATTACK, which is a strong reference point. OpenFact and TextTrojaners are clear leaders, with the former slightly better-performing, especially against Transformers victim. However, we need to emphasise that the averaged ranking does not show the whole picture and various methods work best for various scenarios. For example, SINAI is the best approach for the BERT-FC combination.
We randomly selected 100 successful adversarial samples from each team’s submission to the scenario including fact checking and surprise victim. During the evaluation, both the original and adversarial samples were presented to the annotators, with diferences highlighted. The annotators were asked to categorise each sample pair into one of the three categories described in Section 3.2. They also provided a confidence score for each annotation (5: very confident, 1: not confident). Each sample pair was judged by at least two annotators. The annotation agreement of the initial annotators was 0.52 (Cohen’s Kappa). A third annotator was invited if there was a conflict between the two initial annotators.
We determined the ranking of participants by the number of samples that fell into the ’Preserve the Semantic Meaning’ category, adhering to the principle that a higher count indicates better performance. This ranking method was used because the task demands that the adversarial samples maintain their original meaning. Table 8 presents the final manual evaluation results for all participants.
In general, we observe we observe a discrepancy between the manual and automatic evaluations (see Table 7). This may be because the manual evaluation task is a fact-checking task. Even a slight replacement of named entities (such as changing the year from 1990 to 1991) could result in a change of meaning.
The leading team (SINAI) in manual evaluation uses an adversarial attack method mainly based on the substitution of characters with homoglyphs and achieves a 99% score. This suggests that the use of homoglyphs can successfully deceive the annotator’s eye. Similarly, team MMU employs similar attack approaches, such as lexical substitution and character replacement, which also achieve a high manual evaluation score (96%).
Team TurQUaz proposes a method of inserting white space to split English words, which achieved third place on the leaderboard (62%). However, this method may sometimes change the meaning of the original fact-checking document, resulting in a lower manual evaluation score compared to the methods proposed by teams SINAI and MMU. Besides, by inserting white space in the original text, the modified text may become non-interpretable by humans, resulting in a high proportion of third category submissions by Team TurQUaz, i.e., “The sentence does not make any sense.”
We observe that OpenFact (11%) and TextTrojaners (7%) obtain lower manual evaluation scores. By manually investigating text modified by Team OpenFact, we notice that some key information, such as time and location, has been changed. Note that such named entities play a vital role in the context of fact-checking downstream tasks. Therefore, changes to these named entities result in a higher number of samples labelled as “Change the Semantic Meaning.” As for the solution of TextTrojaners, the choice of alternative words solely depends on the context rather than the word to be replaced. This can significantly deviate the meaning. In addition, the operation of word removal without further word agreement adjustment may lead to nonsensical sentences.
The first conclusion from the results obtained is clear: the state of the art in AE generation for misinformation detection, established by previous solutions, has advanced considerably. Various solutions were submitted to the shared task, but they are based on the established lines of research in the area: word replacements (preserving meaning similarity) or character replacements (preserving visual similarity).
The word-level solutions (esp. TextTrojaners and OpenFact) were performing the best in most scenarios, but not all of them. Fact checking is a clear outlier due to its nature: every word matters, making it hard to perform any change without drastically afecting the meaning. This opens the avenue for character-level modifications and, indeed, such solution (SINAI) provided the best results in manual evaluation.
We also need to acknowledge the limitations of the evaluation setup. It aims to predict the likelihood of an AE fooling the victim classifier and transmitting the intended message, as encoded through BODEGA score. However, human readers are the intended recipients of misinformation, and they are also able to refuse to engage with a message that seems suspicious, artificial or distorted, e.g. due to use of letter with non-standard shapes. Thus, the success of AEs will also depend on the visual appearance of the manipulated content, which is not directly evaluated in the current setup. Quantifying this efect would be challenging in manual evaluation and even more so in an automatic setup.
In any case, the results that we do have leave one thing clear: popular architectures for text classification are very vulnerable to attack with AEs. While the adversarially-trained model posed slightly harder challenge, ultimately AEs were found for nearly all cases in these scenarios as well.
How can we protect the real-world deployments of text classifiers against such attack? The first barrier can be established by limiting the access to the victim model. We can see from the results that the advance over previous state of the art was accompanied by a raise in the number of queries sent, well into hundreds and thousands for each generated AE sample. Nevertheless, all machine-learning-based solutions for content filtering should be only deployed after a thorough analysis of their adversarial robustness.
In the InCrediblAE shared task, six teams participated with various solutions, both operating with the word-level and character-level changes. The participants’ approaches (and two reference solutions) were evaluated using five misinformation-detection scenarios and three victim models.
In total, 53,544 text modifications were considered and automatically assessed in terms of classifier confusion, meaning preservation and character similarity. The submitted solutions easily outperformed previous work in all of the tested scenarios. The manual evaluation highlighted the special role of the fact-checking tasks and the eficacy of character replacement in performing modification imperceptible to humans.
We hope that the combined efort of the participants and organisers of the InCrediblAE shared task will succeed in both highlighting the importance of robustness testing and showcasing the best solutions. To facilitate this outcome, the code and resources necessary for performing the automatic evaluation remain openly available.9
The work of P. Przybyła is part of the ERINIA project, which has received funding from the European Union’s Horizon Europe research and innovation programme under grant agreement No 101060930. This work has been also partially funded by the European Commission under contract numbers HE101070278 and ISF-101080090. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the funders. Neither the European Union nor the granting
authority can be held responsible for them. We also acknowledge support from Departament de Recerca i Universitats de la Generalitat de Catalunya (ajuts SGR-Cat 2021) and from Maria de Maeztu Units of Excellence Programme CEX2021-001195-M, funded by MCIN/AEI /10.13039/501100011033. P. G. Allen, RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019). URL: https: //arxiv.org/abs/1907.11692v1. doi:10.48550/arxiv.1907.11692. arXiv:1907.11692. [28] L. Li, R. Ma, Q. Guo, X. Xue, X. Qiu, BERT-ATTACK: Adversarial Attack Against BERT Using BERT, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, 2020, pp. 6193–6202. URL: https: //aclanthology.org/2020.emnlp-main.500. doi:10.18653/v1/2020.emnlp-main.500. [29] T. Sellam, D. Das, A. Parikh, BLEURT: Learning Robust Metrics for Text Generation, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online, 2020, pp. 7881–7892. URL: https://aclanthology.org/2020. acl-main.704. doi:10.18653/v1/2020.acl-main.704. [30] V. I. Levenshtein, Binary codes capable of correcting deletions, insertions, and reversals, Soviet
Physics Doklady 10 (1966) 707–710.
[31] Y. Mu, M. Jin, C. Grimshaw, C. Scarton, K. Bontcheva, X. Song, Vaxxhesitancy: A dataset for
studying hesitancy towards covid-19 vaccination on twitter, in: Proceedings of the International
AAAI Conference on Web and Social Media, volume 17, 2023, pp. 1052–1062.
[32] D. Wilby, T. Karmakharm, I. Roberts, X. Song, K. Bontcheva, GATE Teamware 2: An
opensource tool for collaborative document classification annotation, in: D. Croce, L. Soldaini
(Eds.), Proceedings of the 17th Conference of the European Chapter of the Association for
Computational Linguistics: System Demonstrations, Association for Computational
Linguistics, Dubrovnik, Croatia, 2023, pp. 145–151. URL: https://aclanthology.org/2023.eacl-demo.17.
doi:10.18653/v1/2023.eacl-demo.17.
[33] J. Valle Aguilera, A. J. Gutiérrez Megías, S. M. Jiménez Zafra, L. A. Ureña López, E. Martínez Cámara,
SINAI at CheckThat! 2024: Stealthy character-level adversarial attacks using homoglyphs and
search, iterative, in: [
Wen, S. Young, Counter-fitting word vectors to linguistic constraints, in: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2016, pp. 142–148. [39] D. Jin, Z. Jin, J. T. Zhou, P. Szolovits, Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment, in: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020„ AAAI Press, 2020, pp. 8018–8025. URL: https://ojs.aaai.org/ index.php/AAAI/article/view/6311. [40] J. Morris, E. Lifland, J. Y. Yoo, J. Grigsby, D. Jin, Y. Qi, TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, Online, 2020, pp. 119–126. URL: https://aclanthology. org/2020.emnlp-demos.16. doi:10.18653/v1/2020.emnlp-demos.16. [41] D. Li, Y. Zhang, H. Peng, L. Chen, C. Brockett, M.-T. Sun, W. B. Dolan, Contextualized perturbation for textual adversarial attack, in: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, pp. 5053–5069.
InCrediblAE Shared Task Annotation All participants who submit entries will have their submissions manually evaluated for semantic similarity. To aid in this human evaluation process, participants in the shared tasks will be required to manually assess a selection of the submitted examples. Manual Scoring We will gather assessments regarding the semantic similarities between attack samples and the original samples. Participants in the shared task are requested to dedicate approximately 60 mins to this manual evaluation (i.e., 100 samples per participant), which will be conducted using this online annotation tool. Judges will rate the sample pairs based on the following scale: (a). Preserve the Semantic Meaning, (b). Change the Semantic Meaning, (c). No sense.
Labels and Definitions (a). Preserve the Semantic Meaning: This label is used when the semantic content of the attack sample closely aligns with that of the original sample. Participants should use this label if the meaning, context, and intent of the compared texts remain essentially unchanged, indicating that the attack sample has efectively maintained the core message of the original. For example: Text Original: middle-earth. middle-earth is the fictional setting of much of british writer j. r. r. tolkien ’s legendarium . middle-earth is a setting.
Text Modified: middle-earth. middle-earth is the fictional background of much of british writer j. r. r. tolkien ’s legendarium . middle-earth is a setting.
(b). Change the Semantic Meaning: Participants should apply this label when there is a noticeable alteration in the semantic content between the original and the attack sample. This label indicates that while the attack sample may be related or similar in some aspects to the original, it diverges enough in meaning or intent to be considered distinct or modified. For example, name entities (including Year, Name, Location, etc.) are changed in the modified text. For example:
Text Original: carol danvers. the character has also been known as binary , warbird , and captain marvel at various points in her history , and has been featured in other marvel licensed products including video games , animated television series , and merchandise such as trading cards . carol danvers is a real person.
Text Modified: carol danvers. the character has also been known as binary , warbird , and captain marvel at various points in her history , and has been featured in other marvel licensed products including video games , animated television series , and merchandise such as trading cards . carol danvers is a fictional person.
(c). No sense: The sentence does not make any sense.
Confidence Scores Please indicate how confident you are in your assigned class in the Confidence Row. The confidence scores range from 1 to 5.
You are invited to participate in this research by contributing to the evaluation of the semantic similarity of adversarial examples. It is important for you to understand the goals of the task and what your participation will involve. Please take the time to read the following information. Please, ask us if there is anything that is not clear or if you would like more information. Thank you very much for your time. What is the task’s purpose? The aim of this research is to evaluate the robustness of the text classifier in adversarial attacks (a detailed description of the task can be found here: https://checkthat. gitlab.io/clef2024/task6/). In this manual evaluation, we will gather assessments regarding the semantic similarities between attack samples and the original samples. Participants in the shared task are requested to dedicate approximately 8 hours to this manual evaluation, which will be conducted using an online tool. Judges will rate the sample pairs based on the following scale: 3 (Preserve the Semantic Meaning), 2 (Change the Semantic Meaning), and 1 (No sense).
Your participation, what it involves and why we are grateful It is up to you to decide whether or not you want to participate in this annotation task. If you do decide to support us in the project, you will be given this information sheet to keep (and be asked to sign a separate consent form). You can still withdraw at any time without any consequences and without giving any reason. The entire data collected from withdrawn participants will be destroyed immediately, and no personal information will be kept. If participants submitted a solution to the shared task that also means their submission to the shared tasks will not be manually scored. If you wish to withdraw, please contact Dr Xingyi Song (details in section 10). You will be asked to annotate adversarial examples into 4 categories using the GATE Teamware Platform (https://annotate.gate.ac.uk/).
in this evaluation will support our research on the development of a more accurate assessment of the robustness of text classifiers and efectiveness of the adversarial attack methods. No major disadvantages or risks are foreseen, however, it is worth mentioning that the content being annotated may case distress. You will be in charge of selecting the content to be annotated, therefore, you are free to only select content that you are comfortable with. If, at any part of the experiment you feel uncomfortable with the content you are accessing, please talk to one of the responsible researchers.
Will my involvement be kept confidential? All the information that we collect from you and about you during the course of the research will be kept strictly confidential and will only be accessible to members of the research team. You will not be able to be identified in any reports or publications unless you have given your explicit consent for this on your participant consent form.
we are required to inform you that the legal basis we are applying in order to process your personal data is that ‘processing is necessary for the performance of a task carried out in the public interest’ (Article 6(1)(e)). Further information can be found in the University of Shefield’s Privacy Notice, which is available online under https://www.shefield.ac.uk/govern/data-protection/privacy/general.
task will be used to assess the efectiveness of adversarial examples generation. Since this data will also benefit other researchers, we plan to release a version of the annotated dataset. You will not be identified and your scoring will be aggregated with multiple other annotators. In the case that you provided us with your e-mail address at the beginning of the scoring task, we will destroy it after the scoring task is finished. The University of Shefield will act as the Data Controller for this study. Who is organising and funding the research? This study is organised jointly by Universitat Pompeu Fabra and the University of Shefield at the CheckThat! Lab at CLEF 2024. Universitat Pompeu Fabra is funded by the European Union’s Horizon Europe research and innovation programme under grant agreement No 101060930. The University of Shefield is funded by the the UK’s innovation agency (Innovate UK) grant 10039055 (approved under the Horizon Europe Programme as vera.ai EU grant agreement 101070093).
Who has ethically reviewed this study? This project has been ethically approved via the University of Shefield’s Ethics Review Procedure, as administered by the Computer Science Department. What if something goes wrong and I wish to complain about the research? If you have any complaints, either from the researcher or something occurring during or following your participation in the project (e.g. a reportable serious adverse event), please contact Dr. Xingyi Song (contact details in section 10). Should you feel your complaint has not been handled to your satisfaction, you can also contact the Head of Department at the University of Shefield, Professor Heidi Christensen (heidi.christensen@shefield.ac.uk) who will then escalate the complaint through the appropriate channels. If the complaint relates to how your personal data has been handled, information about how to raise a complaint can be found in the University’s Privacy Notice: https://www.shefield.ac.uk/ govern/data-protection/privacy/general.
Contact for further information Details of who you should contact if you wish to obtain further information are as follows: Dr. Xingyi Song, Department of Computer Science, University of Shefield, 211 Portobello, Shefield S1 4DP, UK. E-mail: x.song@shefield.ac.uk Telephone: +44 114 222 18577.
User Text COVID-19 the average diameter of the virus particles is around 120 nm (.12 m). Any mask including N95 masks can’t filter this particle size, to filter it completely would prevent the subject being able to breath. Therefore, the mask are just the new normal, a fashion statement. A #COVID19 particle is about 1 to 4 microns, an N95 will block 95% of tiny air particles, down to 0.3 microns, and surgical masks aren’t efective at blocking particles smaller than 100 microns. Hence surgical masks cannot stop the spread of #COVID19.