The ERINIA project is aimed to address the challenges posed by the increasing importance of automatic assessment of text credibility. Text classifiers are commonly used by platforms hosting user-generated content, including social media, to aid or replace human moderation in filtering out text that is undesirable for some reason - bullying, hate speech, fake news, etc. Unfortunately, deep neural networks are known for their vulnerability to adversarial examples, i.e. data instances with small modifications that preserve the original meaning, yet change the prediction of the target classifier. Here we describe the research actions of the ERINIA project, planned to tackle this challenge by assessing the robustness of currently used classifiers in the misinformation context, creating better methods for discovering adversarial examples and detecting machine-generated content.
Herein we summarise the ongoing project Evaluating the Robustness of Non-Credible Text Identification by Anticipating Adversarial Actions (ERINIA), carried out with the TALN group1 at CEUR Workshop Proceedings
The challenges that misinformation poses, both at the level of individuals and societies, are
widely known [
Using the tools available in the field of artificial intelligence (AI), especially machine learning
(ML), has long been recognised as a possible help in these eforts [
To deal with misinformation, but also other undesirable types of content (e.g. hate speech or
bullying), the major platforms have implemented content filtering systems involving a mixture
of AI- and human-based elements [
For example, consider a hypothetical scenario, where a malicious actor causes confusion and distress by spreading entirely made-up rumours, e.g. about a hazardous fallout, using alarming headings such as Radioactive dust approaching from the Mediterranean!. The platforms used for hosting such content, e.g. social media and search engines, should detect all mentions of this information and act accordingly: recommending debunking articles, discouraging from sharing or banning the spreaders altogether. But if the ML model used for detection is not robust enough, the actor might rephrase the heading, e.g. as Radioactive dust coming from the south!, avoid the detection and continue the misinformation campaign.
The reasons outlined above lead to a need to assess the robustness of the credibility assessment models, i.e. their ability to maintain the expected accuracy level even in adversarial setting, where the content creator attempts to mislead the classifier. This is precisely the goal of the ERINIA project, implemented by developing methods and resources that help the discovery of the adversarial examples, improving our understanding of how easy it is to attack the common text classification methods. This could inform the discussion on whether such algorithms are ift to shape the content of the media channels of great importance for the modern societies. 3Note that we are only discussing textual content here, as the most wide-spread carrier of misinformation. 4https://www.reuters.com/technology/twitter-exec-says-moving-fast-moderation-harmful-content-surges-202212-03/
The investigation of adversarial examples (AEs) in the context of NLP is a relatively new efort
[
The first goal of this action is to improve understanding of the current situation. This is
necessary because content management platforms are already using machine learning-based
tools [
The results of this action is the BODEGA (Benchmark for Adversarial Example Generation
in Credibility Assessment) framework, published recently [
The results of the experiments show that the fine-tuned language model is more robust
than BiLSTM, but the success of an attack depends on the scenario, and in some cases the
AEs based on character replacements (DeepWordBug) perform better, while in others more
complex solutions (BERTT-ATTACK) are appropriate. It also appears that tasks with a longer
text (i.e. news bias assessment) are more vulnerable to attacks than those with shorter input
(i.e. fact checking). BODEGA is based on the OpenAttack framework [
As explained above, discovering AE is paramount to understand the vulnerability of a classifier before it is deployed in a sensitive application. The task can be seen as performing a search in a vast space – we have ( ∗ ) possibilities to perform word replacements in a text of length using a dictionary of size . Most of the currently used methods rely on a human-designed heuristic, iterated until a classifier’s decision is flipped.
However, it might be more eficient to train an ML model to design a procedure for a AE generation automatically. The ML framework used to train agents that can learn a behaviour in a given environment is reinforcement learning (RL). In this approach, an agent can perform
actions that change the state of its environment and receive rewards, indicating the success of the current strategy. During training, the agent tries many diferent strategies and gradually learns to choose those that provide the highest rewards. The application of this framework for discovering AEs is quite natural: state corresponds to the current text of the example, actions to making modifications (e.g. word replacements) and rewards to AE quality measurement.
Essentially, this would mean training an ML model to find weaknesses in another ML model.
Reinforcement learning has already been shown to lead to solutions that humans find surprising,
e.g. in computer games [
The modifications of the original text, turning it into an AE, are not unlimited: if too many are made, they change the meaning of the text, which fails to fulfil its original role. For example, a credible news piece might become unreliable after enough word replacements are performed in its text. Thus, another direction we aim to explore is meaning preservation, i.e. finding ways in which a text might be modified into an AE that preserve its original semantics.
This problem is not entirely novel, as it can be seen as paraphrasing [21], but including additional constraints on the modified text, namely that it changes the classifier’s decision. Other tasks of paraphrasing with constraints include text simplification, i.e. making a document easier to read and understand [22], and style transfer, i.e. rewriting the text in a diferent style [23]. We hope to draw the inspiration from these areas to improve the AE generation methods in order to test the robustness of credibility assessment against such examples.
Finding solutions that perform well in adversarial scenario requires understanding the
perspective of the adversary, i.e. the author of the misinformation content. Since fake news articles are
carefully designed to maximise their emotional appeal [24], writing them manually is likely to
take significant efort. Thus, it seems plausible that the misinformation spreaders might turn
to automatic text generators to increase their speed, especially when faced with the need to
prepare many version of the same article to bypass content filtering. The NLP community was
aware of such possibilities when the first models generating human-like text emerged [
Detecting machine-generated content might be helpful in recognising attempts to circumvent content filtering procedures. While the current text generators produce content that untrained humans cannot detect [25], this task might be performed by automatic classifiers instead. Therefore, automatic text detection has been added to ERINIA goals to address this emerging challenge.
Our contribution [26] has been realised within the framework of the shared task AuTexTification: Automated Text Identification [27], a part of 5th Workshop on Iberian Languages Evaluation Forum (IberLEF 2023). We propose a collection of sequential features that measure the predictability of tokens, i.e. how likely they are according to language models. The underlying assumption is that model-generated text is formulaic and repetitive, while human authors can compose surprising and creative writing. The predictability is supplemented with features describing grammatical correctness, word frequency and linguistic patterns. Finally, a neural network combining LSTM [28] and RoBERTa [29] is trained on the examples of human- and machine-generated content provided through the shared task.
Our solution has achieved the best performance in the binary classification task, both for English and Spanish input [27]. We hope this contribution will aid further development in the ifeld, including in context of misinformation detection.
Given the seriousness of the challenges posed by adversarial character of the misinformation, we don’t expect them to be solved by a single project of limited size, such as ERINIA. Instead, we hope to deliver impact by enabling and encouraging further work in this area by NLP researchers and other stakeholders.
In particular, we expect the following new directions to follow our project and similar eforts: • Improved understanding of text classification robustness. We hope that our analysis of the vulnerability of the models used in content filtering will contribute to a wider discussion on whether these models are fit for the role they play in the current online ecosystem. • Better ways to discover adversarial examples. We know for sure the AEs exist, as even the simple methods available now uncover plenty of them. Thus, we expect the development of new methods for finding AEs, including our contributions based on reinforcement learning and meaning preservation, will allow these weaknesses to be spotted and fixed before the classifiers are deployed in publicly accessible systems. • Established procedures for verifying robustness of newly introduced models.
Thanks to the provision of BODEGA, there is a fast and easy way to test a given text classification algorithm against common attacks. Continuing this efort by adding new misinformation tasks and updated algorithms (both for attack and defense) is essential for establishing clear workflows to verify if a given model is ready for its deployment in an adversarial scenario. • More robust text classification models. The currently common architectures based on deep neural networks appear to be particularly susceptible to adversarial examples. We expect the exploration made within ERINIA and similar eforts will be followed by the design of text classifiers that are more robust, without sacrificing the classification accuracy. • High-certainty text authorship assessment. Crucially, platforms hosting user-generated content need reliable methods to uncover if given text was indeed authored by a user, or an automatic generator. While the performance of our submission was better than of other approaches, we do not believe it is yet suficient for a high-stakes application scenario. Hopefully, more accurate methods will follow, delivering higher certainty in practical usecases.
Finally, we need to emphasise that while many of the issues tackled in this project may appear technical and hard to understand, they are relevant to the society as a whole. ML models have an increasingly important role on shaping the public debate and media landscape and we consider it paramount that their role is transparently demonstrated by the content platforms, understood by their users and controlled by policymakers.
The work is a part of ERINIA project that has received funding from the European Union’s Horizon Europe research and innovation programme under grant agreement No 101060930. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union. Neither the European Union nor the granting authority can be held responsible for them. We acknowledge partial support from grant number MCIN/AEI/10.13039/501100011033 under the Maria de Maeztu Units of Excellence Programme (CEX2021-001195-M) and by Google Cloud through Research Credits6