Argumentation, Graph Neural Networks, Misinformation Detection The problem of detecting misinformation in social network data has received increasing attention CMNA'21: Workshop on Computational Models of Natural Argument, September 2-3, 2021, Online CEUR Workshop Proceedings
(P. Nightingale)
recently, not least due to the COVID-19 public health emergency [
For Twitter data two of the most successful recent approaches have been one the one hand
detection based on large-scale language models [
Our approach seeks to combine and enrich these two approaches by combining graph structure, in the form of an abstract argumentation framework derived from stance information with linguistic node level features from a language model and argumentative features based on the acceptability status of arguments under diferent semantics.
This paper presents our work-in-progress towards this goal including the planned
methodology and experimental setup. This research builds on earlier work presented at SAFA 2020 [
As part of this research, we plan to make the following contributions: • Show that adding the structure of an argumentation graph can improve detection of misinformation relative to baseline language models • Determine the optimal way to construct abstract argumentation graphs from stance information • Determine the best architectures for combining linguistic and argumentative features for input into Graph Convolutional Networks (GCN)
The notion of acceptability is key to abstract argumentation.
Definition 2 (Acceptability). An argument A ∈ args is called acceptable with respect to an extension ⊆ if for every ∈ with attacks there is an argument ′ ∈ with ′ attacks .
Extensions are subsets of argumentation frameworks that are collectively acceptable.
Extensions are evaluated based on semantics that define rules for what arguments can be accepted
together [
Convolutional Neural Network (CNN) models have been highly successful in computer vision
tasks. Graph Convolutional Networks seek to apply this kind of model to graph structured data.
In the original model proposed by Kipf et al. [
There is a large extant literature on misinformation detection in its various guises. The two
approaches most directly relevant to our research are those that rely on large-scale language
models such as GPT-3 [
However, the most directly parallel work in the literature is found in Toni and Orascu’s
[
Our approach to the problem follows a four step process: 1. Construct an abstract argumentation graph based on stance information 2. Generate linguistic features by creating embeddings for the input tweets, using a language model 3. Generate argumentative features by resolving the acceptability of the arguments in the argumentation framework 4. Train a Graph Convolutional Network using the argumentation graph and the generated features inserted at the node level In the following section, we will explore each of these steps in more detail.
We will construct the argumentation framework based on existing stance information present
in our source datasets. The problem of stance detection has a substantial literature of its own
[
While stance categorization can vary between datasets it is generally possible to classify the relationship between a source tweet, that is the target for classification, and additional tweets in its propagation graph into a polarity of positive, negative, or neutral. We use this polarity to construct abstract argumentation frameworks containing arguments representing each tweet and attacks based on diferent combinations of the following schemes for mapping stance into attack relationships: • Adding attack relationships from any node that has a negative stance towards the source node to the source node itself • Adding attack relationships from the source node to any node that has negative stance towards it • Adding attack relationships between nodes that have difering stance to the source node.
So if tweet A is positive toward the source node and tweet B is negative, we would add either unilateral or bilateral attack relationships, depending on our chosen scheme • Limiting these attack relationships to only those in a tweet’s sub-propagation graph relative to the source node • Adding neutral nodes to the graph both as isolated components, only linked to themselves and with attack relationships towards the source node or node with a negative stance towards the source node, thereby reclassifying neutral nodes as positive or negative for the purposes of the argumentation graph
These schemes will be tested in diferent experimental setups to determine which perform better in diferent contexts. While some of these schemes may seem prima facie strange from a common sense point-of-view, they are designed to allow diferent ways for neighbourhood information to aggregate in the graph convolutional network, which may help in the classification task.
We generate linguistic features by creating embeddings for all tweets in our dataset using a variety of language models. First and foremost, we use the large-scale transformer based language models such as BERT. But we will also include simpler models such as Word2Vec for comparison. We generate these using diferent embedding sizes for comparison. These embeddings are added as node features in the GCN model along with the argumentative features. We will, however, also train a baseline classifier using only linguistic information for the sake of comparison.
We will try diferent feature settings with our experimental setup to determine, which are more successful in predicting misinformative tweets.
Argumentation specific features are generated by incorporating the acceptability status both for sceptical and credulous acceptance under the Complete, Preferred, Stable, Semi-Stable, and Stage semantics. The features will be pre-calculated using an abstract argumentation solver and added as node features by encoding acceptable as 1 and unacceptable as 0.
These features will be tested in diferent experimental setups to determine whether the addition of acceptability information improves overall performance.
We follow the GCN Architecture outlined in Malmqvist et al. [
The architecture includes the following elements:
1. Pre-computed linguistic and argumentative features along with the normalised adjacency
matrix for the argumentation framework
2. An input layer receiving these inputs
3. 6 repeating blocks of a GCN layer [
We will test the model on two datasets that contain both veracity and stance information and
are commonly used in the research literature: Pheme [16] and RumourEval [
We will construct a baseline model using only linguistic features and one using only the argumentation graph and argumentative features. Then we will generate diferent model variants using various combinations of language models and argumentation graph construction schemes some including and some excluding argumentative features.
We will compare the results of these models based on their performance on the benchmark datasets to establish what combinations yield the best results for detecting misinformation. Finally, we will present our results in the context of other work using the same datasets.
This paper presents the work-in-progress for our experiments in applying abstract argumentation in conjunction with linguistic features for detecting misinformation. We hope that this will extend the applicability of argumentation based methods for misinformation detection as well as give greater understanding of how argumentation can be used to enrich and improve deep learning architectures. This will also help decrease the gap between the formal world of abstract argumentation and natural language expressions of argument by incorporating both types of information in the same deep learning model. So far early work has demonstrated the technical feasibility of this approach and we aim to be able to present concrete results in September 2021. [15] Nitish Srivastava, Geofrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 15(56):1929–1958, 2014. [16] Arkaitz Zubiaga, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Peter Tolmie.
Analysing How People Orient to and Spread Rumours in Social Media by Looking at Conversational Threads. PLOS ONE, 11(3):1–29, 2016.