Research on computational argumentation is currently being intensively investigated. The goal of this community is to find the best pro and con arguments for a user given topic either to form an opinion for oneself, or to persuade others to adopt a certain standpoint. While existing argument mining methods can find appropriate arguments for a topic, a correct classification into pro and con is not yet reliable. The same side stance classification task provides a dataset of argument pairs classified by whether or not both arguments share the same stance and does not need to distinguish between topic-specific pro and con vocabulary but only the argument similarity within a stance needs to be assessed. The results of our contribution to the task are build on a setup based on the BERT architecture. We fine-tuned a pre-trained BERT model for three epochs and used the first 512 tokens of each argument to predict if two arguments share the same stance.
Argumentation is an activity in everyday human
life. We argue in domains such as health, law
and politics either trying to find standpoints which
are acceptable by being supported by reasons or
to persuade others to a certain point of view and
if necessary to carry out certain actions.
Computational Argumentation (CA) aims to find
argument representations and models which are well
suited to do computation with arguments. CA is
a new and fast growing field of research. In the
simplest case, an argument is a claim supported
or opposed by at least one premise
In the task SAME SIDE (STANCE) CLASSIFICATION a simplified variant is to be examined, namely whether two given arguments to a topic have the same stance. For example, p1 and p2 have different stances, but p1 and p3 = “The danger from radioactive contamination should be avoided” would have the same stance to the topic nuclear energy. The particular difficulty lies in the fact that p1 and p3 are syntactically very different. So we have to decide on a semantic level whether the stances are the same.
In this paper we present a method where we
fine-tune a pre-trained BERT
1www.args.me 2www.argumentext.de
In our implementation we make use of BERT
Bar-Haim et al. (2017a) address stance classification of premises towards a claim topic. Here, the classification task is divided further into simpler sub-tasks. Bar-Haim et al. (2017b) extend this work by a more extensive sentiment lexicon and contextual features.
Most presented approaches classify the relation of a premise towards a claim. In contrast to the same stance classification the relation between two premises is considered. Further we do not apply feature engineering, but rely on the neural network to extract good features. 3
The arguments from the provided dataset were extracted from the four web sources idebate. org, debatepedia.org, debatewise. org and debate.org. Each instance consists of the seven fields that are depicted in Table 1.
The two most discussed topics “abortion” and “gay marriage” were chosen and two experiments were set-up for the same side stance classification task. The first experiment addresses the classification within topics and consists of a training set with arguments for a set of topics (abortion and gay marriage) and a test set with arguments related to the same set of topics. Table 2 illustrates an overview of the data within topics.
For the second experiment, which addresses the classification across topics, the training set contains arguments for a topic (abortion) and the test Label id topic argument1 argument1 id argument2 argument2 id is same stance
The id of the instance The title of the debate. It can be a general topic (e.g. abortion) or a topic with a stance (e.g. abortion should be legalized).
A pro or con argument related to the topic.
The ID of argument1.
A pro or con argument related to the topic.
The ID of argument2.
True or False. True in case argument1 and argument2 have the same stance towards the topic and False otherwise. set arguments are related to another set of topics. The class Same Side contains 31,195 instances, the class Different Side 29,853. 4
In this section we describe the experimental setup and evaluate our approach utilizing BERT and compare it to a SVM baseline. 4.1
In order to measure the performance of our approach, the following hypotheses were formulated and are subject of this evaluation:
H1: A Transformer-based sequence classification improves upon the SVM baseline. H2: The large Transformer model outperforms the smaller base model.
H3: Longer input sequences yield better classification than shorter sequence lengths.
H4: Classification of full sentences performs better than including partial input sentences. 4.2
First, we divided the provided data set into
training and test sets (90% and 10%). This results
in 57,512 training pairs and 6,391 test pairs for
within topics classification as well as 54,943
training and 6,105 test pairs for cross topics taken all
from the shared task labeled training data. Then
we used BERT
Figure 1 shows the results within the same topic with varying maximum sequence length. Figure 2 shows the results between topics. Argument pairs whose length exceeds the maximum sequence lengths are uniformly truncated. In within topic evaluation large yields higher accuracy than base in four of five cases. In the cross topic evaluation the result is similar. Therefore hypothesis 2 can be accepted. Nevertheless the smaller base model is quite close to large.
The SVM baseline, supplied by the shared task organizers, achieves 54% accuracy in within topic and 58% cross topic. The base model improves upon this already with the smallest sequence length of 32 tokens. Thus hypothesis 1 can be accepted. This result is possibly due to a Transformer having a larger model capacity and employing better suited representations for natural language text compared to an SVM.
Next, we take a look at argumentative input of varying maximum sequence length. We can see from Figure 1 and Figure 2 that the classification benefits from more contextual information. Hypothesis 3 can therefore be accepted. One question is why a model using 64 tokens already performs quite well. Figure 3 shows the distribution of the 3https://github.com/huggingface/ transformers 0.90 0.85 summed argument lengths. We can observe here that the majority (76%) consists of less than 512 tokens. As we can infer from Figure 4, the distribution of the lengths is usually even considerably below 512 tokens. This explains why models with rather short contextual information perform quite well already.
Since the input sequences are truncated the
Transformer model also trains with incomplete,
partial natural language sentences. In order to see
whether a model can better learn from full
sentences we filter out all partial sequences which
are longer than 512 tokens, reducing the available
training/testing data (no trunc train/test). As can
be seen in Table 3 and Table 4 the Transformer
is able to learn from partial sentences. Therefore
hypothesis 4 needs to be rejected. The highest
results are achieved when testing is also done on
untruncated full sentences. This result is an indicator
of what could be achieved with Transformers of
larger or variable maximum sequence length such
as explored by
512 base large 512 0.60 32 64 128
256
Max sequence length
In this paper we have contributed to the SAME
SIDE (STANCE) CLASSIFICATION task and
proposed a method which uses a fine-tuned BERT
model to determine whether two given arguments
have the same stance. The baseline of the
organizers was outperformed with our method. In
our evaluation the large model performs better
than the base model. Our results also show that
longer input sentences are classified better than
shorter ones, and that classifying whole sentences
does not perform better than classifying partial
sentences. According to the organizers’
leaderboard
This work has been funded by the Deutsche Forschungsgemeinschaft (DFG) within the project ReCAP, Grant Number 375342983 - 2018-2020, as part of the Priority Program “Robust Argumentation Machines (RATIO)” (SPP-1999).