Classifying the stance of an argument towards its target is an important step in many applications of computational argumentation. A simpler variant of stance classification was proposed as a shared task recently, called sameside stance classification: Given two arguments on the same topic, decided whether they have the same stance. In this paper, we present our approach to the shared task, exploring the potential of modeling same-side stance as a similarity learning task. For this purpose, we train a siamese neural network on pairs of arguments represented in an embedding space. In the two scenarios of the shared task, within topics and cross topics, our approach achieved an accuracy of 0.53 and 0.56 respectively.
In computational argumentation, stance
classification is the task of identifying the position of a claim
or a whole argument (usually either pro or con)
towards some target, such as a controversial topic
or another claim. Identifying the stance of a
natural language argument is a major step in argument
search engines (Wachsmuth et al., 2017), debating
technologies
The same-side stance classification task,1 a simplified variant of stance classification, was proposed as a shared task in the context of the RATIO research program on argumentation,2 and its results were presented at the 6th Workshop on Argument Mining.3 The task is defined as:
Given two arguments on the same topic, classify whether the arguments have the same stance towards the topic or not. 1Same-side task, https://sameside.webis.de 2RATIO, http://www.spp-ratio.de 3ArgMining, https://argmining19.webis.de As suggested by the organizers, solving this task does not require knowledge about the topic of the argument, but focuses more on modeling features of the argument pairs actually capturing stance, thus making the task potentially easier. Still, knowing whether two arguments are “on the same side” helps in many downstream tasks, e.g., for structuring discussions, for measuring the bias in a debate, and for propagating a (known) stance of an argument to other arguments.
To approach same-side stance classification, a dataset was provided in the shared task, where each instance consists of two textual natural language arguments from a debate portals, along with a text covering the topic they address. Two experimental set-ups were introduced: (1) within topics and (2) cross topics. In the former, the training set and the test set contain the same topics. In the latter, the test topis are disjunct from the training topics.
In this paper, we investigate the hypothesis that
arguments with the same stance are more lexically
and/or semantically similar than those with
different stance. In particular, we explore the potential
of similarity-learning approaches in addressing the
given task. To this end, we first represent each
argument in an embedding space derived from the
words they span. Then, we learn to map the
arguments to a new space where similar arguments
(having the same stance) are closer to each other, and
other arguments are further away. Concretely, we
represent each argument by document embedding
computed by the Flair library
For both experimental set-ups, we evaluate our approach by first tuning its parameters on the validation set and then evaluating it on the test set. Within topics, our approach achieved an accuracy of 53% in the shared task, whereas it classified 56% of the cross topics test cases correctly. While these values are rather in the middle of the task leaderboard, our analysis provides insights into the adequacy of siamese neural networks for the task. It seems likely that using the provided topic information and/or experimenting with different embedding techniques would boost their effectiveness. 2
The detection of stance as pro or con (and
possibly none, neutral, or similar) is a crucial step in
many technologies related to computational
argumentation. Much research has been dedicated to
this task
Basically, we seeks to learn a similarity function
that reflects the likelihood of two arguments having
the same stance. For this we first represent these
arguments in a semantic embedding space. A big
body of research has investigated different ways for
learning word embeddings, including
Given argument embeddings, we transfer them
to a new embedding space where arguments with
same stance are similar. To this end, we employ a
siamese neural network, which have been first
introduced by Bromley et al. (1994) to approach the
task of signature verification. Later, its architecture
has been utilized for metric learning in tasks such
as face verification
... ... ... ... argument embeddings
(averages) Two-layer neuSraialmneetsweork Two-layer feendetfworowrkard with shared weights feendetfworowrkard ... ...
transformed embeddings combined embedding
... (daibffseoreluntece) sigmoid
unit
similarity score in [0,1]
...
...
ers. In natural language processing, siamese neural
networks have been used, e.g., for learning
sentence similarity
We hypothesized that arguments having the same stance towards a given topic are usually more similar semantically than arguments with opposite stance. To model this similarity, we represent each argument in an embedding space and then learn a similarity function that reflects the likelihood of having the same stance. Figure 1 gives an overview of our approach, detailed in the following.
Concretely, we map each argument to an
embedding using the contextual string embedding model
proposed by Akbik et al. (2018). The model
utilizes a character-level LSTM
Afterwards, we utilize a siamese neural network to learn a similarity function over the encoded arguments. The input of the neural network is pairs of arguments encoded as vectors in the embedding space and a label y indicating whether the two arguments have same stance or not. The encoded arguments are then passed through two feed-forward neural networks that share their weights. An absolute difference is computed from the two output representations that is finally passed through one layer with a single output and a sigmoid activation. As a loss function L, we use binary cross entropy to minimize the difference between predicted scores (y^) and the true labels (y):
L = y log(y^) + (1 y) log(1 y^) The idea behind is to make arguments with the same stance as similar as possible and those with opposite stance as dissimilar as possible. 4
This section describes experiments with our approach within the shared task as well as their results.
Implementation As mentioned above, we use the contextual string embeddings of Akbik et al. (2018). Specifically, we resort to the pretrained model provided in the Flair library, which is trained over news articles.4 We represent each argument as a vector of 4096 dimensions. The siamese neural network we employed is implemented as a two feed-forward neural networks of two layers with ReLU as an activation function. The two layers share weights, resulting in an output vector of 128 4Flair, github.com/zalandoresearch/flair System
Prec. Rec. Acc. Prec. Rec. Acc.
dimensions. For the shared task, both models were
trained on batches of size 16 using Adam optimizer
Training 63,903 training argument pairs on two
topics (“abortion” and “gay marriage”) were
provided by the task organizers. For the within topics
scenario, we randomly split the provided data into
a training set (44732 instances) and a validation
set (19171 instances). Then, we chose the model
with the best accuracy on the validation set. In the
cross topics scenario, we randomly sampled 1000
pairs of arguments on “gay marriage” for validation.
We trained our model on the provided training set
and chose the configuration that performed best on
the validation set. In particular, this configuration
achieved an accuracy of 0.72 in the within topics
scenario and 0.54 in the cross topics scenario.
Results Table 1 shows the final results of our
approach on the held-out test set in the shared task,
in comparison to all other participating systems.
We achieved an accuracy of 0.53 within topics,
and 0.56 in the cross topics scenario. All top
approaches on the leaderboard fine-tuned some
variant of BERT
Looking at the accuracy drop within topics from validation (0.79) to test (0.53), our approach seems to have overfitted to the specific content of the training arguments. Interestingly, its effectiveness on across topics remains stable, indicating that the siamese neural network does learn something general to the task of same-side stance classification. 5
Same-side stance classification is a simplified version of stance classification where the goal is to classify whether two arguments on the same topic have the same stance or not. In this paper, we have presented the approach that we participated with in the first same-side shared task. Our approach was meant to explore the potential of modeling the task as similarity learning using a siamese neural network. The resulting model achieved 0.53 accuracy in the within topics test set and 0.56 on the cross topics test set, putting it roughly into the middle of the leaderboard. Unlike us, the best systems all utilized BERT embeddings.
A follow-up work could study the integration of siamese neural networks with embeddings such as those from BERT. Besides, so far we refrained from integrating the given topic into our approach for simplicity. Making use of topic information to solve the task may also be worth attempting.
Adam: CoRR,
Henning Wachsmuth, Martin Potthast, Khalid AlKhatib, Yamen Ajjour, Jana Puschmann, Jiani Qu, Jonas Dorsch, Viorel Morari, Janek Bevendorff, and Benno Stein. 2017. Building an argument search engine for the web. In Proceedings of the 4th Workshop on Argument Mining, pages 49–59, Copenhagen, Denmark. Association for Computational Linguistics.