This work describes a simple Transfer Learning methodology aiming at discriminating evidences related to Argumentation Schemes using three di erent pre-trained neural architectures. Although Transfer Learning techniques are increasingly gaining momentum, the number of Transfer Learning works in the eld of Argumentation Mining is relatively little and, to the best of our knowledge, no attempt has been performed towards the speci c direction of discriminating evidences related to Argumentation Schemes. The research question of this paper is whether Transfer Learning can discriminate Argumentation Schemes' components, a crucial yet rarely explored task in Argumentation Mining. Results show that, even with small amount of data, classi ers trained on sentence embeddings extracted from pre-trained transformers can achieve encouraging scores, outperforming previous results on evidence classi cation.
In the last few years, the use of Transfer Learning methodologies generated in
remarkable hype in the State of the Art of many Natural Language Processing
tasks. Particularly, the Transformer known as \Bidirectional Encoder
Representations from Transformer" (BERT) has shown extremely good results,
establishing several new records in terms of metrics results [
Despite the high celebrity recently achieved by Transfer Learning techniques,
these methodologies have been applied relatively few times in Argumentation
Mining [
Section 2 will describe the Transfer Learning methodology and the two main settings for the experiments. Section 3 will describe the datasets used for the experiments in the two scenarios. Sections 4 and 5 will show the experimental results on the two scenarios. Section 6 will describe the related works. In Section 7, some nal considerations will conclude the paper. 2
Transfer Learning methods are generally divided in two approaches: the rst approach is called ne-tuning and it consists of using a pre-trained neural architecture (i.e., a Transformer architecture trained on thousands of inputs) as a starting point to perform further training steps on a downstream task (training, thus, the neural architecture on downstream data). The second approach, instead, is that of using a pre-trained neural architecture just to extract the outputs that the Neural Architecture generate for a given input at a speci c stage of the neural architecture. For example, a sentence can be used as input and the output generated by the neural architecture can be extracted and used as sentence embeddings, that can represent our sentence in other downstream tasks (noticeably, the extraction of the generated output to be used as embedding can be performed at di erent stages of the neural architecture, not necessarily at the nal layer). In this paper, the second approach will be employed: a famous pre-trained architecture will be selected, some sentences will be used as inputs for this neural architecture, and the output coming from the neural architecture will be employed as sentence embeddings to represent our data in a series of downstream classi cation tasks.
For the pre-trained embeddings we will employ three pre-trained models:
the rst one is the famous neural transformer called BERT [
To extract the embeddings from the neural models, each input sentence must be rstly tokenized according to the requirements of the given model. Typically, with BERT, a [CLS] and a [SEP] special tokens are inserted at the beginning and at the end of the input (we are interested in the rst one which is the token holding the classi cation output we are interested to extract from the input sentence). Moreover, the length of each input sentence is set to a max length: all sentences longer than that limit are shortened, while all sentences shorter than that limit are padded with the special [PAD] token. This process makes sure that all inputs have the same length before entering the neural architecture. After the tokenization, inputs are passed into the neural architecture of a BERT transformer, while deactivating the calculation of gradients.
After having transformed each input sentence of the test sets into tokens and having used these tokens as inputs for the BERT neural architecture, the resulting extracted embeddings have been used, in turn, as input of a classi cation using two classi cation procedure: a Support Vector Machine (SVM) classi er and a Logistic Regression classi er (LRC). Notice that for the experiment on D3 our SVM employed a Linear Support Vector Classi er (Linear SVC), while in all other experiments we employed a standard Support Vector Classi er (SVC).
The classi cation method is One vs All. Which means that the classi cation has been performed per each class, considering one class against all the other classes, a typical approach in multiclassi cation and multilabel scenarios. Finally, all classi ers have been evaluated on the relative testing set.
The experiments have been divided into di erent scenarios: 1. Baseline scenario: in this scenario, the classi cation was performed on the same setting of two previous works, taken as baselines for comparison. 2. Extended scenario: in this scenario, the classi cation was performed on new settings, using an extended version of two datasets from the baseline scenario. 3
The experiments of this work have been applied to the datasets listed in Table 1, reporting reports also the number of instances for each dataset. These datasets have been selected because their annotations describe classes of argumentative evidence directly related to speci c Argumentation Schemes. Importantly, during the experiments, all datasets have been split into train and test sets, following a standard 80/20 ratio.
Regarding the baseline scenario, D1 and D2 are a portion of Al Khatib et al.
2016 and Aharoni et al. 2014 respectively, two important dataset designed by
IBM. Only two classes from the original datasets have been selected, reproducing
the scenario in [
Regarding the extended scenario, the dataset D1+ is an extension of D1:
instead of extracting just two classes, it considers three classes. The inputs of
the dataset from Al Khatib et al. 2016 [
Importantly, the datasets which have been employed in this work are among
the few available datasets containing instances of argumentative evidences which
can be related to Argumentation Schemes. Namely, the dataset in Al Khatib
et al. 2016 [
The datasets in Aharoni et al. 2014 [
The dataset in Liga and Palmirani 2019 [
These three datasets can thus be used to assess whether classi ers are able to discriminate between di erent cluster of argumentative evidences. Since these argumentative evidences are strictly related to speci c clusters of Argumentation Schemes, the ability of classi ers to discriminate di erent clusters of argumentative evidences is, in our opinion, a crucial step towards Argumentation Scheme discrimination. 4
The classi cations in this Section show that the proposed approach is able to
outperform recent results in the Argumentation Mining literature. With this
purpose, recent results on D1, D2 and D3 are reported [
SBVerMtBLaRse SDVisMtilBELRRT SRVoBMERLTRa BS
In this paper, all F1 scores per class are calculated as the mean macro F1 scores, taken from each One-vs-All classi cation. All these scores are nally averaged and reported as mean F1 (per each classi er, i.e. SVM and LR).
As can be seen from Table 2, results outperform previous results for the same scenario, showing the ability of Transfer Learning techniques to achieve high performances. As indicated by the bold numbers in Table 4, for D1, D2 and D3 there are always at least four classi ers out of six which outperform the baseline. 5
The next series of experiments have been performed on an extended version of two datasets from the baseline scenario (D1 and D2), to assess how performances change in a multiclass scenario.
Regarding the classi cations on D1+, one can see that the best performances are achieved by the Logistic Regression classi er (LR) trained on sentence embeddings extracted using DistilBERT. To have a better understanding of these results, the confusion matrix of the best classi er in this scenario (i.e., Logistic Regression on DistilBERT) are reported with the confusion matrix from the best classi er of the baseline scenario (i.e., Support Vector Machine from DistilBERT embeddings from Table 2) in Figure 1.
D1 (study vs testimony)
SVM on DistilBERT
D1+ (study vs others) D1+ (testimony vs others)
LR on DistilBERT LR on DistilBERT
Regarding the classi cations on D2+ and D2++, one can see that the best performances are achieved by the Logistic Regression classi er (LR) trained on sentence embeddings extracted using DistilBERT and Bert Base. Also in this case, to have a better understanding of the results, the confusion matrices of the best classi ers in this scenario (i.e., Logistic Regression from DistilBERT embeddings and from Bert Base) are reported with the confusion matrix from the best classi er of the baseline scenario (i.e., Logistic Regression from RoBERTa embeddings from Table 2) in Figure 2.
Notice that while confusion matrices for D1 and D2 (in green) show a binary classi cation, the other confusion matrices in blue (relative to D1+, D2+ and D2++) show a one-vs-all classi cation. These blue matrices show that classi ers are able to recognize classes also in a multiclass scenario. While Figure 1 shows an unbalance (which is probably due to the predominance of the class anecdotal), results in Figure 2 seems more balanced: the diagonal is always a 30/60 ratio, indicating the goodness of predictions. 6
Unfortunately, datasets speci cally designed in a way that allow a direct link
between classes and speci c Argumentation Schemes are very few. A promising
and growing resource, in this sense, is the corpora in AIFdb [
LR on RoBERTa
D2+ (study vs others)
LR on DistilBERT
D2+ (expert vs others)
LR on DistilBERT D2++ (study vs others) D2++ (expert vs others)
LR on DistilBERT LR on DistilBERT D2++ (study vs others) D2++ (expert vs others)
LR on Bert Base LR on Bert Base
Moreover, although there have been di erent works of text classi cation in Argumentation Mining, only few studies focused on classi cation tasks aiming at facilitating the discrimination of Argumentation Schemes.
Rinott et al. 2015 [
The datasets analyzed in this work are composed of argumentative evidences which are directly related to di erent clusters of arguments. For example, many instances which can be found in the datasets of this paper are directly related to the cluster of source-based arguments. Other instances of argumentative evidences are instead speci cally related to the Argumentation Scheme from Expert Opinion, while others are related to the cluster which includes the Argument from Negative Consequences and the Slippery Slope Arguments (which do not belong to the cluster of source-based arguments).
We believe that the ability to discriminate di erent clusters of argumentative
evidences is a crucial step in the classi cation of Argumentation Schemes. For
example, the discrimination of clusters of Argumentation Schemes can be
performed in a pipeline of binary classi cations starting from source-based versus
non-source-based arguments and continuing towards more speci c binary
classi cations (similarly to the path of dichotomous choices followed by ASK, the
annotation system recently elaborated in [
In general, the results presented in this paper seem encouraging, showing that pre-trained embeddings can outperform previous results in the eld of Argumentation Mining related to the classi cation of argumentative evidences. An interesting aspect is that the proposed classi ers show encouraging results not only in the discrimination among di erent source-based argumentative evidences, but also in classi cations involving source-based versus non-source-based argumentative evidences (i.e. with dataset D3).
However, further analysis is needed to verify if and how Transfer Learning
techniques can discriminate argumentative evidences in such a way that they
can facilitate Argumentation Scheme discrimination. In this regard, the present
paper is just a preliminary exploration of a promising possible approach. In
future works, other Transfer Learning techniques should be assessed too. For
example, it could be useful to assess the performances between the two main
Transfer Learning techniques: sentence embeddings and ne-tuning. Also, other
pre-trained models should be employed and compared (e.g., Xlnet[
A long-term goal is being able to connect natural language argumentative evidences to their speci c Argumentation Schemes, which can be a further step in the development of an arti cial Natural Argumentation Understanding.