Combining tree kernels and tree representations to classify argumentative stances

Abstract

This work investigates how the combination of di erent tree representations with di erent Tree Kernel functions in uences the results of the classi cations in two speci c case studies. One case study is related to the classi cation of argumentative stances of support, the other one is related to the classi cation of stances of opposition. Results show that some Tree Kernels achieves not only higher results but also a higher level of generalization. Moreover, it seems that also the kind of tree representation in uences the performances of classi ers. In this study, we thus explore this relation between tree representation and di erent Tree Kernels, considering also compositional trees.

Keywords

• Argument Mining Tree kernel Argumentative stance

Introduction

Lippi and Torroni 2015 [ 13 ], the problem of having to engineer features to catch complex structures can be solved by employing classi ers that works directly on structures, such as Tree Kernel classi ers. Following this suggestion, recent studies have shown how Tree Kernels classi ers can discriminate between di erent kinds of argumentative stances of support [ 10 ] and opposition [ 11 ].

For example, Liga 2019 [ 10 ] described a rst attempt to use Tree Kernel classi ers to discriminate argumentative stances of support in the context of a binary classi cation. On the other side, Liga and Palmirani 2019 [ 11 ] showed that a similar approach can be applied also to opposition stances and to a multiclass classi cation problem. The present work takes inspiration from these two studies and reproduces their settings by using di erent tree representations and tree kernel functions, to assess whether or not there is a speci c combination that better suits the task of classifying/discriminating argumentative stances.

In Section 2, Tree Kernels classi ers will be introduced along with a description of di erent tree representations and Tree Kernel functions. In the Section 3, we will brie y describe the related works in the domain of AM, considering the studies which employed Tree Kernels in AM and describing the two abovementioned studies, [ 10 ] and [ 11 ], from a general perspective. In Section 4, we will reproduce the settings of Liga 2019 [ 10 ] applying di erent tree representations and di erent Tree Kernel functions to the same scenario and analyzing the results of each classi er. In Section 5, we will do the same process, employing di erent tree representations and di erent Tree Kernel functions in the setting of the experiment in Liga and Palmirani 2019 [ 11 ]. Lastly, in Section 6 we will open a short discussion and conclude the work. 2

Tree Kernels and tree representations

A Tree Kernel is simply a similarity measure. It works by comparing the similarity between tree-structured pieces of data. Supposing that we want to use Tree Kernel classi ers for textual data, there are two main elements to consider.

The rst element is the kind of tree-structure to employ, namely the kind of tree representation we want to use. The second element is de ning which fragments of the tree structures should be involved into the calculation of the similarity. The following sections brie y describe these two aspects. 2.1

TREE REPRESENTATIONS

The idea is that our data (e.g. textual data such as sentences) must be represented into a speci c tree-structured shape to allow a Tree Kernel function to calculate the similarity between di erent pieces of tree-structured data. For example, a sentence can be converted into some kind of tree representation such as a dependency tree or constituency tree.

In the following part, we will shortly describe some of the most famous tree representations for the conversion of textual data into tree structures. They can be considered as particular kinds of Dependency Trees which combine grammatical functions, lexical elements and Part-of-Speech tags in di erent ways [ 7 ]. GRCT The Grammatical Relation Centered Tree (GRCT) representation is a very rich data representation [ 6 ]. It involves grammatical, syntactical, lexical elements together with Part-of-Speech and lemmatized words. In this representation, after the root there are syntactical nodes (grammatical relations), then Part-of-Speech nodes and nally lexical nodes. In other words, a tree of this kind is balanced around the grammatical nodes, which determines the structure of dependencies.

LCT Also Lexical Centered Tree (LCT) representations involve grammatical, lexical and syntactical element, along with Part-of-Speech tags. However, the structure of the tree is di erent. In fact, it is \centered" over Lexical nodes, which are at the second level, immediately after the root. Part-of-Speech nodes and grammatical functions nodes are equally children of the lexical elements. LOCT The Lexical Only Centered Tree (LOCT) representation contains just the lexical elements. Intuitively, the contribution of LOCT representation can be particularly determinant whenever the tasks to be achieved mostly depend on lexical elements. cGRCT and cLCT The compositional Grammatical Relation Centered Tree (cGRCT) and the compositional Lexical Centered Tree (cLCT) representations are very similar to the the Grammatical Relation Centered Tree (GRCT) and the Lexical Centered Tree (LTC) representation. The di erence here is that the representations allow compositional operators. This aspect will be explained more in depth in the section related to CSPTKs. In fact, cGRCTs and cLCT can be used with Compositionally-Smoothed Partial Tree Kernels (CSPTKs) which are designed speci cally for the purpose of considering compositionality [ 3 ]. 2.2

TREE KERNELS

A kernel function can be considered as a similarity measure that perform an implicit mapping ' : X ! V where X is a input vector space and V is a highdimensional space. A general kernel function can be represented as follows: k(x; x0) = h'(x); '(x0)iV (1)

Importantly, the h:; :iV in the above formula must necessarily be considered an inner product, while x and x0 belong to X and represent the labelled and unlabelled input respectively. If we consider, for example, a binary classi cation task with a training dataset D = f(xi; yi)gin=1 composed of n examples, where y 2 fc1; c2g (with c1 and c2 being the two possible outputs of a binary classi cation), the nal classi er y^ 2 fc1; c2g can be calculated in the following way: y^ = n X wiyik(xi; x0) = i=1 n X wiyi'(x):'(x0) i=1 (2)

Where the weights wi are learned by the trained algorithm.

When using Tree Kernels, the function must be adapted to allow the calculations over tree nodes. In this regards, a general Tree Kernel function can be calculated as follows [ 16 ]:

K(T1; T2) =

X

X n12NT1 n22NT2 (n1; n2) (3)

In the above equation, T1 and T2 are the two trees involved in the calculation of the similarity, while NT1 and NT2 are their respective sets of nodes and (n1; n2) is the number of common fragments in node n1 and node n2.

Importantly, (n1; n2) can be seen as a function considering common fragments between trees. Depending on how this function is con gured (i.e., which fragments are considered involved into the calculation of the similarity), di erent Tree Kernels can be obtained.

Given that our data is tree-structured, the second important element is the de nition of the which fragments must be involved when calculating the similarity between trees. De ning which fragments to involve also means de ning the Tree Kernel function, because the names of the Tree Kernel functions usually derives from the fragment de nition.

In the following part, some famous Tree Kernel functions will be shortly described; each of them de nes, in a di erent way, which fragments should be involved into the calculation of the similarity.

STK In a SubTree Kernel (STK) [ 18 ], a fragment is any subtree, i.e. any node of the tree along with all its descendants.

SSTK A SubSetTree Kernel (SSTK) [ 5 ] considers as fragments the so-called subset-trees, i.e. it considers any node along with its partial descendancy. Since in SSTKs the only constraint of not breaking grammar production rules, and since fragments' leaves can be also non-terminal symbols, they can be considered a more general representation compared the previously mentioned STKs. PTK A Partial Tree Kernel [ 16 ] is a convolution kernel that considers partial trees as fragments. Similarly to SSTKs, a partial tree is a fragment of a tree which considers a node and its partial descendancy. However, partial trees allow also partial grammar production rules. The fact that production rules can be broken (i.e. partial), makes PTs even more general than SSTs. This is the reason why PTKs should provide a higher ability to generalize.

SPTK A PTK can be also \Smoothed" PTK (SPTK) [ 6 ], which adds a further semantic layer into the calculation of node similarity. SPTKs allows to calculate similarities between dependency structures whose surfaces (i.e. the lexical nodes, or words) are partially or totally di erent. They introduce a lexical similarity which allows the generalization of tree structures through the semantic layer by representing words not just as mere symbols, but as semantic entities. CSPTK The scenario can be further expanded considering CompositionallySmoothed Partial Tree Kernels (CSPTK) [ 4 ], which apply a composition function between nodes to better represent contextual relations between words.

We have previously showed how compositional trees look like (i.e. cGRCTs and cLCTs), but we did not explain how compositionality works.

The key point of CSPTKs is that they can compute compositional trees integrating Distributional Compositional Semantics (DCS) operators into the kernel evaluation acting on both lexical leaves and non-terminal nodes. On the one side, SPTK already o er a modeling of lexical information, since they extend the similarity between tree structures allowing a smoothed function of node similarity, which makes them able to compare better trees which are semantically related even if their nodes and leaves di er.

However, SPTKs have an major limitation: they cannot consider compositional interactions between the lexical elements of the trees (i.e. between the words of the trees).

The meaning of the verb \to save" can be better captured only if we consider the verb in composition with the words it is refered to, namely \ les" and \people". In this sense, CSPTK can better capture the role of more complex syntagmatic structures and compositions. Regarding the calculation of , it is similar to SPTK, but the smoothing function is adapted according to [ 3, 4 ]. 3

Related works

So far, only few studies have employed Tree Kernels in the eld of AM. One of the rst studies that mentioned the use of Tree Kernels in this eld is Rooney 2012 [ 17 ], where kernels are used with Part-of-Speech tags and with sequences of words with the aim of detecting whether or not a sentence is related to an argumentative element (i.e. premise, conclusion, or both).

In 2015, Lippi and Torroni wrote an important study in which Tree Kernels are employed in a context-independent scenario, with the aim to detect argumentative claims [ 13 ]. In this case, the authors employed PTKs using constituency trees as tree representations. Similar approaches have been applied also in speci c domains like the legal domain [ 14, 12 ] and the medical domain [ 15 ], where SSTK have been applied over constituency trees.

Outside the eld of AM, Croce et Al. [ 7 ] focused on the combination of di erent Tree Kernels and tree representations. However, no study has proposed yet a comparison of this kind in AM, particularly in the classi cation of argumentative stances of support and opposition.

As already stated, this work is based on two previous studies, Liga 2019 [ 10 ] and Liga and Palmirani 2019 [ 11 ]. Interestingly, the two above-mentioned studies try to discriminate among stances of support and opposition that can be related to speci c Argumentation Schemes [ 20 ]. While the rst paper focused on the stances of support potentially related to the Argumentation Scheme from Expert Opinion (which is a speci c kind of source-based Argumentation Scheme [ 9 ]), the second study has a particular focus on the argumentative stances related to the Argumentation Schemes from Negative Consequences and the Slippery Slope argument [ 19 ]. In Liga 2019, Tree Kernels have been used in combination with TFIDF vectors to automatically discriminate between di erent kinds of argumentative support, while in Liga and Palmirani 2019 a similar methodology was employed to detect argumentative stances of opposition.

Importantly, these two studies explore the ability of di erent kinds of Tree Kernel to perform the tasks of classifying argumentative stances (the rst study employs PTK [ 16 ], while the second study employs SPTK [ 6 ]) However, both studies employ the same kind of tree representation: namely, they only employ Grammatical Relation Centered Tree (GRCT representations). However, as already stated, there are other kinds of tree representation that may be employed in the same settings. For this reason, the present study reproduce the same scenarios of these two works by employing di erent kinds of tree representation to assess whether or not a particular kind of tree representation can be more suitable for the task of classifying argumentative stances of support and opposition.

To the best of our knowledge, no study of argumentative stance classi cation has so far presented a comparative analysis of the use of di erent kinds of tree representations and Tree Kernels. In particular, this study will compare the performance of the 5 tree representations described in Section 2.1 At the same time, we will combine these representations with the 5 Tree Kernels described in Section 2.2.

We will now describe the two settings of the present paper. The rst one, related to the rst study [ 10 ], will be de ned Setting One; the second one, related to the second study [ 11 ], will be de ned Setting Two. The experiments have been performed using the JAVA framework KeLP [ 8 ], and a 70/30 train-test split ratio. 4

Setting One

The rst study combined two famous AM datasets in the same setting ([ 1 ] and [ 2 ], namely). We will refer to these datasets as DataOne [ 2 ] and DataTwo [ 1 ]. The aim of the study was to develop a series of classi ers able to di erentiate between argumentative support coming from the opinion of an expert and argumentative support coming from studies or statistics.

Importantly, in this study, the ability of Tree Kernels to generalize over different data was explored by training the classi ers on one dataset and testing them on both datasets [ 10 ]. For this reason, we will consider Setting One as divided into two scenarios.

In the rst scenario, all classi ers are trained on the training subset of DataOne (and tested not only on the testing subset of the same dataset, but also on the whole DataTwo dataset). In the second scenario, all classi ers are trained on the training subset of DataTwo (and tested not only on the testing subset of the same dataset, but also on the whole DataOne dataset).

As shown in the left side of Table 1, which reports the number of instances per class per dataset, the two datasets are quite balanced. The label expert has 372 and 311 instances in DataOne and DataTwo respectively; while the label Study/statistics has 281 and 258 instances in DataOne and DataTwo respectively. The top part of Table 2 is referred to the rst scenario, while bottom part is referred to the second scenario. For reason of space, compositional trees are reported jointly with their non-compositional counterpart: compositional tree representations (cGRCT and cLCT) should be thus considered related only with CSPTKs.

The experiments were performed using the con gurations of the decay factors reported in the right side of Table 13.

Setting One - Scenario A Results of the top part of Table 2 related to GRCT/cGRCT (left sub-table, on the top), show a similar performance on the dataset DataOne (ranging from .85 to .87), but above all they show a growing degree of generalization over DataTwo: from a minimum of .72 (STK) to a maximum of .77 (CSPTK and SPTK). This shows that the degree of generalization increases when using PTKs and SPTKs compared to STKs and SSTKs.

A similar trend can be seen also with regard to LCT/cLCT (central subtable), where the degree of generalization increase similarly from .72 (STK) to .77 (SPTK). However, performances are sliglty more polarized on DataOne (ranging from .84 to .88).

An even more polarized trend is reported on the sub-table on the right, related to LOCT. In this case, performance on the dataset DataOne range from .80 to .87 and this is the only case in which PTK outperform SPTK. Also the performances on DataTwo are more polarized compared to the other two subtables: for the LOCT sub-table scores range from .63 (STK) to .75 (SPTK). 3 Decay factors are meant to penalise long tree fragments, in order to mitigate the risk that their size might excessively a ect similarity scores. In this paper, is the vertical decay factor, while is the horizontal decay factor. Setting One - Scenario B Results of bottom part of Table 2 related to GRCT/cGRCT (left sub-table), show that PTKs performed better than the other kernels: they reach a mean F1 score of .74 on DataTwo (which is the dataset on which the classi ers of this scenario have been trained) while all the other kernels range from .71 to .72. Also in this case, results over the other dataset (which in this scenario is DataOne) show a growing capability of generalization ranging from .79 (STK) to .84 (CSPTK and SPTK).

The trend over LCT/cLCT representations (central sub-table) show a similar picture: PTK is the kernel with the best performance over DataTwo (with a Mean F1 score of .72) while the other kernels range between .68 and .71. Also in this case, performances over DataOne show a growing trend ranging from .80 to .84.

Also in this scenario, the sub-table on the right, related to LOCT, is the most polarized one: results on DataTwo range from .64 to .69, with PTK outperforming SPTK; while results on DataOne range from .71 to .83, with SPTK showing again the best ability to generalize over the other dataset. In the second study [ 11 ], a similar approach has been employed on a dataset created ad hoc for the analysis of argumentative stances of opposition. The dataset has been created by extracting the comments of opposition that citizens wrote on the public website of Nevada Legislature against a bill aiming at regulating euthanasia. The limitation of this study is that the annotation has not been accomplished yet and the number of instances per class is still unbalanced. However, an interesting aspect of this dataset is that a granular labelling system has been proposed in order to assess the ability of Tree Kernel classi ers to detect di erent kinds of argumentative opposition in a multi-class setting. The original setting presents four levels of granularity in which the dataset can be divided. In the present paper, we are going to select just the granularities with the best balance in the number of instances, namely granularity 2 and granularity 3.

For this reason, similarly to what has been done with Setting One, also Setting Two has been divided into two scenarios. In the rst scenario, all classi ers are trained and tested considering three labels (\Slippery Slope", \Other" and \Testimony"). In the second one, all classi ers are trained and tested considering four labels (\Slippery Slope", \Other", \Judgements" and \Testimony"). The number of sentences grouped by class is listed on the left in Table 3, while the con gurations of the decay factors used in the experiment are reported on the right in Table 3. The top part of Table 4 is referred to the rst scenario, while bottom part is referred to the second scenario. It is important to underline that this results should be observed by watching not only at the F1 scores (which can be misleading, since they are trained upwards by the results of the class \Other").

To partially overcome this problem, we should instead focus on the F1 scores of the single classes, as described in the following subsections.

Also in this case, compositional trees are reported jointly with their noncompositional counterpart and should be considered related only with CSPTKs. Setting Two - Scenario A Results of the top part of Table 4 related to GRCT/cGRCT (left sub-table), show that the the Mean F1 score is achieved by the PTK classi er (.76). However, it should be remarked that STK and CSPTK seem to perform better on the class \Testimony" (reaching a F1 score of .71).

In the central sub-table, related to LCT/cLCT, the SPTK and the CSPTK are the best ones both in terms of Mean F1 score (.73-.74) and in terms of balance between the scores for the \Slippery Slope" and \Testimony" classes (which is .59-.60 and .70-.71, respectively). Conversely, the STK, SSTK and PTK show nearly one decimal point less ( oating between the values .50-.51).

Regarding the sub-table related to the LOCT representation, there is a clear superiority of the SPTK over the other kernels not only in terms of Mean F1 score (.75), but also in terms of balance between \Slippery Slope" and \Testimony" scores (.67 and .69, respectively). In fact, although PTK reaches .68 on the class \Slippery Slope", it stops at .58 on the class \Testimony".

Setting Two - Scenario B Regarding the second scenario of the Setting Two (the one considering four labels), results can be seen in the bottom part of Table 4. The sub-table on the left, related to GRCT/cGRCT, shows that performances of PTK and CSPTK classi ers are slightly better in terms of mean F1 score (.69). However, when one considers the results of the classes separately, one can see that the SSTK classi er shows better results in the classi cation of the \Slippery Slope" class (.67), while the CSPTK over the cGRCT representation shows better results in the classi cation of \Judgements" (.66) and \Testimony" (.74).

Regarding the central sub-table (LCT/cLCT), it seems that the CSPTK classi er is the one that has the best performance in terms of mean F1 (.68). It is also the classi er that reach the best performance in the classi cation of the class \Testimony" (.77). Regarding the class \Judgements", the best performances are achieved by the SSTK (.64) and the CSPTK (.63), while the class \Slippery Slope" is the one with the worst performances, with SPTK and CSPTK as best classi ers (stopping at .52).

Regarding the sub-table on the right (related to the LOCT representation), all mean F1 show a similar performance ranging from .60 to .63. Moreover, it can be seen that the performances for the class \Judgements" are the worst ones, where the best score is achieved by the STK (.54). On the other side, SSTK and SPTK achieve the best scores with the class \Slippery Slope" (.65), while the \Testimony" class has the SPTK as most performing classi er (reaching .66). 6

Discussion and Conclusions

From the results, some clear patterns can be observed. In general, it seems that PTKs and above all (C)SPTKs collect the best performances. This appears particularly evident in Setting One, when watching the numbers of Table 2 from the top to the bottom.

Another trend that can be seen from Setting One is the growing degree of generalization (always watching from the top to the bottom of the tables).

Finally, another interesting trend can be observed on Setting One, namely the growing degree of polarization, watching from the left (GRCT/cGRCT) to the right (LOCT), related to the scores between the most and less performing kernels in each column. It seems that the last table (LOCT) is the most polarized one, reaching up to .13 points of di erence between the most and less performing score (i.e. in the column DataTwo, Scenario A; and in the column DataOne, in Scenario B).

To the best of our knowledge, this study is the rst attempt to o er a comparative analysis of the combination of ve tree representations and ve tree kernels in the classi cation of argumentative stances of opposition and support.

A limitation of this work is that of being related to the speci c data employed. The contribution of di erent tree representation should be assessed also in di erent scenarios and with di erent kinds of argumentative data.

Moreover, it is important to investigate the relation between the type of tree representation, the tree kernel function employed and the targeted argument to be classi ed. In other words, are there tree representation that can express better speci c kinds of argument? Are there tree kernel functions that better calculate the similarities between these argumentative representations? This study suggests that the answer to these questions is positive, showing a rst attempt of investigation in this direction.

References

1. Aharoni , E. , Polnarov , A. , Lavee , T. , Hershcovich , D. , Levy , R. , Rinott , R. , Gutfreund , D. , Slonim , N.: A benchmark dataset for automatic detection of claims and evidence in the context of controversial topics . In: Proceedings of the First Workshop on Argumentation Mining . pp. 64 { 68 ( 2014 )

2. Al Khatib , K. , Wachsmuth , H. , Kiesel , J. , Hagen , M. , Stein , B. : A news editorial corpus for mining argumentation strategies . In: Proceedings of COLING 2016 , the 26th International Conference on Computational Linguistics: Technical Papers . pp. 3433 { 3443 ( 2016 )

3. Annesi , P. , Croce , D. , Basili , R.: Towards compositional tree kernels . In: Proceedings of the Joint Symposium on Semantic Processing. Textual Inference and Structures in Corpora . pp. 15 { 23 ( 2013 )

4. Annesi , P. , Croce , D. , Basili , R.: Semantic compositionality in tree kernels . In: Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management . pp. 1029 { 1038 . ACM ( 2014 )

5. Collins, M. , Du y , N.: Convolution kernels for natural language . In: Advances in neural information processing systems . pp. 625 { 632 ( 2002 )

6. Croce , D. , Moschitti , A. , Basili , R.: Semantic convolution kernels over dependency trees: smoothed partial tree kernel . In: Proceedings of the 20th ACM international conference on Information and knowledge management . pp. 2013 { 2016 . ACM ( 2011 )

7. Croce , D. , Moschitti , A. , Basili , R.: Structured lexical similarity via convolution kernels on dependency trees . In: Proceedings of the Conference on Empirical Methods in Natural Language Processing . pp. 1034 { 1046 . Association for Computational Linguistics ( 2011 )

8. Filice , S. , Castellucci , G. , Croce , D. , Basili , R.: Kelp: a kernel-based learning platform for natural language processing . Proceedings of ACL-IJCNLP 2015 System Demonstrations pp. 19 { 24 ( 2015 )

9. Lawrence , J. , Visser , J. , Reed , C. : An online annotation assistant for argument schemes . In: Proceedings of the 13th Linguistic Annotation Workshop . pp. 100 { 107 . Association for Computational Linguistics ( 2019 )

10. Liga , D. : Argumentative evidences classi cation and argument scheme detection using tree kernels . In: Proceedings of the 6th Workshop on Argument Mining . pp. 92 { 97 ( 2019 )

11. Liga , D. , Palmirani , M. : Detecting \slippery slope" and other argumentative stances of opposition using tree kernels in monologic discourse . In: Rules and Reasoning . Third International Joint Conference, RuleML+RR 2019 ( 2019 )

12. Lippi , M. , Palka , P. , Contissa , G. , Lagioia , F. , Micklitz , H.W. , Sartor , G. , Torroni , P. : Claudette: an automated detector of potentially unfair clauses in online terms of service . Arti cial Intelligence and Law pp. 1 { 23 ( 2018 )

13. Lippi , M. , Torroni , P. : Context-independent claim detection for argument mining . In: Twenty-Fourth International Joint Conference on Arti cial Intelligence ( 2015 )

14. Lippi , M. , Torroni , P. : Margot: A web server for argumentation mining . Expert Systems with Applications 65 , 292{ 303 ( 2016 )

15. Mayer , T. , Cabrio , E. , Lippi , M. , Torroni , P. , Villata , S. : Argument mining on clinical trials . Computational Models of Argument: Proceedings of COMMA 2018 305 , 137 ( 2018 )

16. Moschitti , A.: E cient convolution kernels for dependency and constituent syntactic trees . In: European Conference on Machine Learning . pp. 318 { 329 . Springer ( 2006 )

17. Rooney , N. , Wang , H. , Browne , F. : Applying kernel methods to argumentation mining . In: Twenty-Fifth International FLAIRS Conference ( 2012 )

18. Vishwanathan , S. , Smola , A.J. , et al.: Fast kernels for string and tree matching . Kernel methods in computational biology 15 , 113{ 130 ( 2004 )

19. Walton , D. : The basic slippery slope argument . Informal Logic 35 ( 3 ), 273 { 311 ( 2015 )

20. Walton , D. , Reed , C. , Macagno , F. : Argumentation schemes . Cambridge University Press ( 2008 )