Benchmark for Argument Mining (AM). We propose a method to homogenize both the evaluation process and the data to provide a common view in order to ultimately produce comparable results. Built as a four stage and end-to-end pipeline, the benchmark allows for the direct inclusion of additional argument miners to be evaluated. First, our system pre-processes a ground truth set used both for training and testing. Then, the benchmark calculates a total of four measures to assess diferent aspects of the mining process. To showcase an initial implementation of our approach, we apply our procedure and evaluate a set of systems on a corpus of scientific publications. With the obtained comparable results we can homogeneously assess the current state of AM in this domain.
to manually navigate the seemingly unending flood of new scientific information.
One of the emerging fields addressing the machineassisted processing of scholarly documents is Argument ment components (and possibly relations) from natural language texts [2]. This information is not only useful for summarization but also for detecting connections between diferent entities such as individual papers or outlets [3]. This kind of network has been described as the Argument Web by Bex et al. [4] — a vision where all argument data is URI-addressable and linked. If we want to work toward the automatic implementation of such a knowledge graph containing arguments from scientific publications, we first need to be able to compare the performance of existing solutions. However, there is currently no widely established, standardized AM benchmarking approach.
Lippi and Torroni [5] point out several problem areas which stand in the way of a homogeneous evaluation: the granularity of the in- and output of AM systems, the variety of genres and domains they focus on, and the representation of arguments in the evaluation data, i.e. the argument model. Additionally, as previously noted by Duthie et al. [6], a wide spectrum of diferent measures are in use, and these are not accurately described or appropriately applied in all cases. To address the issues above,
0000-0002-0257-3318 (F. Ruosch); 0000-0002-2076-9584
an overview of eforts in the domain of scientific publications. In the second part, we address existing measures used to evaluate the performance of AM. Finally, we describe available benchmarks.
Despite the diferent interpretations of what AM entails [8], there is the well-established information extraction approach, as popularized by several experts in the ifeld [ 9, 2, 5]. Stab et al. [10] explain AM as a multistage pipeline that extracts the arguments from text, usually by ifrst separating non-argumentative from argumentative units, then classifying the argument components and, ifnally, identifying their structure with relations. We adopt this definition because it fits best with our ultimate goal of creating the Argument Web of Science [4] for which we need to extract information about argumentative units and their relations. Other AM papers [11], treat the mining process as a search task to retrieve arguments from a pre-computed set according to their relevance for a query or keyword.
For a detailed overview of literature of the last 20 years in the field of AM for scientific publications, we point the inclined reader to the survey of Al Khatib et al. [12]. They do not only present an overview about the eforts made but also indicate current applications and identified challenges. text, i.e. the boundaries of the identified components. For the relation scores, these segments are aligned between annotations. Considering the Levenshtein distance [15] and also the location in the text, the components are mapped. Then, the number of correctly predicted connections (also with respect to their types) is calculated for propositional (attack, support) and dialogical (considering the speaker’s intent) relations.
Even though CASS is very flexible (i.e. scheme agnostic), it still has some drawbacks. Firstly, it assumes the existence of dialogical annotations, which is not common in current automated AM approaches. Also, there is no public implementation such that it could be put into practice. Finally, it wholly omits the component classification part of the AM pipeline by only focusing on the segmentation (i.e. boundaries of the components) of the text. By introducing our own evaluation method, we aim to remedy the points mentioned above.
specifically as benchmarks.
NoDE [16] consists of a total of three data sets covering 2.2. Measures diferent domains: online discussions, a stage play, and the revision history of Wikipedia articles. The source There is a wide range of Information Retrieval (IR) mea- for the first part were diferent online debate platforms sures used to evaluate AM systems. Typically, IR evalua- which allow members to discuss controversial topics such tions presume the existence of a gold standard or ground as violent video games or abortion. Secondly, arguments truth that a proposed solution is evaluated against [13]. were extracted from the play “Twelve Angry Men”, where The F1 (also F-score or F-measure) [14] can be used to a jury discusses the culpability of a young man in a murassess the accuracy of the predictions made by a system der case. The third data resulted from comparing two by calculating the harmonic mean of the precision and diferent Wikipedia dumps based on the edits of the five the recall, both of which have also been applied on their most revised pages. All three sets were annotated by a own for performance evaluation. For multi-class tasks team of two ( = 0.70 – 0.74) and, in total, they contain 792 (such as AM, where we aim to identify various compo- pairs, each connecting two arguments with information nents or relations) diferent versions of F1 exist, based on about entailment. Partly, they are also annotated with how the score is averaged for the classes. The macro-F1 support- or attack-relationships. It is of note that it is variant weights all classes equally for the combination not possible to use this benchmark to evaluate the whole into a single F1, while micro-F1 considers the number of AM pipeline since it does not contain any information occurrences for each label. Not only are we unable to about the boundaries of arguments in continuous text. directly compare results reported for diferent variants of Aharoni et al. [17] present a data set based on Wikipedia the F-score, some literature also chooses not to include pages for a range of controversial topics. In the labeling the specifics of which weighting method was employed. process, they first extracted claims from the articles, fol
Duthie et al. [6] raise the issue that traditional mea- lowed by supporting evidences. Given that each claim sures from the field of IR may over penalize when simply is identified context-dependently, they are inherently asapplying them for each of the pipeline stages successively. signed to a topic. Every evidence is then connected to For example, wrongly or not at all identified components a claim and given a type (study, expert, or anecdotal). directly influence and reduce the calculated performance The labeling was conducted by 20 inhouse annotators of the relation prediction task. To address this and other with a Cohen’s of 0.39 for the claims and 0.40 for the shortcomings, they introduce the Combined Argument evidences. The corpus covers a total of 33 topics with Similarity Score (CASS) [6]. It splits the evaluation of AM 1392 claims and 1291 evidences. Notably, all evidences into three individual scores which are then aggregated are supporting and no attack relation is annotated. It also into a single number. First, the segmentation step evalu- does not contain explicit information about the location ates the similarity of diferent partitionings of the same of the components in the text (and thus the boundaries).
These two works share one major drawback: Instead of providing a framework including one or more evaluation measures and a state-of-the-art benchmarking methodology, they solely present a new data set, that can be used as a ground truth. Thus, no uniform method to assess the performance of AM system is established since the choice of the measure has not been fixed. We address this issue in our work.
marking pipeline. Then, we specify the measures
employed to assess performance for the diferent stages.
Finally, we describe our argument representation
unification efort.
3.1. Overview
our framework, the system can then address the training,
where applicable, and execution step, which are both
integrated into the end-to-end pipeline. We explain each
of these functionalities separately below.
(
with the goal of not only providing an easy to access sys- (
Furthermore, each system needs to specify a mapping role. Not having identified certain components also takes (represented by the graph icon on the bottom of Figure 1) away the opportunity to relate them and, thus, is not only to create a uniform view of the argument representation penalized in the previous step but also has an impact on and to make the results comparable. It is employed for the relation prediction score. Consequently, we give the pre-processing, to create a specific data set, and for the possibility to either use the argumentative units as idenevaluation, to map all systems to the same argumenta- tified by the system (i.e., the intermediate results) or to tion scheme. By specifying the mapping with Semantic recourse to the ground truth as the input for this step. Web technologies (OWL3), we not only ensure that it is When using the computed intermediate results, we match machine-readable and interoperable, but we also facili- the components to the ground truth to ensure fairness so tate its extension and reuse. that the boundaries do not need to coincide exactly. Instead, we assign each identified unit to one in the ground 3.2. Evaluation Measures truth, if they overlap in at least one token. For components covering multiple ones in the ground truth, we Every task is treated as a (multinomial) classification. select the one with the largest intersection. This does However, we use several evaluation methods because not only allow for difering boundaries, it also ensures they difer slightly in the granularity and format of the that localization information of the units is factored in. data as well as their goal. We explain the measures and By constructing triples out of the two components and their reasoning for every step of the pipeline. the relation (subject, predicate, object), we obtain lists
In the first task, the aim is to classify sentences as of predicted and gold data. This turns the problem into (non-)argumentative. If a sentence contains at least one identifying retrieved/missed, relevant/irrelevant triples. argument component, it is defined as argumentative [ 5]. Therefore, we can again employ the F1-score. One caveat After extracting these annotations from the mined results is that we also need to consider the symmetric nature of as well as from the ground truth, we compare two lists some relation types. By converting the data into triples, of the same length with binary values using micro-F1, to ensure that a possible label imbalance does not afect the 4https://scikit-learn.org/stable/modules/generated/ sklearn.metrics.f1_score 5https://github.com/cfournie/segmentation.evaluation 6https://pypi.org/project/nervaluate 2https://github.com/explosion/spacy-alignments 3https://www.w3.org/OWL we risk not awarding a correct prediction if it is reversed (object and subject transposed) for a symmetric relation.
To amend this issue, we always arrange them in such a way that the subject has the smaller identifier number than the object. Since no relation is reflexive, this results in unique triples.
To illustrate the feasibility of our benchmarking framework, we showcase it with an example data set and a limited number of systems. This section first introduces the used corpus, before elaborating on the selection of argument miners. Ensuingly, we explain the alignment of the diferent argumentation schemes and, finally, present a set of initial results.
To produce comparable results, a common view of how an argument is represented in data is necessary. This is 4.1. Setup achieved by aligning diferent argumentation schemes through mappings. Given the widespread adoption [5] of For our showcase, we use the corpus presented by Lauscher the claim/premise model [21] and its simplicity, we chose et al. [7], currently the only available collection of fully it for our benchmark and use the attacks- and supports- argument annotated scientific papers in English. The aurelations to connect components with the import notion thors extend the Dr. Inventor data set [22] by annotating that we do not restrict neither range (source) nor the arguments for 40 publications in the field of computer domain (target) for both. graphics containing 10’780 sentences in total. According
To align representations, we need two types of map- to the guidelines [23], several types of components have pings: one for the components (claim and premise) and been annotated: background claim (i.e., a claim about one for the relations (supports and attacks). There are someone else’s work), own claim (i.e., proprietary contritwo diferent scenarios: either one scheme is more com- bution), and data (i.e., the evidence). Furthermore, they plex than the other (i.e., it has more components and/or identify relations between the argumentative units: conrelations or has other levels of specificity) or they are the tradicts, supports, semantically same, and part of. The same but use a diferent naming convention (e.g., syn- corpus is publicly available and can be downloaded from onyms or similar but not identical terms such as attacks the project’s homepage.7 versus attack). There is also the special situation for the According to our previously defined requirements, we components that a model is as simple as to only segment- select an initial list of systems to be included in the showing text into non- and argumentative parts. In this case, case. TARGER [24] identifies and tags argument units we do not assess the system’s ability to classify argumen- as claims or premises on token-level from free text intative components due to the lack of information and, put. It implements a BiLSTM-CNN-CRF [25] and uses thus, no mapping is necessary. pre-computed word embeddings, such as GloVe [26].
More complex schemes can be reduced to a simpler Mayer et al. [27] present an AM approach for the domodel with the concepts of e q u i v a l e n t - and/or s u b c l a s s - main of healthcare employing bi-directional transformers o f -relations. Every component and relation from the and combining them with neural networks (LSTM, GRU, original representation is assigned to exactly zero or one CRF), which we label as TRABAM (for TRansformercorresponding element of the benchmark model, depend- Based AM) in this paper. Not only do they address the ing on whether their complement exists and according task of identifying argument components (claim, evito their definition in the original model descriptions. El- dence, and major claim) with a sequence tagging soluements without a counterpart are mapped to no type tion but they also identify relations between these units since they can not be considered in the evaluation. It is phrased as a multichoice problem (attack, support, non). important to note that no annotations are discarded since Trautmann et al. [28] also formulate AM as sequence tagthe ground truth data is recomputed for every run and, ging problem and define the task of Argument Unit Recogif the mapping changes, the alterations are incorporated nition and Classification (AURC). They argue for a more automatically. ifne-grained identification of spans than on sentence
In the case of using diferent naming, we only need to level. At the same time, the authors present a solution employ the e q u i v a l e n t -relation. The same concept may using the established sequence labeling model of Reimers be called diferently but still carry the identical semantics. et al. [29] which employs BILSTMs in combination with Claims are labeled as conclusions, while premises have a word embeddings. plethora of names in literature such as data, evidence, or We include two more systems that, despite being prereason [5]. Similarly, the attacks-relation is also known trained externally, have received attention in the state of as contradicts. Based on the definitions, we can create a the art due to their respective approaches. However, we one-to-one-mapping between model elements and, sub- do not strictly add them to the benchmarked results in sequently, a uniform view of the argument model.
compiled_corpus.zip ArguminSci
MARGOT 3d 12h 37m 1d 06h 05m 2d 22h 41m order to ensure fair comparisons (i.e., of systems trained and executed uniformly and homogeneously within the framework). We consider these additions relevant to extend the range of initially available results and to demonstrate the inclusion of systems. While ArguminSci [30] is a suite of tools that enable the analysis of a range of rhetorical aspects, We solely employ the unit for argument component identification. Taking natural language text, it processes the vector representation of sentences with a pre-trained BiLSTM, feeds the results into a singlelayer network, and, finally, applies a softmax-classifier to identify and tag tokens as argumentative components.
The three labels coincide with the ones used in the ArgSci corpus: own claim, background claim, and data. MARGOT [31] makes use of the information contained in the structure of sentences, identifies claims and evidences, and detects their boundaries. Employing a subset tree kernel [32], the similarity of constituency parse trees is assessed and sentences classified accordingly as containing part of an argument.
As a baseline, we also evaluate the results of assigning the most frequent labels. Every token is outside of an argumentative component (O), and the relations are all non-existent (noRel). times). TARGER takes the least amount of time for both training and execution, and its accuracy is similar to that of the other two systems, save for the classification of the sentences. It scores S = 0.653, which is several percentage As previously pointed out, we adopt the most general points behind both AURC (S = 0.792) and TRABAM (S = and widely-adopted model defining claim and premise for 0.832) (see Table 2 for performance indicators). However, the conceptual representation of arguments. We connect TARGER (B = 0.483) manages to beat AURC (B = 0.470) components using the attacks and supports relations. The for the boundary identification. TRABAM still outperelements of the models (i.e., concepts and relations) of the forms both of them (B = 0.506) in every aspect, while individual systems are aligned to this unifying model via also exhibiting the additional functionality to predict the relations that denote e q u i v a l e n c e or s u b s u m p t i o n , imple- relations. TARGER (C = 0.656) is almost even with TRAmented using RDF. Figures 2 and 3 visualize the mapping BAM (C = 0.662) for the component identification score. for the schemes of the components and relations, respec- Still, TRABAM is the sole system performing relation tively. prediction R = 0.318 and does so to score while relying
Our experiments were executed on a Debian virtual on the components as annotated in the ground truth. machine with a single CPU with eight cores at 2.2 GHz The two pre-trained systems achieve worse results. and 209 GB of RAM. This comes partly as a surprise, given that at least ArguminSci was trained on the same data set. It clearly 4.2. Results outperforms MARGOT on the sentence classification (S = 0.600 and S = 0.454, respectively), but has a similar All systems required more than 30 hours to train and sev- score for boundary detection (B = 0.115 and B = 0.097) eral hours to execute on the test data (see Table 1 for run and is even beat for the component identification (C = ArguminSci
MARGOT BASELINE 0.091 and C = 0.133). The biggest obstacles in both the implementation and
A possible explanation for ArguminSci’s poor perfor- the execution of the benchmark were the variety of apmance is the fact that it does not always produce well- plied approaches and diferences in methodologies. Furformed tags for all the chunks. These annotation errors thermore, the format of the in- and output varied, which are factored into the calculation of both the B and C score. necessitated a lot of custom code for every system. UltiNaturally, the non-existent training time very much ac- mately, it was possible to develop an end-to-end benchcelerates the whole pipeline and in contrast to the other mark for a handful of argument miners, which produces systems, pre-trained ones can annotate the whole test directly comparable results to gauge the state-of-the-art set in a matter of minutes instead of hours or even days. in the field. Although these results are based on the as
When comparing the system results to the baseline, sumption that a ground truth data set labeled with high it can be observed that using the most frequent labels inter-rater agreement exists ex ante, the curation of anis only rewarded for the sentence classification score notated data remains a challenge in AM [33]. Here, the (S = 0.457), but yields zeroes across the board for the advent of deep learning techniques and their demand for other individual measures. This is intended, since the data as well as the opportunity to incorporate the crowd benchmark is designed to only consider identified actual in the annotation process [34] should produce relief in argumentative content (components or relations), which the long term. is useful for building a graph representation of content. As future work, we plan to evaluate our proposed approach and to provide a larger list of results obtained by our benchmark to analyse the state of AM in the 5. Conclusions domain of scholarly documents. We hope our work will serve as a step toward quantifying the quality of the Argument Web [4] of Science that the current state of the art could potentially achieve.
In this paper, we have presented BAM, a novel and unified approach to benchmarking Argument Mining. We described its modular architecture, consisting of four pillars (pre-processing, training, execution, and evaluation). To produce a first set of results and illustrate its application, we fully showcased our benchmarking framework which included several state-of-the-art AM systems (TARGER, TRABAM, and AURC) and, partially, (without training) two other systems (MARGOT and ArguminSci).
The main insight is that it was possible to create a unified benchmark to produce comparable results. Different systems could be integrated with some additional code and, subsequently, could execute our pipeline. Our experiments showed that longer execution time does not necessarily imply better performance. Also, more specialized systems do not guarantee higher scores in the tasks they cover compared to other approaches with more capabilities. From our results, we see not only the diferences among the AM tools but also between the evaluated aspects with more complex tasks [12] resulting in lower scores. Furthermore, a gap between the best performing system and human annotators is also still evident in the domain of scholarly documents.
tional Science Foundation (SNSF) under contract number 200020_184994 (CrowdAlytics Research Project). The authors would also like to thank the anonymous reviewers for their constructive feedback. [4] F. Bex, J. Lawrence, M. Snaith, C. Reed, Implement- [17] E. Aharoni, A. Polnarov, T. Lavee, D. Hershcovich, ing the argument web, Communications of the R. Levy, R. Rinott, D. Gutfreund, N. Slonim, A ACM 56 (2013) 66–73. doi:1 0 . 1 1 4 5 / 2 5 0 0 8 9 1 . Benchmark Dataset for Automatic Detection of [5] M. Lippi, P. Torroni, Argumentation mining: State Claims and Evidence in the Context of Controverof the art and emerging trends, ACM Transactions sial Topics, Proceedings of the first workshop on on Internet Technology 16 (2016) 1–25. doi:1 0 . 1 1 4 5 / argumentation mining (2014) 64–68. doi:1 0 . 3 1 1 5 / 2 8 5 0 4 1 7 . v 1 / w 1 4 - 2 1 0 9 . [6] R. Duthie, J. Lawrence, K. Budzynska, C. Reed, The [18] E. Loper, S. Bird, Ntlk: The natural language toolkit, CASS Technique for Evaluating the Performance arXiv preprint cs/0205028 (2002). of Argument Mining, Proceedings of the Third [19] M. Honnibal, I. Montani, S. Van Landeghem, Workshop on Argument Mining (ArgMining2016) A. Boyd, spaCy: Industrial-strength Natural Lan(2016) 40–49. doi:1 0 . 1 8 6 5 3 / v 1 / w 1 6 - 2 8 0 5 . guage Processing in Python, 2020. URL: https : [7] A. Lauscher, G. Glavaš, S. P. Ponzetto, An / / doi.org / 10.5281 / zenodo.1212303. doi:1 0 . 5 2 8 1 / Argument-Annotated Corpus of Scientific Publi- z e n o d o . 1 2 1 2 3 0 3 . cations, Proceedings of the 5th Workshop on Ar- [20] C. Fournier, Evaluating text segmentation using gument Mining (2018) 40–46. doi:1 0 . 1 8 6 5 3 / v 1 / w 1 8 - boundary edit distance, in: Proceedings of the 51st 5 2 0 6 . Annual Meeting of the Association for Computa[8] S. Wells, Argument Mining: Was Ist Das?, Proceed- tional Linguistics (Volume 1: Long Papers), 2013, ings of the 14th International Workshop on Com- pp. 1702–1712. putational Models of Natural Argument (CMNA14), [21] D. Walton, Argumentation Theory: A Very Short Krakow, Poland (2014). Introduction, Springer US, Boston, MA, 2009, pp. [9] P. Saint Dizier, The lexicon of argumentation for 1–22. URL: https://doi.org/10.1007/978-0-387-98197argument mining: methodological considerations, 0_1. doi:1 0 . 1 0 0 7 / 9 7 8 - 0 - 3 8 7 - 9 8 1 9 7 - 0 _ 1 . Anglophonia. French Journal of English Linguistics [22] B. Fisas, F. Ronzano, H. Saggion, A multi-layered (2020). annotated corpus of scientific papers, Proceed[10] C. Stab, C. Kirschner, J. Eckle-Kohler, I. Gurevych, ings of the Tenth International Conference on LanArgumentation mining in persuasive essays and guage Resources and Evaluation (LREC’16) (2016) scientific articles from the discourse structure per- 3081–3088.
spective, CEUR Workshop Proceedings 1341 (2014). [23] A. Lauscher, G. Glavas, S. P. Ponzetto, K. Eckert, [11] C. Stab, J. Daxenberger, C. Stahlhut, T. Miller, Annotating arguments in scientific publications, B. Schiller, C. Tauchmann, S. Eger, I. Gurevych, Ar- 2018. gumenText: Searching for Arguments in Heteroge- [24] A. Chernodub, O. Oliynyk, P. Heidenreich, A. Bonneous Sources, Proceedings of the 2018 conference darenko, M. Hagen, C. Biemann, A. Panchenko, of the North American chapter of the association for TARGER: Neural Argument Mining at Your Fingercomputational linguistics: demonstrations (2018) tips, Proceedings of the 57th Annual Meeting of the 21–25. doi:1 0 . 1 8 6 5 3 / v 1 / n 1 8 - 5 0 0 5 . Association for Computational Linguistics: System [12] K. Al Khatib, T. Ghosal, Y. Hou, A. de Waard, D. Fre- Demonstrations (2019) 195–200. doi:1 0 . 1 8 6 5 3 / v 1 / itag, Argument Mining for Scholarly Document p 1 9 - 3 0 3 1 .
Processing: Taking Stock and Looking Ahead, in: [25] X. Ma, E. Hovy, End-to-end sequence labeling Proceedings of the Second Workshop on Schol- via bi-directional lstm-cnns-crf, arXiv preprint arly Document Processing, Association for Com- arXiv:1603.01354 (2016). putational Linguistics, 2021, pp. 56–65. URL: https: [26] J. Pennington, R. Socher, C. D. Manning, Glove: //2021.argmining.org/. Global vectors for word representation, in: Pro[13] H. Schütze, C. D. Manning, P. Raghavan, Introduc- ceedings of the 2014 conference on empirical methtion to information retrieval, volume 39, Cambridge ods in natural language processing (EMNLP), 2014, University Press Cambridge, 2008. pp. 1532–1543. [14] C. van Rijsbergen, Information retrieval, 2nd edbut- [27] T. Mayer, E. Cabrio, S. Villata, Transformer-based terworths, 1979. argument mining for healthcare applications, in: [15] V. I. Levenshtein, Binary codes capable of correct- Frontiers in Artificial Intelligence and Applicaing deletions, insertions, and reversals, in: Soviet tions, volume 325, 2020, pp. 2108–2115. doi:1 0 . 3 2 3 3 / Physics-Doklady, volume 10, 1966, pp. 707–710. F A I A 2 0 0 3 3 4 . [16] E. Cabrio, S. Villata, NoDE: A Benchmark of Nat- [28] D. Trautmann, J. Daxenberger, C. Stab, H. Schütze, ural Language Arguments, Frontiers in Artificial I. Gurevych, Fine-Grained Argument Unit RecogIntelligence and Applications 266 (2014) 449–450. nition and Classification, AAAI (2020) 9048–9056. doi:1 0 . 3 2 3 3 / 9 7 8 - 1 - 6 1 4 9 9 - 4 3 6 - 7 - 4 4 9 . URL: https://doi.org/10.1609/aaai.v34i05.6438.