Consulting web search engines has become an everyday procedure for many internet users. One speci c task that gained attention in recent work is the retrieval of arguments for controversial topics. Most of the preexisting di culties that search engines have to face also apply for this task. However, certain challenges become even more important, such as providing an appropriate heterogeneity in the result set. We present an argument search engine for the argsme corpus. Our focus is on preprocessing the corpus while also addressing the heterogeneity problem and implementing a query expansion feature. Furthermore, we provide a brief evaluation of our retrieval results.
Nowadays, search engines are used by everybody. Consulting a search engine is the easiest way to nd desired information like today's weather, news articles or just any arbitrary image. However, there are still some problems for which modern search engines fail to deliver satisfying answers yet. One of these challenges is the search for arguments in large document collections, e.g. for debates such as \Are plastic bottles good?" or \Are speed limits wrong?". Search engines that nd arguments for these kinds of queries could be classi ed as argument search engines. There are already some solutions available, e.g. Args.me [22] or ArgumenText [20].
This report is created in the context of the Touche shared task on argument
retrieval [
First of all in Section 2, we summarize the state of the art of information retrieval in the context of argument search engines. Section 3 introduces our search engine's architecture. Following, we give a short overview of the corpus, describe some necessary preprocessing steps and show the ideas behind all separate components. In Section 4 we present the results of the nal evaluation.
Argument search refers to collecting relevant premises and conclusions to a given topic that is usually of controversial nature. The goal of such a search engine is to provide the user with supported statements that help him to gather knowledge about his topic of interest and potentially assist his decision making. 2
Previous work has been carried out that tackled various tasks of argument search,
including automatically detecting evidence that support a given claim [
The heart of an argument search engine is a proper retrieval model. The challenge here is to nd the best arguments w.r.t. a corpus and a free-text query.
Several argument search engines are e.g. args.me [22] or ArgumenText [20].
These engines are based on the retrieval model Okapi BM25. Potthast et al. [
Predicting argument quality is a challenging task. Wachsmuth et al. [21] discuss
the concept of argument quality. Wei et al. [24] rank argumentative reddit posts
in order to nd the most persuasive ones. Their approach is machine
learningbased. The same applies for the approach proposed by Persing and Ng [
The aim of clustering here is to guarantee a diverse result set in order to present
the user a variety of di erent arguments. Carbonell and Goldstein [
Another approach to build a diverse ranking incrementally is proposed by
Kaptein et al. [
Our argument search engine's architecture is depicted in Figure 1. The central module is the Apache Lucene Core4 search library that realizes indexing and retrieval. Before indexing, the corpus is preprocessed. During preprocessing, quality ratings for all documents are computed.
At run time, the user's queries are enriched by a query expansion module. Then, after retrieving a set of relevant documents via the Lucene Core, results are ranked considering both the scores obtained by the retrieval system and the quality ratings. Eventually, the last component can perform clustering on the top ranked results. 3.1
The quality of the arguments contained in the corpus is heterogeneous. Some documents do not contain arguments at all. Therefore we aim to assign ratings to the documents indicating their argumentative quality. More formally, we create a mapping q : D ! [0; 1] (1)
where D is the corpus as a set of documents. If q(d1) > q(d2), the argumentative quality of d1 is considered higher than that of d2.
These ratings are then used in our retrieval model, as the user should only receive arguments with high quality. To compute them, we employ a machine learning approach. Argument quality is a rather elusive concept that can not be quanti ed directly. Wachsmuth et al. [21] break it down into three aspects: { Logical quality : are the premises acceptable and do they really imply the conclusion? { Rhetorical quality : is the argument formulated in a persuasive manner? { Dialectical quality : does the argument contribute to resolving the issue?
Capturing the logical dimension of argument quality with computational features is a hard task. Solving it is beyond the scope of this project.The dialectical quality dimension is not available in our corpus either. Other than the corpus of reddit posts used by Wei et al. [24], the argsme corpus does not contain information about replies to a post or citations of a post. Thus, the only quality dimension we aim to quantify is rhetorical quality.
To achieve this, we compute 22 features for each argument. Most of them can
already be found in Wei et al. [24] and Persing and Ng [
Linguistic competence features aim at quantifying the argument's author's linguistic skills: average sentence length, word length, type/token ratio, number of punctuation marks per sentence, di erent POS-Tags (whole text), conjunctives per sentence, modal verbs per sentence, emojis (Use of emojis might coincide with rather colloquial language and lack of seriousness), non-stopwords ratio.
Sources and Examples: claims are more persuasive when they are supported by examples and sources. The following rule-based features are intended to capture them: number of references per sentence, examples per sentence, URLs per sentence, percentages per sentence, year speci cations per sentence
Subjectivity, ad hominem and emotionality: Arguments are more persuasive
when they are presented in an objective manner, without anecdotal evidence or
attacking the opponent personally. We aim to quantify subjectivity and
emotionality with the following features:
{ number of rst person pl. pronouns per sentence indicate subjectivity. We do
not count rst person singular words. Persing and Ng [
Before normalizing all features we lter out odd documents based on rules. To give an example, the average word length in an argument is expected to be between 2 and 16. Odd documents are assigned the rating 0.0. Such documents are typically spam or short meta-posts like e.g. \I accept", \Vote Pro" etc.
As training data we use the Webis-ArgQuality-20 Corpus [
We train several several machine learning models: Linear Regression, Decision Tree Regression and Support Vector Regression (SVR) with di erent kernels. For each type of model we train one instance on rhetorical quality and another instance on combined quality. Both instances' parameters are optimized via grid search.
All models perform rather poorly, con rming that argument quality prediction is a di cult problem. SVR with a quadratic kernel achieves the best results (MSE of 1.641 for rhetorical and 1.475 for combined quality). Moreover, we train an ensemble model (Linear Regression) using the predictions of all models as features. As expected, it outperforms all single models (MSE of 1.468 for rhetorical and 1.322 for combined quality).
An interesting detail is that all models, even those trained on rhetorical quality, perform better in predicting combined argument quality than in predicting rhetorical quality. In other words, overall quality seems to be easier to grasp than rhetorical quality, at least with our approach. This hypothesis is statistically signi cant for p < 0:01. One explanation may be that some of our features also capture aspects of dialectical and logical quality: For example, providing sources to support a claim could indicate logical correctness. Features related to subjectivity and emotionality might at least be able to suggest low dialectical quality, as a very emotional and/or subjective post is often unlikely to contribute to resolving an issue.
Finally, to obtain the desired quality function q : D ! [0; 1], we let the trained models predict the combined quality of every argument in the argsme corpus. We compute the predictions of the best single model (quadratic SVR trained on combined quality) and the ensemble model, leading to two candidates qsvr; qens for q. Figure 2 shows the distributions of the ratings generated by both models.
We choose qsvr for q, even though the SVR model's MSE is higher than that
of the ensemble method. The main reason for this decision is that there are
almost no \bad" arguments according to qens, which is certainly inaccurate.
Like the implementation of args.me we decide to use Apache Lucene for the
indexing and retrieval tasks. We index the extended corpus which is generated
during preprocessing. In the rst step of query processing, stopwords are
removed from the query. Before results are retrieved, the query is extended using
additional query expansion methods. For the ranking, we implement di erent
methods. As a baseline, we use Lucene's BM25 implementation. Furthermore,
we extend the Lucene search core with an implementation of the DPH concept
[
A query is a short representation of the user's information need. However, these few words may not be su cient to encompass the entire concept that the user wants to express. This can lead to highly relevant documents not being found by the retrieval system due to vocabulary missmatch. That is, the user may choose terms for his query that do not appear in a relevant document. To mitigate this gap, automatic query expansion methods can be used. In this section we brie y describe the components of AQE and our implementation.
For our query expansion component we decided to use one simple baseline approach and two more sophisticated concepts. As baseline, we employ WordNet to fetch semantically similar words for each individual query term. This method can not grasp the concept of the entire query as one unit. However, as many of the queries provided for the shared task only consist of few terms, such as \speed limit" or \nuclear weapons", this simple AQE method can potentially provide a useful enhancement.
The other two expansion procedures both rely on word embeddings. We use
fastText6 to obtain vector representations from the argsme corpus. We
combine these locally trained representations with pre-trained embeddings o ered
by fastText. Then, we adapt a query expansion method as proposed by Diaz et
al. [
All newly found terms are then weighted and the best ones (matching the modeled language) are selected to augment the query.
Even though the work presented by Zuccon et al. [26] does not directly focus on AQE, we also use their insights to realize another expansion method. They investigate di erent ways to estimate translation probabilities for terms that belong to the same language model. Similarly, our goal for AQE is to nd words w that are likely to be "translations" of the initial query terms: pt(wjq) = u2qpt(wju)p(ujq) (4) where pt(wju) describes the probability of translating term u into w which can be approximated by a normalized cosine similarity.
Naturally, both expansion techniques operate on each query as a whole to incorporate their relatedness. Eventually, the expansion terms and their respective weights are returned to our search core. Note that for the current implementation only one expansion method is used at a time. 3.4
A more experimental component of our search engine is the clustering/reranking module.
In retrieving arguments, not only the argumentative quality of the returned results is important. Another aspect of an argument search engine's utility is the heterogeneity of the returned arguments. In every use case, the user bene ts from receiving a wide variety of semantically di erent arguments.
A problem of the argsme corpus is that an argumentative document usually contains more than one argument. Nevertheless, documents may often be semantically similar. Moreover, optimizing heterogeneity can con ict with optimizing quality. Both goals need to be balanced.
As the Touche task is evaluated using nDCG, we rst make sure that our results are of high quality (w.r.t to the query and the argumentation quality). Then, the top 8 results are clustered and reranked in order to diversify the top results.
Semantic clustering is implemented using Latent Semantic Analysis
(Deerwester et al. [
In the following, let R be the ranking and R[i] the document with rank
i in R. Similar to Deselaers et al. [
ki=11 k1 dist(R[i]; R[i N ov(R[i]) := k]) (5) We weight the dissimilarity depending on the number of ranks between R[j] and R[j k]: documents should not be similar to their immediate predecessor.
Based on novelties, we de ne a measure for R's diversity/heterogeneity: heterogeneity(R) := jjR=j2 1j N ov(R[j]) (6) The likelihood that a user actually looks at a document d decreases with d's rank. Because of that, our heterogeneity measure weights each document's novelty depending on its rank.
Next, heterogenity of a ranking R needs to be balanced with R's quality. To achieve this, we compute a reranking R0 of R that maximizes quality(R0) + (1 ) heterogeneity(R0) (7)
We use nDCG with our retrieval model's ratings to compute quality(R0). Note that the problem of nding an optimal R0 can be framed as Mixed Integer Program. However, since we restrict ourselves to reranking only the top 8 documents, we nd an optimal solution using brute force. Table 1 shows the e ect of our reranking on a dummy corpus.
The hyperparameter 2 [0; 1] in Equation 7 could be set by the user. Alternatively, could be further investigated in order nd a reasonable value. This is beyond the scope of our project. For the evaluation, we turn o the clustering component (i.e. set to 1), because nDCG does not consider heterogeneity. 4
For the nal evaluation of our system, we decide to use the combination of DPH
and baseline query expansion. Additionally, we augment the scoring function
with our quality ratings as described in Equation 2 using = 0:5. The clustering
component is not used, since it can not be expected to have a positive impact on
nDCG scores, as pointed out in Section 3.4. Among the various retrieval models,
DPH should show the best performance according to the ndings of Potthast
et al. [
The result of the nal run which was evaluated via tira.io [
We implemented an argument search engine for the argsme corpus. The results, however, are not very convincing yet. We suppose they could be improved in future work, considering the following aspects. As the search engine proved to bene t from our argument quality ratings, these ratings could be further investigated. More sophisticated features and models could be tested. What is more, the weighting of the ratings in our retrieval model, i.e. the hyperparameter in Equation 2, could be optimized. One of the major downsides of our approach is that it does not analyse the semantics of potentially relevant documents. Thus, the precision is often rather low. Future work could tackle this issue. A closer investigation of the query expansion component (e.g. investigating more queries) would probably improve our search engine's results, too. We implement a reranking component to diversify the top-ranked results. However, we were not able to evaluate its quality within the scope of this work. What is more, the reranking component is only implemented in a proof-of-concept style, limited to the top 8 documents.
To conclude, we aimed to address the complex problem of argument retrieval using several di erent methods. There is much space for extending and enhancing our approach in order to improve its performance. Linguistics, Lisbon, Portugal (Sep 2015). https://doi.org/10.18653/v1/D15-1050, https://www.aclweb.org/anthology/D15-1050 20. Stab, C., Daxenberger, J., Stahlhut, C., Miller, T., Schiller, B., Tauchmann, C., Eger, S., Gurevych, I.: Argumentext: Searching for arguments in heterogeneous sources. In: Proceedings of the 2018 conference of the North American chapter of the association for computational linguistics: demonstrations. pp. 21{25 (2018) 21. Wachsmuth, H., Naderi, N., Hou, Y., Bilu, Y., Prabhakaran, V., Thijm, T.A., Hirst, G., Stein, B.: Computational argumentation quality assessment in natural language. In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers. pp. 176{187 (2017) 22. Wachsmuth, H., Potthast, M., Al-Khatib, K., Ajjour, Y., Puschmann, J., Qu, J., Dorsch, J., Morari, V., Bevendor , J., Stein, B.: Building an argument search engine for the web pp. 49{59 (Sep 2017). https://doi.org/10.18653/v1/W17-5106, https://www.aclweb.org/anthology/W17-5106 23. Warriner, A.B., Kuperman, V., Brysbaert, M.: Norms of valence, arousal, and dominance for 13,915 english lemmas. Behavior research methods 45(4), 1191{ 1207 (2013) 24. Wei, Z., Liu, Y., Li, Y.: Is this post persuasive? ranking argumentative comments in online forum. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 195{200 (2016) 25. Zhai, C., La erty, J.: A study of smoothing methods for language models applied to ad hoc information retrieval. In: Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery, New York, NY, USA (2001). https://doi.org/10.1145/383952.384019, https://doi.org/10.1145/383952.384019 26. Zuccon, G., Koopman, B., Bruza, P., Azzopardi, L.: Integrating and evaluating neural word embeddings in information retrieval. In: Proceedings of the 20th Australasian document computing symposium. pp. 1{8 (2015)