ECNU at MC2 2018 Task 2: Mining Opinion Argumentation

Abstract

This paper describes our participation in MC2 2018 task2: mining opinion argumentation. We build a tweet retrieval system, which is mainly composed by four parts: data preprocessing, retrieval, redundancy detection and reranking. Only the highly relevant and argumentive tweets are sent to the user based on the topics. In addition, three state-of-the-art information retrieval models as BB2 model, PL2 model and DFR model are utilized. The retrieval results are combined for nal delivery.

Keywords

• opinion argumentation
• retrieval
• argumentative ranking

Introduction

An argumentation is, broadly speaking, a claim supported by evidence [ 6 ]. In corpus-based text analysis, argumentation mining is a new problem that addresses the challenging task of automatically identifying the justi cations provided by opinion holders for their judgment. Several approaches of argumentation mining have been proposed so far in areas such as legal documents, on-line debates, product reviews, newspaper articles and court cases, as well as in dialogical domains [ 8, 10, 6 ].

There are situations where the information we need to retrieve from a set of documents is expressed in the form of arguments. Recent advances in argumentation mining pave the way for a new type of ranking that addresses such situations and can positively reduce the set of documents one needs to access in order to obtain a satisfactory overview of a given topic. We build a proof-of-concept argumentative ranking prototype. We found that the results it provides signi cantly di er from and possibly improve those returned by an argumentation-agnostic search engine. Argumentative ranking does indeed provide results that are quite di erent from those that are obtained by a \traditional" search engine. In this task, relevant information is expressed in the form of arguments [ 6 ].

Success of such argumentation ranking will require interdisciplinary approaches based on the combination of di erent research issues. In fact, to better understand a short text and be able to detect the argumentative structures within a microblog, we could restore a \text contextualization" as a way to provide more information on the corresponding text [ 3 ]. Providing such information in order to detect argumentative tweets would highlight relevant ones. In other words, tweets expressed in the form of arguments. Thus, argumentation mining in this situation will tend to act in the same way of an Information Retrieval (IR) system where potential argumentative tweets had to come rst. A similar approach that addresses such a purpose is presented in [ 2 ], where the output of the priority task will be a ranking of tweets according to their probability of being a potential threat to the reputation of some entity.

In this task, given a set of festivals name from most popular festivals on FlickR English and French language, participants have to search for the most argumentative tweets in a collection covering 18 months of news about festivals in di erent languages [ 4 ]. The identi ed tweets have to be a summary of ranked tweets according to their probability of being argumentative tweets. Such sets of tweets could be treated easier by priority, by a festival organiser. For each language ( English and French ), a monolingual scenario is expected : Given a festival name from a topic le, participants have to to search for the set of most argumentative tweets in the same query language within the microblog collection.

The reminder of the paper is organized as follows. Section 2 describes our approach. In Section 3, experimental results are presented. Finally, the paper is concluded in Section 4.

Related Tweets

Redundancy

Detection Topics Results Retrieval

Data Preprocessing

Corpus

Reranking Argumentative

Vocabulary In this section, we demonstrate the architecture of our system, which is shown in Figure 1. It shows that our system mainly consists of four parts, namely data preprocessing, retrieval, redundancy detection and reranking. The details of each part are demonstrated in the following sections.

Data Preprocessing Before we start to run the system, we preprocess the dataset. We rst solve the tweets as follow steps: { Converting the letter in tweet to lowercase letters. { Turning several spaces into one space. { Replacing http:// or https:// in tweet with `<URL>`. { Replacing @USERNAME in tweet with `<USERNAME>'. { Replacing number in tweet with `<NUMBER>'. { Replacing repeated character sequences of length 3 or greater with sequences of length 3.

{ Removing punctuation in tweet Then, we use NLTK for tokenization, stemming and splitting the sentences. Retrieval With the daily tweet stream, we leverage the Terrier search engine [ 9 ] for indexing and retrieval. Three state-of-the-art information retrieval(IR) models, namely the BB2 model, the PL2 model and the DFR BM25 model [ 1 ], are utilized for this task. Speci cally, with the three IR models, we can obtain three scores for a tuple as (Topic, Tweet). Each IR model returns 3000 most related tweets.

By assuming that di erent retrieval models may compensate each other by combination, we do a linear combination of the scores to obtain better performance.

Redundancy Detection Since the pushed tweets are expected to cover a variety of arguments given by a user about a culture event, we delete identical tweets through the similarity between two tweets. Speci cally, when a candidate tweets speci c to a topic, we devise a redundancy detection strategy to determine whether it is redundant or not. To calculate the similarity score between two tweets, we rst obtain the corresponding words set as S(T1) and S(T2). Then, the similarity score Score(T1; T2) is formulated as:

Score(T1; T2) = jS(T1) \ S(T2)j jS(T1) [ S(T2)j (1) where S(T1) \ S(T2) is the intersection of S(T1) and S(T2), S(T1) [ S(T2) represents the union of S(T1) and S(T2), j j denotes the size of the set. If Score(T1; T2) is large than the threshold , we determine there are redundant. Reranking We rerank the related tweets by considering whether the tweet contains the topic, the length of the tweets and the number of argumentative words in tweets. In order to obtain lexical feature, we download some English argumentative vocabularies (e.g. admirable,cool, admire, adorable adore, advantage and so on) and combine them together. For French, we translate the English vocabulary into French through Google translation API. Finally, we rerank the tweet T the for topic T opic according to the following function: f (T; T opic) = +

Tlength + (1 ) Narg (2) 8 0; Topic is not in T = < 1; Topic is in T and is continuous : ; Topic is in T and is not continuous (3) where 2 [0; 1] represents whether topic T opic contained in tweet T and whether the topic is continuous in tweet, Tlength is the length of the tweet T after normalizing, Narg denotes the number of words in argumentative vocabulary after normalizing, 2 [0; 1] represents the weight between Tlength and Narg. The complete stream of 70,000,000 microblogs is available. English and French are a respectively 12 and 4 festival name. They represent a set of some popular festivals on FlickR for which we have pictures. Topics were carefully selected by the organizer to ensure that selected topics have enough related argumentative tweets in our corpus. Such manual selection was conduct to to ensure a possible evaluation. 3.2

Evaluation The o cial evaluation measures planned are: NDCG and Pyramid. { NDCG This ranking measures will give a score for each retrieved tweet with a discount function over the rank. As we are mostly interested in top ranked arguments, this ranking measures meet our expectation. This measure was also used in TREC Microblog Track [ 5 ]. A tweet is : - Highly relevant when it is a personal tweet with an argument that directly referred to the festival noun (topic) and may contain more then one justi cation . - Relevant when it comportes at least two of graduation criteria cited above - Not relevant if no graduation criteria was found - Exemple of tweet gradution { Pyramid [ 7 ] This evaluation protocol was chosen to evaluate how much the identi ed set of argumentative tweets about a festival name is diversi ed. In fact, participant results are expected to cover a variety of arguments given by a user about a culture event. Such an evaluation protocol will allow us to determine if the identi ed summary of ranked tweets expresses the same content in di erent words or involve di erent arguments about a given festival name. The experiment results are shown in Table 1 and Table 2. Our observation shows that the proposed model works better than baseline in most cases. 4

Conclusions

In this paper, we present our work in two scenarios of the MC2 2018 task2 mining opinion argumentation . We build a tweet retrieval system. It mainly performs four steps to determine whether to push a tweet or not. We apply three stateof-the-art IR models for search. Various retrieval results are combined for nal delivery. Noting that the combination strategy does not work very well, we will extract more useful features and focus on the learning to rank approaches in the future.

References

1. Amati , G., Van Rijsbergen , C.J. : Probabilistic models of information retrieval based on measuring the divergence from randomness . ACM Transactions on Information Systems (TOIS) 20(4) , 357 { 389 ( 2002 )

2. Amigo , E. , De Albornoz , J.C. , Chugur , I. , Corujo , A. , Gonzalo , J. , Mart n , T., Meij , E. , De Rijke , M. , Spina , D. : Overview of replab 2013: Evaluating online reputation monitoring systems . In: International Conference of the Cross-Language Evaluation Forum for European Languages . pp. 333 { 352 . Springer ( 2013 )

3. Bellot , P. , Moriceau , V. , Mothe , J. , SanJuan , E., Tannier , X. : Inex tweet contextualization task: Evaluation, results and lesson learned . Information Processing & Management 52 ( 5 ), 801 { 819 ( 2016 )

4. Latiri , M. , Cossu , J.V. , Latiri , C. , SanJuan , E.: Clef 2018 . International Conference of the Cross-Language Evaluation Forum for European Languages Proceedings LNCS ( 2018 )

5. Lin , J. , Efron , M. , Wang , Y. , Sherman , G. , Voorhees , E.: Overview of the trec-2015 microblog track . Tech. rep. ( 2015 )

6. Lippi , M. , Sarti , P. , Torroni , P. : Argumentative ranking

7. Nenkova , A. , Passonneau , R. , McKeown , K. : The pyramid method: Incorporating human content selection variation in summarization evaluation . ACM Transactions on Speech and Language Processing (TSLP) 4 ( 2 ), 4 ( 2007 )

8. Oraby , S. , Reed , L. , Compton , R. , Rilo , E. , Walker , M. , Whittaker , S. : And that's a fact: Distinguishing factual and emotional argumentation in online dialogue . NAACL HLT 2015 p. 116 ( 2015 )

9. Ounis , I. , Amati , G. , Plachouras , V. , He , B. , Macdonald , C. , Johnson , D.: Terrier information retrieval platform . In: European Conference on Information Retrieval . pp. 517 { 519 . Springer ( 2005 )

10. Schulz , C. , Eger , S. , Daxenberger , J. , Kahse , T. , Gurevych , I. : Multi-task learning for argumentation mining in low-resource settings ( 2018 )