Social networks on the Internet are becoming increasingly important in our society. In recent years, this type of media, through communication platforms such as Twitter, has brought new research issues due to the massive size of data exchanged and the important number of ever-increasing users. In this context, the CLEF 2018 Mining opinion argumentation task aims to retrieve, for a speci c event (festival name or topic), the most diverse argumentative microblogs from a large collection of tweets about festivals in di erent languages. In this paper, we propose a four-step approach for extracting argumentative microblogs related to a speci c query (or event) while no reference data is provided.
Social networks on the Internet allow communities of users to exchange and share
resources worldwide (ideas, opinions, data...) to an increasingly wide audience.
Researchers, particularly in Natural Language Processing (NLP) and
Information Retrieval (IR) domains, have seized this phenomenon, unprecedented by
the number of users that these networks aggregate and the size of the data
exchanged (texts, videos, audio...), opening up new research issues. Through these
communication platforms, users can gather around a speci c event (news [
The CLEF 2018 Mining opinion argumentation task aims to automatically
identify messages of social web users positions about a cultural event expressed
through the Twitter social network platform. The idea is to identify claims about
a festival name, or topic, out of a massive collection of microblogs. The objective
? This work was funded by the GaFes project supported by the French National
Research Agency (ANR) under contract ANR-14-CE24-0022.
is to provide relevant information expressed in the form of a summary of
argumentative tweets about a query (here a festival name or a topic) that should
re ect a maximum of di erent points of view. This follows a previous task
initiated in [
These last years, sentiment analysis and opinion mining [
In this article, we propose an original four-steps approach to train a CNN model for extracting argumentative microblogs related to a speci c query (or event) while no reference data is provided (and no data will be annotated).
The paper is organized as follows. Section 2 explains our proposed four-steps approach to identify a set of argumentative microblogs from a cultural event. Section 3 describes the experimental protocol, including a description of the task and the data used. Finally, Section 4 presents the results obtained in the CLEF 2018 Mining opinion argumentation task before concluding and exposing perspectives in Section 5. 2
In this section, we describe our proposed method to extract argumentative messages from a targeted query. Figure 1 summarizes our four-steps approach. The rst step (Section 2.1) consists in preprocessing raw unlabeled messages to make them \cleaner", i.e. make the data more easily interpretable and generalizable by an automatic process. The second step (Section 2.2) takes as input the cleaned data and proposes a method to extract two datasets (Argumentative and Non Argumentative) while no labeled data is provided. From these two datasets, a convolutional neural network (CNN) is trained in Step 3 to recognize argumentative and non argumentative messages (Section 2.3). Finally, the last step (Section 2.4) seeks to extract, from a set of messages related to a query (test set), the list of messages which contains the most argumentative elements while including a maximum of diversity in the opinions conveyed. 2.1
In a general way, text messages need a preprocessing step to be then used as e ciently as possible in many NLP tasks. Usually, this process includes a global \cleaning" of the data. We rst propose to tokenize words in order to better treat them individually. For example, the tweet \It's Friday, it's Swansea Jazz Festival its cocktail night at Morgan's." becomes \It 's Friday , it 's Swansea Jazz Festival its cocktail night at Morgan 's .".
Some speci cities of tweet microblogs are also taken into account. Since URLs can be added in messages, we propose to make them unique by changing any URL present in a tweet by <URL>. Nonetheless, as we think that hashtags (#example) and references to other users (@user) are important information, we did not make any preprocess on it.
In many NLP applications [
Raw unlabeled Step 1: Preprocessing messages
Cleaned messages
Step 2: Unlabeled data selection Query
Test
Train Data related to the query
Argumentative messages data
Non Argumentative
messages data input
2.2
While no reference data is available, we propose to \infer" this reference using a semi-supervised approach. For this unlabeled argumentative message data selection process, we rstly only keep messages tagged as the focused query language by the Twitter platform. Since this is an automatic process, errors in language identi cation may occur (and datasets may be di erent with another language identi cation tool).
We then get lists of opinion words that come from the French expanded
emotion lexicon FEEL [
Note that we do not de ne our training corpus only regarding these lists of opinion words: if so, it would simply amount to seeing the presence or absence of an opinion word to decide if a message is argumentative or not. We introduce new knowledge related to the corpus of microblogs studied: we make the hypothesis that a message can also be informative if it contains emoticons, particular punctuation signs such as ? or !, if the personal pronoun Je (in French) or I (in English) is employed, or if at least one hashtag is present. Possessive pronouns and personal pronouns are considered indicators of argumentative tweets for their expressive propriety. In particular rst person and second person place the author in a communicational context expressive or conative.
In summary, we then have 5 features to decide if a message is informative (emotion words, emoticons, particular punctuation signs, personal pronoun, and hashtag). If a message contains at least 4 of these 5 features, or an emotion word plus 2 of the 4 other features, it is considered as argumentative. At the contrary, if a message does not have any of these characteristics, or only 1 feature (excluding an opinion word), it is considered as non argumentative.
This nally allows us to get two datasets for training: Argumentative and Non argumentative. Note that these train datasets have been extracted from all data excluding the data related to the targeted query, which constitutes here our database to search argumentative messages (i.e. test set). To constitute this test set, we consider a message related to a query if the words of the query are present in the message. For example, if the targeted query is \Avignon" for the French language, all messages containing the term Avignon and being tagged as French are in the test set, while all the remaining messages in the corpus (tagged as French) may be used to constitute the training data. 2.3
Convolutional neural networks (CNNs) represent one of the most used Deep
Neural Network model in computer vision [
In sentence classi cation, each input consists of a sequence of words of variable length. Each word w is represented with a n-dimensional vector (word embedding) ew of constant size. All the word representations are then concatenated in their respective order and padded with zero-vectors to a xed length (maximum possible length of the sentence).
The parameters of our model were chosen so as to maximize performance on the development set (10% from the train data presented in Section 2.2): the width of the convolution lters is set to 5 and the number of convolutional feature maps is 200. We use ReLU activation functions and a simple max-pooling. One fully connected hidden-layers are of size 128. For each layer, a standard dropout of 0.4 (40% of the neurons are disabled in each iteration) is used. The backpropagation algorithm used for training is Adadelta. 2.4
This last step allows us to constitute the list of argumentative message candidates. To do so, all the data test set (i.e. messages related to the query - see Section 2.2) is processed through the previously trained CNN. As a result, a score is assigned to each message that represents the probability of this message to be argumentative. A rst ranked list can then be obtained with this classi cation process.
However, this rst list does not respect the expected criterion of diversity of opinions: the list should re ect the maximum of argumentative points-of-view from a query (or event). In order to only keep enough di erent views, we compute a cosine similarity between a candidate message and the messages stored in this new list. Messages having a similarity higher than 0.5 are then excluded. For example, for the query Rock festival in English, if we get the following ordered list of candidate argumentative messages : 1. common dave ! ! fuck the festival setting ! ! bless u with your awesome sitting acoustic rock ! ! ! ! ! #foo ghters #pinkpop 2. managed to rock up in bordeaux on the weekend of both the gay pride festival and the main wine expo . #party 3. managed to rock up in bordeaux on the weekend of both the gay pride festival and the main wine expo
The rst message is automatically added to the nal candidate list. Then, the cosine distance will be computed between the rst and the second message: since they are di erent enough, the second message will also be added to the nal list. For the third message, the cosine distance is computed with all the messages from the nal list (messages 1 and 2): for the second message, the cosine distance is too close, the message 3 then does not nally appear in the nal list of argumentative message candidates.
The proposed approach has been assessed in the context of the CLEF 2018
Mining opinion argumentation task [
The general objective of the task is to nd, for a speci c topic or event, the most argumentative microblogs. These short messages come from a large collection of tweets about festivals in di erent languages. The idea is to get a list of ranked tweets, for each topic in a targeted language, according to their probability of being argumentative. Also, one key point lies in the opinion argumentation diversity provided in this list: a wide range of di erent points-of-view expressed in the tweets must be present (i.e. avoiding as much as possible identical argumentations).
This task may be of great interest to get a quick overview of opinions shared during an event from social networks since it is usually impossible to manually analyze all emitted messages. As a result, a set of 100 messages for each query (i.e. topic or event) must be given, each one being associated with a probability that the tweet is argumentative. 3.2
The CLEF 2018 Mining opinion argumentation task comes with a large
collection of microblogs containing a stream of 70 million tweets in 134 di erent
languages extracted from the Twitter platform. This dataset has been collected
over a period of 18 months from May 2015 to November 2016 using a
predened set of keywords related to cultural festivals in the world [
Regarding the targeted task, organizers propose to focus on two languages: French and English, from which 4 and 12 topics ( .e. queries) have been de ned respectively. These queries have been chosen to match with festival names or topics. As explained by the organizers, these queries have enough related argumentative tweets to be evaluated. Table 1 lists these di erent topics or festival names for each considered language (French and English). For readability reasons, this list is presented in a descending order from the most popular topic (i.e. having the highest number of tweets) to the less popular one (i.e. having the smallest number of messages), each language being considered independently.
5 A retweet is a forwarded message on Twitter. It is not an original post, but it is considered as a message.
Note that a message is linked to a festival name (or a topic) if it is present in the tweet content, no matter the language considered for now since there is no sure way (i.e. not automatic) to know the language of a tweet. The term Festival is excluded from this search since we assume that it is a Festival oriented corpus. An example of a tweet related to the Cannes Festival, where Cannes occurs:
At Cannes Film Festival, Dheepan Wins Palme dOr.
By analyzing Table 1 more precisely, we nd that the festivals do not have the same activity as for the messages exchanged, with a huge di erence between the most popular queries and the less popular ones. The Cannes Festival, which is the only festival name considered in both English and French languages, is the most represented in terms of posted messages. This is not surprising since it is a world famous festival. In the same way, the selected topics (Rock, Jazz, Summer and Art), chosen for being generic words, have a high level of activity, even if Summer appears well above others. Finally, the remaining festival names have the lowest number of tweets.
While these rst observations may inform about the general corpus and this imbalanced queries data, Table 2 presents the dataset used for training our proposed system. Two subsets for training have been extracted for each query (Argumentative and Non argumentative). As expected, we nd that many fewer tweets are annotated argumentative. A last subset, called Test, is composed of all the tweets containing the query. More information about this unlabeled argumentative tweet data selection process can be found in Section 2.2.
Globally, we can rstly note that the imbalance in the data sizes (Table 2)
is clearly reduced compared to Table 1. The Cannes festival remains the most
commented festival name and the topics Rock, Jazz, Summer and Art datasets
still have a high number of associated messages. Finally, for some festival names
(especially for the English language), a very limited number of test data will be
available, which may make it di cult to get 100 argumentative microblogs.
The metric used to evaluate systems submitted to CLEF 2018 Mining opinion
argumentation task is the Normalized Discounted Cumulative Gain (NDCG) [
Table 3 summarizes the results obtained by our system for the CLEF 2018
Mining opinion argumentation task in terms of NDCG score. For this task, two
reference evaluation sets (i.e. sets of argumentative tweets) are considered: a manual
one, which corresponds to a ne manual annotation from the whole corpus, and
a pooling one, which corresponds to a manual annotation from the tweets
considered as argumentative by participants. For sake of comparison, three other
systems are evaluated: the CLEF 2018 baseline [
By rstly focusing on the manual reference set, we can see that our proposed systems reach the best NDCG scores. Surprisingly, our baseline system (only opinion words) reaches similar results than our proposed system. This could be explained by the fact that opinion words may not be the only information to de ne what is an argumentative tweet. When focusing on the pooling reference, results are quite di erent: other participants systems reach much better performance. As a conclusion, we think that our system seems more robust regarding the whole corpus (best performance in the manual reference) by providing more diverse results than other participants (low performance in the pooling reference). All these observations are similar on French and English queries. 5
In this paper, the problem of retrieving argumentative microblogs from a large collection of messages was addressed. This work took place in the context of the CLEF 2018 Mining opinion argumentation task that aims to retrieve, for a speci c event (festival name or topic), the most argumentative tweets from a Twitter festival-oriented corpus. To do so, we proposed an original CNN-based approach that takes into account the fact that no reference data is available (i.e. no tweets are annotated for training). As a result, a ranked list of the 100 most argumentative tweets, including an argumentative probability score for each message, has been provided for each query.
Results obtained on this evaluation campaign task appear encouraging, considering in particular the di culty of the task. Indeed, our proposed approach reached best performance among all the participants on the manual reference. We also noted that this approach provides results very di erent from other participants, which has been observed on the pooling reference results. This could in particular open up perspectives of complementarity of the systems proposed for this evaluation campaign.
Many research perspectives can be gleaned from this preliminary work. Firstly,
a robust language identi cation tool should be employed to select appropriate
database. Another more interesting perspective would be to take account of the
language level and particularity of tweet contents: indeed, microblogs exhibit
particular linguistic characteristics (ungrammaticality, community-speci c
linguistic traits, misspelling...), not treated in this work. For example, a preprocessing
method, such as [