In this article we deal with the Topic Detection and Priority Detection subtasks from RepLab 2013, trying clustering and classi cation methods as well as term selection techniques in order to know its performance in two sub collections of tweets: single and extended (single tweet plus derived tweets). Our tests show good performance in spite of we used very few resources.
Twitter has become a very popular social interaction place where the users
give their opinions about the companies and their products via tweets. Because
of that Twitter has become a signi cant repository of opinions on companies,
brands, and persons, such entities have interest to protect their reputation and
try to deal with non founded gossips that can a ect their image and incomes.
RepLab 2013 [
Performing such a monitoring of tweets is a signi cantly challenging task
given that the tweet messages are very short and noisy. Some authors try to face
these problems through the idea of concept term expansion in tweets performing
one or more clustering phases of and priority level, as well as unsupervised
clustering techniques. Additionally, some authors use supervision for priority
level assessment [
In this paper we explored clustering and classi cation methods as well as term selection techniques in order to know its performance over two sub collections of tweets: single and extended (single tweet plus derived tweets). Our tests show good performance in spite of we used very few resources. In the following section we describe the data and its preprocessing. Section 3 outlines the methods applied to data collection for Topic Detection subtask. At Section 4 we detail attributes and our approach followed for tweet Priority Detection subtask. The results are presented at Section 5, and nally, our conclusions are depicted in Section 6. 2
The corpus of tweets that was used in the experiments was formed considering two kinds of texts: main text of the tweet (Main) as well as the derived conversation from the main text (All) Thus, from everyone, training and testing collections, we build up two collections, namely: { Main-Training, { All-Training, { Main-Testing, and { All-Testing.
These collections allow us to organize the experiments on pairs: given a particular method for some subtask, it was applied on each collection as two independent runnings.
In order to perform Topic Detection subtask it is important to mention that in all experiments it was only taken into account the information contained in the text of the tweets. It is worth mentioning that we classify and cluster tweets for each entity in order to ease the subtask. Additionally, for each entity we work on two subsets: English and Spanish tweets, ltered through the Language attribute.
All experiments for Topic Detection subtask carried out the same
preprocessing: removing stop words, morphological and in ectional endings (Porter
Stemming)[
Also, Priority Detection subtask was applied to each entity, considering some attributes extracted from the Training and Testing collection; it was not used Main/All collection as in Topic Detection subtask. Furthermore, at Priority Detection subtask the tweet text and some of the related attributes were extracted from the tweet html le and they were stored as plain text le. When there were responses to the tweet text, they were attached to the text plain text le. At this process, we observed, in the gold standard, some tweets were not related to any entity and then, they were discarded. 3
At the rst four experiments of Topic Detection subtask, we pretended to know how useful is training set aiming to identify topics in twitter. Experiments seven and eight are completely unsupervised and try to compare its performance to the previous rst to sixth tests. 3.1
We applied two classi cation algorithms, Naive Bayes and Sequential Minimal
Optimization Support Vector Machines (SMO SVM) [
We performed three pairs of experiments on an unsupervised manner, two of them select terms accordingly with the percentage of terms which obtains the best performance on the training dataset. A nal pair of experiments applies a method to automatically select the set of terms used at the representation of tweets. 3.3
In order to improve unsupervised Topic Detection, two term selection methods
were tested: the well known document frequency index, DF, number of documents
which contain the term; and the transition point, TP, frequency which divides
into high and low the term frequencies [
This test was supported on the diversi cation and uni cation concepts proposed
by Zipf [
Given a collection of documents C = fd1; : : : ; dng, it is de ned as U (ti) = 1 X sim(ti; tj ); r
j6=i S^(C) =
2 n(n 1)
X sim(di; dj );
i6=j
where t is the representation of the term t given by the classes in which t occurs,
r = #ftj jsim(ti; tj ) 6= 0)g, and sim is a similarity measure. In our experiments
we used Jaccard coe cient as similarity measure, and the classes, in order to
represent terms (t), were provided by the clustering of the tweets on the same
working collection (without term selection). Here we used the K-Star
clustering algorithm [
Summarizing, the method follows two steps: 1. Select terms basing on U and S^: (a) Calculate U (t) for all terms of C and sort them in increasing order, namely TU = [U (t1); : : : ; U (tk)]. (b) Divide TU into m parts, in order to provide m sets of terms: Vi; (1 i m) it represents the rst i parts of terms (our experiments used m = 10). (c) Compute the array [S^(Ci)] whose elements correspond to each selection set Vi, and determine the index of the maximum descending value of S^(Ci): j. 2. Apply of the K-Star clustering algorithm to Cj . 3.4 and 4 4.1
From the plain text les, a set of seven attributes were calculated, those are described as follows: 1. Referenced users: calculated from the number of user tags or email found within the tweet text, i.e. the number of tokens with the form @string are considered as referenced users. 2. Hashtags: the number of hashtags symbols are counted (#). 3. Web addresses: the number of http tokens are considered. 4. Tweet length in characters. 5. Frequency of retweets. This is a measure contained in the tweet information and it is considered as an attribute. 6. Frequency of favorites. This is a measure of popularity of the tweet, it is contained in the tweet information and also is considered as an attribute too. 7. Conversation Generated. This is a boolean attribute calculated from the presence or absence of responses to each tweet.
So, every tweet text le belonging to each entity was processed in order to calculate those seven attributes for the sake to classify them as MILDLY IMPORTANT, ALERT and UNIMPORTANT according to the training set. 4.2
It was used the WEKA application to perform three runs in order to classify
the test tweets as it was required for the Priority Detection subtask. So, three
classi ers where applied to the les of attributes calculated for each tweet
belonging to the entities: the tree inducer algorithm J48, the Naive Bayes and the
SMO function (Support Vector Poly Kernel) [
At it was stated, the training was executed over the training collection In some cases there were entities with missing tweets belonging to the ALERT class. Those les were lled with 30 tweets from all the entities for the sake to preserve the three classes in all the training set and, thus, to obtaining appropriated classi er's models. So, the three classi ers were applied to the test set as were mentioned. 5 5.1
Four pair of experiments were carried out. Each pair deals with a pair of collections: Main-Testing (Main-Training), and All-Testing (All-Training), as described at Sec. 2. By instance, the Pair One consists of the runnings UAMCLyR topic detection 1 and UAMCLyR topic detection 2 which use the collections Main-Testing and All-Testing, respectively.
Table 1 depicts how each experiment pair was performed: method, approach (classi cation/clustering), and the term selection criterion.
Finally, at Table 2 we show for each run the used collection, and in descending
order the F values based on Reliability and Sensitivity as well as the Baseline,
de ned in [
As we can see, all our experiments are above of the Baseline. We observed a better performance with clustering than classi cation. Term selection based on U and S^ provided the best result, however, this method was not able to determine best terms when the collection was extended from Main to All.
The results of the three experiments of Priority Detection subtask can be
seen in Table 4 following a descending order according to F value based on
Reliability and Sensitivity and the Baseline, as de ned in [
In all cases of Topic Detection, clustering approach outperformed to classi cation approach. Additionally, we realized that when the derived conversation from tweets was included, the detection got worse. Particularly, it can be observed that in all supervised Topic Detection experiments, the term selection method was unable to correctly discriminate the relevant terms when extending the corpus; i.e. from Main to All collection. The term selection based on uni cation provided the best results, perhaps because it was calculated directly from test collection. However, uni cation term selection it is not sensitive to the increasing of the vocabulary. We plan to carry out additional tests mainly to the term selection techniques using at clustering of tweets.
It can also be claimed that it is possible to detect priority in tweets based on models of classi cation that rely only in some attributes calculated from the metadata features of tweets. From this models it can be obtained acceptable results when classifying new instances. As further work, the method of Priority Detection could be tested in two ways in order to be improved: { to separate the tweets by language { and the extraction of models based on other attributes which take into account linguistic features of texts.