Microblogs, like Twitter, provide a unique crowdsourcing platform where people across the world can post their opinions or observations about real world events. Twitter is the real time data source which has massive user-generated content. Since tweets are posted by multiple users with diverse views, many tweets have redundant content. Due to enormous volume of the tweets, tweet visualization is the biggest challenge. We can address this challenge by creating a summary from relevant tweet with respect to given topic.
The aim of the Text summarization Data Challenge Track is to evaluate and benchmark different summarization systems on standard social media dataset. The text summarization track is offered in two levels. In the first level, tweets which are posted on the first day of the earthquake in Italy were provided. Tweets posted on second and third day of the Italy earthquake were provided in the second level.
Summarization methods can be divided into two types (i) Extractive Summarization (ii) Abstractive summarization. We have focused on extractive summarization. Basically, Extractive Summarization methods are further divided into 3 types which are (i)) graph based (ii) cluster based (iii) Centroid based. TREC1 has started Microblog track since 2011 with an adhoc retrieval task and converged it into real time summarization in 2016. CLIP [2] used a word embedding technique to expand query. They have used BM25 model to calculate relevance score between tweets and query. For summarization, they used jaccard similarity across relevant tweets. Luchenet. al [4] used simple keyword matching technique which assigns more weight to the original term compared to the expanded term. For summarization, they have used simple word overlap.
Given topics Q = <SMERP-T 1 , SMERP-T 2 , SMERP-T 3 , SMERP-T 4 >, and Tweets DataSet T = <T 1 , T 2 ,..,T n > from the dataset, we need to compute the relevance score between tweets and topics in order to create topic-wise summary S = <S Q1 ....S Qn >.Where S Qi is the set of topic-wise relevant and novel tweets. We can model topic specific summary as below.
SQ1=<T 1 ,T 2 ,…..,T n > where T i ,T j Є T For given topic, Relevance score between tweet and topic must be greater than specified threshold T rel . In addition to this, these tweets should be novel i.e. similarity between all tweet of the summary should less that the novelty threshold T nov .if any tweet T i is included in the summary for a particular topic then it should satisfy the following constraints.
Length of summary of profile(S Qi ) <= 300 word Relevance score(t i , Q i ) >T rel Sim(t i ,t j ) <T nov for all t j Є S Qi
Our Methodology
topics have been provided in TREC format by the track organizers. The topics consist of a title, description and a narrative. The topics might be referred to as queries in the paper. Further, we elaborate our approach.
Topics consist of a title in which the general information needed is given. A description, which is sentence long and a narrative, the content of which is paragraph long gives an elaborate picture of the topic. <narr> Narrative: A relevant message must contain information about relief-related activities of different NGOs and Government organizations engaged in rescue and relief operation. Messages that contain information about the volunteers visiting different geographical locations would also be relevant. Messages indicating that organizations are accumulating money and other resources will also be relevant. However, messages that do not contain the name of any NGO / Government organization would not be relevant.
The topic to query conversion starts with removal of stopwords. We run Stanford POS tagger 2 .The noun and verb labeled keywords are extracted and added to the query. We believe that topics are vague so by human intervention, the query is built.
We have used a lexical database WordNet3 for topic expansion which puts English words into sets of synonyms, synsets. The top two synonyms are extracted and added to the query using Wordnet.
After downloading the tweets, only English tweets were worked on. Further, retweets and tweets with only hashtags, emoticons or special characters were not considered. Also, tweets with less than 5 words were ignored. We removed all the stopwords and non-ASCII character from the tweets.
We have used cosine similarity to calculate the relevance score between tweet and expanded query in the first level. In the second level we have retrieved relevant tweets using language model with Jelinek-Mercer smoothing with parameter λ=0.1.
Tweets are posted by many users at different times from different parts of the world.
To create the text summary from the tweets is a challenging task. Ideally, summary should include all relevant tweets with constraint that it should not include redundant information. Tweet summarization is a multiple document summarization problem. Each tweet can be considered as a single document.
To create the summary, we have selected top tweets from each topic whose relevance score is greater than specified relevance threshold T rel . We have empirically set value of T rel . Now for the next eligible tweet, we calculate it's similarity with tweets already added in the summary so as to ensure novelty between them. Again a Jaccardthreshold t nov =0.6 was decided empirically and tweets below it were added into summary. Lower the similarity score, greater is the dissimilarity ensuring more novelty.
In both levels, our summarization method remain same. However, we have used different tweet retrieval techniques. SMERP 2017 track organizers have considered ROUGE-L as primary metric to evaluate performance of all the runs. The following tables show our results in comparison with the top run.
In this paper, we have implemented a method based on extractive summarization. Table 1 and Table 2 show that our results are comparatively lower than IIEST. In the future we will investigate our underperformance and will carry out post-hoc/ error analysis. We would like to design a summarization system based on deep neural network and logistic regression.