The paper presents IRIT's approach used at INEX Tweet Contextualization Track 2013. Systems had to provide a context to a tweet. This year we further modified our approach presented at INEX 2011 and 2012 underlain by the product of scores based on hashtag processing, TF-IDF cosine similarity measure enriched by smoothing from local context and document beginning, named entity recognition and part-of-speech weighting. We assumed that relevant sentences come from relevant documents therefore we multiply sentence score by document relevance. We also used generalized POS (e.g. we merge regular adverbs, superlative and comparative into a single adverb group). We introduced sentence quality measure based on Flesch reading ease test, lexical diversity, meaningful word ratio and punctuation ratio. Our approach was ranked first, second and third over 24 runs submitted by all participants on different reference pools according to informativeness evaluation. At the same time it obtained the best readability score.
Twitter is an online social network and microblogging that enables to send and read
text messages up to 140 characters [
The paper presents IRIT’s approach used at INEX Tweet Contextualization Track
2013. We consider tweet contextualization task as multi-document extractive
summarization. This year we further modified our approach presented at INEX 2011 [
The paper is organized as follows. Firstly, we recall the principles of the 20112012 system we developed and describe the modifications we made. Then, we present the results and discuss them. Future development description concludes the paper. 2 2.1
Preprocessing includes several steps.
Firstly, we treat tweets themselves, i.e. special symbols like hashtags and replies.
The hashtag symbol # “is used to mark keywords or topics in a Tweet. It was
created organically by Twitter users as a way to categorize messages” and facilitate a
search [
Moreover, important information may be found in @replies, e.g. when a user reply
to the post of a politician or other famous person. “An @reply is any update posted by
clicking the "Reply" button on a Tweet” [
We assume that relevant sentence come from relevant documents, so we applied a
search engine to find them. We use the tweet as a query. We choose the Terrier
platform [
The next step is to parse tweets and retrieved texts. For the English subtrack we
applied Stanford CoreNLP which integrates such tools as POS tagger [
Then, we merged annotation obtained by parsers and Wikipedia tagging. 2.2
We modified the extraction component developed for INEX 2011-2012. The general idea of the approach 2011 was to compute similarity between the query and sentences and to retrieve the most similar passages.
We model a sentence as a set of vectors. The first vector represents the tokens occurred within the sentence (unigram representation). Tokens are associated with lemmas. A lemma has the following features: POS, frequency and IDF. The second vector corresponds to bigrams. In both vector representation stop-words are retrieved. However, functional words, such as conjunctions, prepositions and determiners, are not taken into account in the unigram representation. NE comparison is hypothesized to be very efficient for contextualizing tweets about news. Therefore, the third vector refers to found named entities. Thereby, the same token may appear in several vectors.
For unigram and bigram vectors, we computed cosine, Jaccard and dice similarity measures, between a sentence and a target tweet. NE vectors are treated in the following way: (1) where is floating point parameter given by a user (by default it is equal to 1.0), is the number of NE appearing in both query and sentence, is the number of NE appearing in the query.
Each sentence has a set of attributes, e.g. which section it belongs to, whether it is a title or header, whether it has personal verbs etc.
We introduced an algorithm for smoothing from the local context. We assumed that the importance of the context reduces as the distance increases. Thus, the nearest sentences should produce more effect on the target sentence sense than others. For sentences with the distance greater than k this coefficient was zero. The total of all weights should be equal to one. The system allows taking into account k neighboring sentences with the weights depending on their remoteness from the target sentence.
Moreover, this year we added smoothing from document beginning. Wikipedia abstracts contain the summary of the entire paper; therefore they can be also used for smoothing.
In 2013, we did not applied anaphora resolution since it did not improve much our
system according to evaluation in 2012 [
We assumed that relevant sentences come from relevant documents therefore we multiply sentence score by document relevance or/and by inverted document rank. We tried to use generalized POS (e.g. we merge regular adverbs, superlative and comparative into a single adverb group). 2.3
We introduced sentence quality measure based on the product of the Flesch reading
ease test [
Flesch Reading Ease test is a readability test designed to indicate comprehension difficulty when reading a passage (higher scores corresponds to texts that are easier to read):
We defined lexical diversity as the number of different lemmas used within a sentence divided by the total number of tokens in this sentence.
Analogically, meaningful word ration is the number of non-stop words within a sentence divided by the total number of tokens in this sentence.
Punctuation score is estimated by the formula: (2) (3)
In order to treat redundancy each sentence was mapped into a noun set. These sets were compared pairwise and if the normalized intersection was greater than a predefined threshold the sentences were rejected. 3
Summaries in English were evaluated according to their informativeness and
readability [
As in previous years, the lexical overlap between a summary and a pool was estimated in three terms: Unigrams, Bigrams and Skip bigrams representing the proportion of shared unigrams, bigrams and bigrams with gaps of two tokens respectively. Official ranking was based on decreasing score of divergence with ALL estimated by skip bigrams.
At the English subtrack we submitted 3 runs differing by sentence quality score and smoothing.
Our best run 275 was ranked first, second and third over 24 runs submitted by all participants on the PRIOR, POOL and ALL respectively (see Table 1; IRIT’s runs are set off in bold). It means that our best run is composed from the sentence of the most relevant documents. Among automatic runs our method was classified first (PRIOR and POOL) and second (ALL): the run 256 is marked as manual. It is also obvious that ranking is sensitive to not only pool selection, but also choice of divergence. According to bigrams and skip bigrams our best run is 275, while according to unigrams the best run is 273. We can also see than the runs 273 and 274 are quite close. In the run 273 each sentence is smoothed by its local context and first sentences from Wikipedia article which it is taken from. The run 274 has the same parameters except it does not have any smoothing. So, we can conclude that smoothing improves Informativeness. In our best run 275 punctuation score is not taken into account, it has slightly different formula for NE comparison and no penalization for numbers.
Readability was estimated as mean average scores per summary over soundness (no unresolved anaphora), non-redundancy and syntactical correctness among relevant passages of the ten tweets having the largest text references. According to all metrics except redundancy our approach was the best among all participants (see Table 2; IRIT’s runs are set off in bold). Runs were officially ranked according to mean average scores. Readability evaluation also showed that the run 275 is the best by relevance, soundness and syntax. However, the run 274 is much better in terms of avoiding redundant information. The runs 273 and 274 are close according readability assessment as well. k n a R n u
R 256 258 275 273 This year we further developed our approach firstly introduced at INEX 2011 which is based on hashtag processing, TF-IDF cosine similarity measure enriched by smoothing from local context and document beginning, named entity recognition and part-of-speech weighting. We enriched our method by sentence quality measure based on Flesch reading ease test, lexical diversity, meaningful word ratio and punctuation ratio. We also used generalized POS (e.g. we merge regular adverbs, superlative and comparative into a single adverb group). Sentence score depends on document relevance and sentence type.
We submitted 3 runs in English differing by sentence quality score and smoothing and 1 run in Spanish.
Our approach was ranked first, second and third over 24 runs submitted by all participants on the PRIOR, POOL and ALL respectively. Among automatic runs our method was classified first (PRIOR and POOL) and second (ALL).
Readability was estimated as mean average scores per summary over resolved anaphora, non-redundancy and syntactical correctness among relevant passages of the ten tweets having the largest text references. According to all metrics except redundancy our approach was the best.
In future we plan to automatize parameter selection by machine learning methods.