Overview of INEX Tweet Contextualization 2013 track

Abstract

Twitter is increasingly used for on-line client and audience shing, this motivated the tweet contextualization task at INEX. The objective is to help a user to understand a tweet by providing him with a short summary (500 words). This summary should be built automatically using local resources like the Wikipedia and generated by extracting relevant passages and aggregating them into a coherent summary. The task is evaluated considering informativeness which is computed using a variant of Kullback-Leibler divergence and passage pooling. Meanwhile e ective readability in context of summaries is checked using binary questionnaires on small samples of results. Running since 2010, results show that only systems that e ciently combine passage retrieval, sentence segmentation and scoring, named entity recognition, text POS analysis, anaphora detection, diversity content measure as well as sentence reordering are e ective.

Keywords

• Short text contextualization
• Tweet understanding
• Automatic summarization
• Question answering
• Focus information retrieval
• XML
• Natural language processing
• Wikipedia
• Text readability
• Text informativeness

-

Text contextualization [ 8, 7 ] di ers from text expansion in that it aims at helping a human to understand a text rather than a system to better perform its task. For example, in the case of query expansion in IR, the idea is to add terms to the initial query that will help the system to better select the documents to be retrieved. Text contextualization on the contrary can be viewed as a way to provide more information on the corresponding text in the objective to make it understandable and to relate this text to information that explains it.

In the context of micro-blogging, which is increasingly used for many purposes such as for on-line client and audience shing, contextualization is specifically important since 140 characters long messages are rarely self-content. This motivated the proposal in 2011 of a new track at Clef INEX lab of Tweet Contextualization.

The use case is as follows: given a tweet, the user wants to be able to understand the tweet by reading a short textual summary; this summary should be readable on a mobile device without having to scroll too much. In addition, the user should not have to query any system and the system should use a resource freely available. More speci cally, the guideline speci ed the summary should be 500 words long and built from sentences extracted from a dump of Wikipedia. Wikipedia has been chosen both for evaluation purpose and because this is an increasing popular ressource while being generally trustable. In this paper, details the 2013 track set up and results. The use case and the topic selection remained stable since 2011[ 8 ], so that 2011 and 2012 topics could be used as a training set. However, In 2013 we considered more diverse types of tweets for this year edition, so that participants could better measure the impact of hashtag processing on their approaches.

The remaining of the paper is organised as follows: In section 2 we describe in detail the 2013 data collection. Section 3 presents the results and Section 4 concludes this paper. 2

Data collection

This section describes the document collection that is used as the resource for contextualization, as well as the topics selected for the test set which correspond to the tweets to contextualize.

The document collection has been built based on a recent dump of the English Wikipedia from November 2012. Since we target a plain XML corpus for an easy extraction of plain text answers, like in past years, we used the same perl programs released for all participants to remove all notes and bibliographic references that are di cult to handle and keep only non empty Wikipedia pages (pages having at least one section).

Resulting automatically generated documents from Wikipedia dump, consist of a title (title), an abstract (a) and sections (s). Each section has a sub-title (h). Abstract and sections are made of paragraphs (p) and each paragraph can contain entities (t) that refer to other Wikipedia pages.

Over 2012 and 2013 editions, evaluated topics were made of 120 (60 topics each year) tweets manually collected by organizers. These tweets were selected and checked, in order to make sure that: { They contained \informative content" (in particular, no purely personal messages); Only non-personal accounts were considered (i.e. @CNN, @TennisTweets, @PeopleMag, @science. . . ). { The document collection from Wikipedia contained related content, so that a contextualization was possible.

From the same set of accounts, more than 1,800 tweets were then collected automatically. These tweets were added to the evaluation set, in order to avoid that fully manual, or not robust enough systems could achieve the task. All tweets were then to be treated by participants, but only the 120 short list was used for evaluation. Participants did not know which topics were selected for evaluation.

These tweets were provided in a text-only format without metadata and in a JSON format with all associated metadata. 3

Results

This year the entire evaluation process was carried out by organizers.

Tweet contextualization [ 6 ] is evaluated on both informativeness and readability. Informativeness aims at measuring how well the summary explains the tweet or how well the summary helps a user to understand the tweet content. On the other hand, readability aims at measuring how clear and easy to understand the summary is.

Informativeness measure is based on lexical overlap between a pool of relevant passages (RPs) and participant summaries. Once the pool of RPs is constituted, the process is automatic and can be applied to uno cial runs. The release of these pools is one of the main contributions of Tweet Contextualization tracks at INEX[ 8, 7 ].

By contrast, readability is evaluated manually and cannot be reproduced on uno cial runs. In this evaluation the assessor indicates where he misses the point of the answers because of highly incoherent grammatical structures, unsolved anaphora, or redundant passages. Like in 2012, three metrics were used: Relevancy (or Relaxed) metric, counting passages where the T box has not been checked; Syntax, counting passages where the S box was not checked either, and the Structure (or Strict) metric counting passages where no box was checked at all.

Participant runs were ranked according to the average, normalized number of words in valid passages.

In 2013, a total number of 13 teams from 9 countries (Brasil, Canada, France, India, Ireland, Mexico, Russia, Spain, USA) submitted 24 runs to the Tweet Contextualization track in the framework of CLEF INEX lab 2013.

Infomativity results are presented in Table 1 and statistical signi cance of di erencies between scores are indicated in Table 2. Table 1 shows readability scores.

This year, the best participating system (199) used hashtag preprocessing introduced in [ 1 ]. The best run by this participant used all available tweet features including web links which was not allowed by organisers. However his second best run without using linked web pages is ranked rst among o cial runs. This participant also tried to weight hashtags based on 2012 results but this did not improve results. Perhaps because topics evaluated in 2012 were too speci c.

Second best participant (182) in informativity and best in readability used state of the art NLP tools. This participant was rst in informativity in 2011 [ 2 ]. Di erences between these two best systems are not statistically signi cant.

Third best participant system (65) was rst in 2012 [ 4 ], so the same system performs well even on a more diversify set of tweets.

Reference system by organisers (62-276) available online through an API is not more among three best systems. This systems is a robust focused information retrieval system [ 6 ] that was not smoothed for tweets. This year we also set up a baseline (62 - 278) using a state of the art IR system on sentences. Its informativity scores are high but its readability is very low.

Overall, informativity and readability scores are this year strongly correlated (Kendall test: > 90%, p < 10 3) which shows that all systems have integrated this constrain. Remenber that since 2012, readability is evaluated in the context of the tweet. Passages not related to the tweet are considered as unreadable.

4 Conclusions

Like in 2012, almots all participants used language models.

Terminology extraction and reformulation applied to tweets was also used in 2013 like in previous editions [ 9 ]. Appropriate stemming and robust parsing of both tweets and wikipedia pages also seems to be an important issue. Most systems having a run among the top ten in informativeness used the Standford Core NLP tool or the TreeTagger.

It also seems that automatic readability evaluation and anaphora detection helps improving readability scores, but also informativeness density in summaries. It is now clear that state of the art summarization methods based on sentence scoring [ 5 ] proved to be helpful on this task even though they need to be combined with an IR engine.

Best run in 2013 also experimented a tweet hashtag scoring technique introduced in 2012 [ 1 ] while generating the summary.

Finally, this time the state of the art system proposed by organizers since 2010 combining LM indexation, terminology graph extraction and summarization based on shallow parsing was not ranked among the six best runs which shows that participant systems improved on this task over the three editions.

References

1. Deveaud , R. , Boudin , F. : Lia/lina at the inex 2012 tweet contextualization track . In: Forner et al. [3]

2. Ermakova , L. , Mothe , J.: Irit at inex 2012: Tweet contextualization . In: Forner et al. [ 3 ]

3. Forner , P. , Karlgren , J. , Womser-Hacker , C . (eds.): CLEF 2012 Evaluation Labs and Workshop, Online Working Notes, Rome, Italy, September 17-20 , 2012 ( 2012 )

4. Ganguly , D. , Leveling , J. , Jones , G.J.F. : Dcu@ inex -2012: Exploring sentence retrieval for tweet contextualization . In: Forner et al. [3]

5. Moreno , J.M.T. , Velazquez-Morales , P. : Two statistical summarizers at inex 2012 tweet contextualization track . In: Forner et al. [3]

6. SanJuan , E., Bellot , P. , Moriceau , V. , Tannier , X. : Overview of the inex 2010 question answering track (qa@inex) . In: Geva, S. , Kamps , J. , Schenkel , R. , Trotman , A . (eds.) INEX. Lecture Notes in Computer Science , vol. 6932 , pp. 269 { 281 . Springer ( 2010 )

7. SanJuan , E., Moriceau , V. , Tannier , X. , Bellot , P. , Mothe , J.: Overview of the inex 2012 tweet contextualization track . In: Forner et al. [3]

8. SanJuan , E., Moriceau , V. , Tannier , X. , Bellot , P. , Mothe , J.: Overview of the inex 2011 question answering track (qa@inex) . In: Geva, S. , Kamps , J. , Schenkel , R . (eds.) Focused Retrieval of Content and Structure, Lecture Notes in Computer Science , vol. 7424 , pp. 188 { 206 . Springer ( 2012 )

9. Vivaldi , J., da Cunha , I.: Inex tweet contextualization track at clef 2012: Query reformulation using terminological patterns and automatic summarization . In: Forner et al. [3]