In this paper we describe our participation in the INEX 2013 Tweet Contextualization track and present our contributions. Our approach is the same as last year, and is composed of three main components: preprocessing, Wikipedia articles retrieval and multi-document summarization. We however took advantage of a larger use of hashtags in the topics and used them to enhance the retrieval of relevant Wikipedia articles. We also took advantage of the training examples from last year which allowed us to learn the weights of each sentence selection feature. Two of our submitted runs achieved the two best informativeness results, while our generated contexts where almost as readable as those of the most readable system.
Tweets are short and ambiguous by nature and it can be hard for a user without any background knowledge to understand what the Tweet is about. The INEX Tweet Contextualization track makes the assumption that it is possible to overcome this lack of knowledge by providing the user with a bunch of sentences that give some context or additional information about the Tweet. While topical information may certainly be the most important for this task, one may also want some political context to understand a sarcastic Tweet for example. Our approach speci cally focuses on the topical context, and aims at producing informative contexts.
Within the framework of this track, sentences must be extracted from the
version of Wikipedia provided by the organizers. Our approach sequentially involves
Information Retrieval (IR) and Text Summarization (TS) techniques. First, we
extend the Tweet's topical context by retrieving related Wikipedia articles that
are likely to contain contextually relevant sentences or passages. Then, we tackle
the context generation step as a summarization task where we summarize the
retrieved Wikipedia articles. The sentences achieving the best linear combination
of weighted features are added to the context (in the 500 words limit established
by the organizers). So far, this approach is the same as the one we experimented
last year [
The rest of the paper is organized as follows. Section 2 describes the process we followed to extract candidate sentences, which includes Tweet formatting and document retrieval on Wikipedia. Then, we describe in Section 3 the various sentence-level features that we used. 2
Considering that the task is to provide context from Wikipedia text, one crucial step is to retrieve Wikipedia articles that are relevant to the Tweet. Hopefully, these articles contain sentences that provide enough contextual information to (fully) understand the meaning of the Tweet. 2.1
#HashtagSplitting and Tweet formatting Hashtags in Tweets are very important pieces of information, since they are tags that were generated by the user. Making a parallel with TREC-like topics, we can view the hashtags as the title while the Tweet itself is the description.
However the main problem with hashtags is that they often are composed
of several words concatenated together (e.g. #WhitneyHouston). We used an
algorithm based on Peter Novig's chapter on \Natural Language Corpus Data"
in [
We also removed all the retweet mentions (RT), user mentions (@somebody) and stopwords (based on the standard INQUERY stoplist) from the Tweets. The nal output of this Tweet formatting process is a clean Tweet without stopwords or useless mentions, as well as a very short and user-generated representation of this Tweet. 2.2
Retrieving relevant Wikipedia articles is the rst crucial part for nding
contextually relevant sentences. For this purpose we use the well-known Markov
Random Field model [
Given an initial Tweet T , the output of the method described in the previous section is a set of hashtags HT and a set of terms QT . We then score a Wikipedia article D according to the following function: s(HT ; QT ; D) = scoreMRF (HT ; D) + (1 ) scoreMRF (QT ; D) where is the parameter which controls the in uence of the hashtags with respect to the entire Tweet text. We describe in the following section how we set this parameter. We used the Sequential Dependence Model instantiation of MRF, which is de ned as follows: scoreMRF (Q; D) =
T
X fT (q; D) + O + U q2Q jQj 1 X fO(qi; qi+1; D) i=1 jQj 1 X fU (qi; qi+1; D) i=1 where the features weights are set according to the author's recommendation ( T = 0:85, O = 0:1, U = 0:05). fT , fO and fU are the log maximum likelihood estimates of query terms in document D, computed over the target collection with a Dirichlet smoothing ( = 2500). 2.3
The importance of hashtags is also contextual. Since they can sometimes be
noise rather than useful pieces of information, we need an automatic way of
setting a varying for each Tweet. We thus rely on a well-known pre-retrieval
query performance predictor: the clarity score [
P (wjHT ) P (wjC) where V is the vocabulary. The hashtags language model is estimated through pseudo-relevant feedback:
P (wjHT ) = X P (wjD)P (DjHT )
D2R The set R of pseudo-relevant documents is composed of the top 5 ranked Wikipedia articles for a the HT query. Then achieves higher values when documents of R are homogeneous and di erent from the background documents of the collection. This parameter thus allows us to predict if hashtags are discriminative, and to weigh their importance in the query accordingly. 3
From the ranked list of Wikipedia articles, we only consider the top 5 articles as relevant. The underlying assumption is that a Tweet may discuss only a very limited amount of topics, due to the 140 characters limit. Since encyclopedic topics are very well delimited between Wikipedia articles, we think that 5 articles is a reasonable number allowing us to avoid topic drift while hopefully providing a comprehensive coverage of the Tweet's topics. After selecting the 5 best ranked Wikipedia articles with respect to a Tweet T , the next step is sentence segmentation. Each article is divided into sentences using the nltk1 toolkit. We describe in this section the various scoring methods we use to estimate their importance with respect to the Tweet's context. 3.1
We computed several features for each candidate sentence in order to further rank them and produce the Tweet's context. There are four categories of features: { centrality of the sentence within the Wikipedia article from which the sentence is extracted, { relevance of the sentence with respect to the Tweet (also including hashtags), { relevance of the sentence with respect to an URL embedded in the Tweet, { relevance of the Wikipedia article from which the sentence is extracted.
All the computed features use cleansed versions of sentences and Tweets. We remove stopwords and stem remaining words using the standard Porter stemming algorithm.
Sentence centrality The importance of a sentence within the document where
it appears is estimated using the TextRank [
and the cosine similarity between the candidate sentence and the entire Tweet, and also between the candidate sentence and the hashtags alone.
to external web pages which generally contain a lot of contextual information. Organizers consider these web pages as the \answer" of the question asked by the Tweet. This is why using these web pages (even automatically) is considered as a manual run in the Tweet Contextualization track. Considering it worked very well for us last year, we still computed some features using the text of these web pages. More speci cally, we compute the word overlap and the cosine similarity between the candidate sentence and the entire text of the linked page, as well as with the title of the web page.
Wikipedia article relevance The articles from which candidate sentences are extracted contain di erent contextual information and thus have di erent importance. Then, a sentence belonging to a high ranked document has a higher chance of being relevant. We use as feature the probability of the document from which the candidate sentence has been extracted. 1 http://nltk.org/
of each sentence as a weighted linear combination of the above features. The weights were learned using the 2012 data and are presented in Table 1.
After every sentence has been attributed a score, they are ordered and the top-ranked sentences are selected to form context (within the limit of 500 words). If two sentences are extracted from the same document, we keep their original order to improve readability and coherence. 4
We submitted three di erent runs this year, which we describe in this section. LIA-title-only-notrain This rst run only uses features c2, c3 and c10 without using the trained weights. Sentences of the context are thus ordered using their linear combination of three features.
LIA-all-notrain For this run we use all features described in the previous section without using the trained weights.
LIA-all-train Finally, this runs uses all features combined with the weights from Table 1. 5
O
cial Results
We report in Table 2 the o cial results released by the organizers of the 10 best
performing systems. The evaluation measure computes divergences [
LIA-all-notrain 0.8861 0.881 0.782 LIA-title-only-notrain 0.8943 0.8908 0.7939 275 0.8969 0.8924 0.8061 273 0.8973 0.8921 0.8004 274 0.8974 0.8922 0.8009 LIA-all-train 0.8998 0.8969 0.7987 254 0.9242 0.9229 0.8331 276 0.9301 0.927 0.8169 270 0.9397 0.9365 0.8481 267 0.9468 0.9444 0.8838 achieved the best results of the track. Although our best performing run was tagged as manual, we did not manually intervene at any time in our contextualization process.
In Table 3 are reported the readability results of the top 10 best systems. Although our best informative run does not achieve the best readability results, we see that it is very close to the run 275. It also produces the less redundant contexts overall. We however do not clearly understand why our three runs achieve such di erent readability results since the context generation process is the same. We can for example hypothesize from Table 2 that the contexts which LIA-titleonly-notrain outputs are very similar to those of LIA-all-notrain. Then what could explain such a huge readability di erence between the two (very similar) approaches? We think that these problems are worth further investigation. Run 275 LIA-all-notrain 274 273 LIA-all-train 254 LIA-title-only-notrain 276 267 270
Mean Average Relevancy Non redundancy Soundness Syntax 72.44% 76.64% 67.30% 74.52% 75.50% 72.13% 74.24% 71.98% 70.78% 73.62% 71.71% 74.66% 68.84% 71.78% 74.50% 71.35% 75.52% 67.88% 71.20% 74.96% 69.54% 72.18% 65.48% 70.96% 72.18% 67.46% 73.30% 61.52% 68.94% 71.92% 65.97% 68.36% 64.52% 66.04% 67.34% 49.72% 52.08% 45.84% 51.24% 52.08% 46.72% 50.54% 40.90% 49.56% 49.70% 44.17% 46.84% 41.20% 45.30% 46.00%
We presented in this paper our contributions to the INEX 2013 Tweet Contextualization Track as well as the o cial results released by the organizers. We saw that a simple contextualization system composed of an e ective Wikipedia retrieval system and a multi-document summarizer achieved the best informativeness results of the track. While it did not achieve the best readability results, it was very close to the best system.