The use case of the Tweet Contextualization task is the following: given a new tweet, participating systems must provide some context about the subject of a tweet, in order to help the reader to understand it. In this task, contextualizing tweets consists in answering questions of the form \what is this tweet about?" which can be answered by several sentences or by an aggregation of texts from di erent documents of the Wikipedia. Thus, tweet analysis, XML/passage retrieval and automatic summarization are combined in order to get closer to real information needs. This article describes the data sets and topics, the metrics used for the evaluation of the systems submissions, as well as the results that they obtained.
The Tweet Contextualization task to be performed by the participating groups of INEX 2012 is contextualizing tweets, i.e. answering questions of the form \what is this tweet about?" using a recent cleaned dump of the Wikipedia. The general process involves: { Tweet analysis, { Passage and/or XML element retrieval, { Construction of the context/summary.
We regard as relevant passages those that both contain relevant information but also contain as little non-relevant information as possible.
For evaluation purposes, we require that a summary uses only elements or passages previously extracted from the document collection. The correctness of summaries is established exclusively based on the support passages and documents. The summaries are evaluated according to: { Informativeness: the way they overlap with relevant passages, { Readability, assessed by evaluators and participants.
The paper is organized as follows. Section 2 details the collection of tweets and documents. Section 3 presents the metrics and tools used for evaluation, as well as results obtained by the participants. Finally, section 4 draws some preliminary conclusions. 2
2.1
Tweets Organizers provided a document collection extracted form Wikipedia, as well as 1000 topics made of tweets from several di erent accounts.
About 1000 tweets in English were collected by the track organizers from Twitter R Search API. They were selected among informative accounts (for example, @CNN, @TennisTweets, @PeopleMag, @science...), in order to avoid purely personal tweets that could not be contextualized. Information such as the user name, tags or URLs have been provided. These tweets were available in two formats: { a two-column text format with only tweet id and tweet text. For example: 169927058904985600 "Tensions are at an all-time high as the @AmericanIdol Hollywood Round continues, Tonight at 8/7c. #Idol" 63 of these tweets were selected manually by organizers. For each of them, we checked that the document collection contained some information related to the topic of the tweet. This means that all 63 tweets had some contextualization material inside the provided collection.
From the accounts used for extraction of these 63 messages, a number of other tweets were automatically selected, bringing to 1000 the total number of tweets to be contextualized by the participants. This is done to ensure that only fully automatic and robust enough systems could accomplish the task.
However, only the 63 tweets that had been manually collected and checked have been used for informativeness evaluation; only 18 of them have been used for readability evaluation (due to the complexity of this evaluation). 2.2
Document collection The document collection has been built based on a recent dump of the English Wikipedia from November 2011. Since we target a plain XML corpus for an easy extraction of plain text answers, we removed all notes and bibliographic references that are di cult to handle and kept only non empty Wikipedia pages (pages having at least one section).
Resulting documents are made of a title (title), an abstract (a) and sections (s). Each section has a sub-title (h). Abstract and sections are made of paragraphs (p) and each paragraph can have entities (t) that refer to other Wikipedia pages. Therefore the resulting corpus has this simple DTD: <!ELEMENT xml (page)+> <!ELEMENT page (ID, title, a, s*)> <!ELEMENT ID (#PCDATA)> <!ELEMENT title (#PCDATA)><!ELEMENT a (p+)> <!ELEMENT s (h, p+)> <!ATTLIST s o CDATA #REQUIRED> <!ELEMENT h (#PCDATA)> <!ELEMENT p (#PCDATA | t)*> <!ATTLIST p o CDATA #REQUIRED> <!ELEMENT t (#PCDATA)> <!ATTLIST t e CDATA #IMPLIED>
<?xml version="1.0" encoding="utf-8"?> <page> <ID>2001246</ID> <title>Alvin Langdon Coburn</title> <s o="1"> <h>Childhood (1882-1899)</h> <p o="1">Coburn was born on June 11, 1882, at 134 East Springfield Street in <t>Boston, Massachusetts</t>, to a middle-class family. His father, who had established the successful firm of Coburn & Whitman Shirts, died when he was seven. [...] </p> <p o="2">In 1890 the family visited his maternal uncles in Los Angeles, and they gave him a 4 x 5 Kodak camera. He immediately fell in love with the camera, and within a few years he had developed a remarkable talent for both visual composition and technical proficiency in the <t>darkroom</t>. (...)</p> (...) </page> 2.3
Submission format Participants could submit up to 3 runs. One run out of the 3 had to be completely automatic: participants had to use only the Wikipedia dump and possibly their own resources (even if the texts of tweets sometimes contain URLs, the Web must not be used as a resource). That is, a participant could not submit more than 3 runs in total.
A submitted summary has the following format: <tid> Q0 <file> <rank> <rsv> <run_id> <text of passage 1> <tid> Q0 <file> <rank> <rsv> <run_id> <text of passage 2> <tid> Q0 <file> <rank> <rsv> <run_id> <text of passage 3> ...
where: { The rst column tid is the topic number. { The second column is currently unused and should always be Q0. It is just a formating requirement used by the evaluation programs to distinguish between o cial submitted runs and q-rels. { The third column file is the le name (without .xml) from which a result is retrieved, which is identical to the <id> of the Wikipedia document. It is only used to retrieve the raw text content of the passage, not to compute document retrieval capabilities. In particular, if two results only di er by their document id (because the text is repeated in both), then they will be considered as identical and thus redundant. { The fourth column rank indicates the order in which passages should be read for readability evaluation, this di ers from the expected informativeness of the passage which is indicated by the score rsv in the fth column. Therefore, these two columns are not necessarily correlated. Passages with highest scores in the fth column can be scattered at any rank in the result list for each topic. { The sixth column run id is called the \run tag" and should be a unique identi er for the participant group and for the method used. { The remaining column gives the result passage in raw text without XML tags and without formatting characters. The only requirement is that the resulting word sequence appears at least once in the le indicated in the third eld.
167999582578552 Q0 3005204 1 0.9999 I10UniXRun1 The Alfred Noble Prize is an award presented by the combined engineering societies of the United States, given each year to a person not over thirty-five for a paper published in one of the journals of the participating societies. 167999582578552 Q0 3005204 2 0.9998 I10UniXRun1 The prize was established in 1929 in honor of Alfred Noble, Past President of the American Society of Civil Engineers. 167999582578552 Q0 3005204 3 0.9997 I10UniXRun1 It has no connection to the Nobel Prize, although the two are often confused due to their similar spellings. 3
In this task, readability of answers [9] is as important as the informative
content. Summaries must be easy to read as well as relevant. Following INEX 2011
Question-Answering task [
This section describes the metrics and tools used to perform the evaluation and gives results obtained by participating systems. 3.1
Baseline System
A baseline XML-element retrieval/summarization system has been made
available for participants. This baseline is the same as 2011 QA@INEX task, and has
been described in [
This year only three teams used the provided perl API and Indri index of the collection.
The total number of submitted passages is 671,191 (31 596 328 tokens). The median number of distinct passages per tweet is 79.5 and the average is 146.5. Only passages starting and ending by the same 25 characters have been considered as duplicated, therefore short sub-passages could appear twice in longer ones. 3.3 Informativeness Evaluation Informativeness evaluation has been performed by organizers on a pool of 63 tweets. For each tweet, we took the 60 best passages based on the rsv score in the th column of the runs from all participants. After removing duplicates per tweet, 16,754 passages were evaluated by organizers. The median number of passages per tweet is 273 and the average is 265.9. Passages have been merged and displayed to the assessor in alphabetical order. Therefore, each passage informativeness has been evaluated independently from others, even in the same summary. The structure and readability of the summary was not assessed in this speci c part, and assessors only had to provide a binary judgement on whether the passage was worth appearing in a summary on the topic, or not. 2,801 passages among 16,754 have been judged as relevant, with a median of 50 passages per tweet and an average of 55.1. The average length of a passage is 30.03 tokens.
Metrics Systems had to make a selection of the most relevant information, the
maximal length of the abstract being xed. Therefore focused IR systems could
just return their top ranked passages meanwhile automatic summarization
systems need to be combined with a document IR engine. In this task, readability
of answers [
In previous 2010 and 2011 INEX Question Answering tracks, evaluations have been carry out using FRESA package which includes a special lemmatizer. In 2011 we provided the participants with a standalone evaluation toolkit based on Porter stemmer and implementing a new normalized ad-hoc dissimilarity de ned as following:
Dis(T; S) =
P = Q =
X fT (t) t2T fT (t)
fT fS (t) fS fT where T is the set of terms in the reference and for every t 2 T , fT (t) is its frequency in the reference and fS (t) its frequency in the summary.
The idea was to have a dissimilarity which complement has similar properties to usual IR Interpolate Precision measures. Actually, 1 Dis(T; S) increases with the Interpolated Precision at 500 tokens where Precision is de ned as the number of word n-grams in the reference. The introduction of the log is necessary to deal with highly frequent words.
As previously announced, we used this software to evaluate informativeness and like in INEX QA tracks, we considered as T three di erent sets based on Porter stemming: { Unigrams made of single lemmas (after removing stop-words). { Bigrams made of pairs of consecutive lemmas (in the same sentence). { Bigrams with 2-gaps also made of pairs of consecutive lemmas but allowing the insertion between them of a maximum of two lemmas.
Bigrams with 2-gaps appeared to be the most robust metric. Sentences are not considered as simple bags of words and the measure is less sensitive to sentence segmentation than simple bi-grams. This is why bigrams with 2-gaps is our o cial ranking metric for informativeness.
Bigrams with 2-gaps appeared to be the most robust metric in previous INEX QA tracks, however in this edition where topics are real tweets, measures based on bigrams with or without 2-gaps are strongly correlated. Meanwhile the measure based on simple uni-grams is also stable but gives a di erent ranking. This will be discussed during the CLEF workshop.
Results Results are presented in Table 1. The 3 top ranked runs improved the baseline. Runs with (*) have been submitted as \manual".
Dissimilarity values are very closed, however di erences are often statistically signi cant as shown in table 2. 3.4
Readability evaluation Human assessment Each participant had to evaluate readability for a pool of summaries of a maximum of 500 words each on an online web interface. Each summary consisted in a set of passages and for each passage, assessors had to tick four kinds of check boxes. The guideline was the following: { Syntax (S): tick the box if the passage contains a syntactic problem (bad segmentation for example), { Anaphora (A): tick the box if the passage contains an unsolved anaphora, { Redundancy (R): tick the box if the passage contains a redundant information, i.e. an information that has already been given in a previous passage, { Trash (T): tick the box if the passage does not make any sense in its context (i.e. after reading the previous passages). These passages must then be considered at trashed, and readability of following passages must be assessed as if these passages were not present. { If the summary is so bad that you stop reading the text before the end, tick all trash boxes until the last passage.
For each summary, the text without tags of the tweet was displayed, thus this year readability was evaluated in the context of the tweet, and passages not related to the tweet could be considered as trash even if there were readable. Metrics and results To evaluate summary readability, we consider the number of words (up to 500) in valid passages. We used three metrics based on this: { Relevancy or Relaxed metric: a passage is considered as valid if the T box has not been ticked, { Syntax: a passage is considered as valid if the T or S boxes have not been ticked, { Structure or Strict metric: a passage is considered as valid if no box has been ticked.
In all cases, participant runs are ranked according to the average, normalized number of words in valid passages.
A total of 594 summaries from 18 tweets have been assessed. The resulting 18 tweets are included in those used for informativeness assessment. Results are presented in Table 3. The last column gives the number of evaluated summaries for correponding run. Only runs that were evaluated on more that 6 summaries, are ranked following the relaxed metric. Missing evaluations were due to formatting problems, too long passages (more than 500 tokens) or missing summaries in the submitted runs. 4
In 2011 we experimented using the wikipedia to contextualize twitted New York times paper titles. There was a large overlapping between the two vocabularies. This year we selected a larger pool of public factual tweets with a much more diversi ed vocabulary. The robust baseline we provided was di cult to outperform on the average. This needs further analysis and will be discussed during the workshop. One reason could be that the baseline approach removes all nonnominals from tweet texts, keeping only nouns and adjectives and this can help in wikipedia search. However, for speci c tweets, to retrieve relevant information from the wikipedia, it was necessary to expand the tweet vocabulary or to use tags inside the tweet. e n i l
Rank Run Relevancy Syntax Structure Nb 1 185 0.7728 0.7452 0.6446 17 2 171 0.6310 0.6060 0.6076 10 3 168 0.6927 0.6723 0.5721 15 4 Baseline 0.6975 0.6342 0.5703 13 5 186 0.7008 0.6676 0.5636 18 6 170* 0.6760 0.6529 0.5611 16 7 165 0.5936 0.6049 0.5442 10 8 152 0.5966 0.5793 0.5433 16 9 155 0.6968 0.6161 0.5315 16 10 178 0.6336 0.6087 0.5289 17 11 169 0.5369 0.5208 0.5181 16 12 193 0.6208 0.6115 0.5145 13 13 163 0.5597 0.5550 0.4983 12 14 187 0.6093 0.5252 0.4847 18 15 154 0.5352 0.5305 0.4748 13 16 196b 0.4964 0.4705 0.4204 16 17 153 0.4984 0.4576 0.3784 14 18 164* 0.4759 0.4317 0.3772 15 19 162 0.4582 0.4335 0.3726 17 20 197 0.5487 0.4264 0.3477 15 21 196c 0.4490 0.4203 0.3441 16 22 196a 0.4911 0.3813 0.3134 15 23 176 0.2832 0.2623 0.2388 13 24 156 0.2933 0.2716 0.2278 9 25 188 0.1542 0.1542 0.1502 11 26 157 0.1017 0.1045 0.1045 13 27 161 0.0867 0.0723 0.0584 14 - 151 0.8728 0.8728 0.8720 5 - 150 0.8493 0.8493 0.7270 3 - 192 0.6020 0.6020 0.6020 2 - 191 0.6173 0.5540 0.5353 3 - 177 0.5227 0.4680 0.4680 3 - 149 0.1880 0.0900 0.0900 4 Table 3. Readability results with the relaxed and strict metric.