The aim of the tweet contextualization INEX (Initiative for the Evaluation of XML retrieval) task at CLEF 2013 (Conference and Labs of the Evaluation Forum) is to build a system that provides automatically information related with di erent tweets, that is, a summary that explains a speci c tweet. In this article, our strategy and results are presented. The methodology for the task in English includes three stages. First, automatic reformulations of the initial queries provided for the task, that is, the tweets, are performed. In this research, we use words sequences that agree with the typical terminological patterns, name entities, hashtags and Twitter users accounts, since we consider that they are representative of tweets' topics. Second, related documents are retrieved from Wikipedia with the search engine Indri, using the reformulated queries. Third, the obtained documents are summarized by using two di erent automatic summarization systems, in order to provide the nal summary associated to each query. Regarding the pilot task for Spanish, our strategy includes a rst stage where automatic reformulations of the initial queries provided for the task (similar to English) are carried out. However, it does not include neither the search engine Indri nor the summarization systems REG and Cortex. In this case, we directly extract relevant text passages from Wikipedia pages using the generated queries and we build the summary with the rst sentences of these pages.
long. However, 140 characters long messages are rarely self-content. The aim of the tweet contextualization INEX (Initiative for the Evaluation of XML retrieval) task at CLEF 2013 (Conference and Labs of the Evaluation Forum) is to build a system that provides automatically information related with di erent tweets, that is, a summary that explains a speci c tweet. In order to get this aim, it is necessary to combine di erent types of processing, from information retrieval to multi-document summarization. This INEX task started in 2010, and the results show that best systems combine passage retrieval, sentence segmentation and scoring, named entity recognition and Part of Speech (POS) analysis. The evaluation of the participant systems involves two aspects: informativity and readability. In 2013, the goal of the task and the evaluation metrics has remained unchanged but tweet diversity has been improved. More specially, a signi cant part of tweets with hashtags have been included in the tweet set. Hashtags are authors' annotation on key terms of their tweets. In the two past years, hashtags have been underused, although they are core components of tweets.
Like in 2012, the use case of this task is the following: given a new tweet, the system must provide some context about the subject of the tweet, in order to help the reader to understand it, i.e. answering questions of the form "what is this tweet about?" using a recent cleaned dump of the Wikipedia. The general process involves: tweet analysis, passage and/or XML elements retrieval and construction of the answer. This context should take the form of a readable summary, not exceeding 500 words, composed of passages from a provided Wikipedia (WP) corpus. The summaries should contain relevant information but also contain as little non-relevant information as possible.
Speci cally, this year, 598 tweets in English have been collected by the organizers from Twitter. They were selected among informative accounts (for example, @CNN, @TennisTweets, @PeopleMag, @science...), in order to avoid purely personal tweets that could not be contextualized. Information such as the user name, tags or URLs are provided in JSON format. These tweets are available in a single xml le with three elds: topic, title and txt.
In this edition, an extra set of topics (only tweet texts) has been released in Spanish to try a di erent language and a slightly di erent task. Topics in Spanish are opinionated personal tweets about music bands, cars and politics. They were manually selected from CLEF RepLab 2013 test set among those without external url and with at least 15 words. Contextualization should help the reader to also understand the opinion polarity, allusions and humor.
We use two di erent strategies for the task in English and Spanish. Our
strategy for English includes three main stages. First, we perform automatic
reformulations of the initial queries provided for the task, that is the title of the
tweeets, using terminological patterns, name entities, hashtags and Twitter users
accounts. Second, we obtain related documents from WP with the search engine
Indri [
This strategy is similar to the one used at QA@INEX track 2010 [
Regarding the pilot task for Spanish, our strategy includes a rst stage where automatic reformulations of the initial queries provided for the task (similar to English) are carried out. However, it does not include neither the search engine Indri nor the summarization systems REG and CORTEX. In this case, we directly extract relevant text passages from WP pages using the generated queries and we build the summary with the rst sentences of these pages.
In Section 2, the state of the art and the resources used are shown. In Section 3, the methodology is explained. In Section 4, the experimental settings and results are presented. Finally, in Section 5, the conclusions and the future work are exposed. 2 2.1
A term can be de ned as a lexical unit that has a speci c sense in a
thematically restricted domain [
Named Entity extraction may be basically de ned as the task to identify
names referring to persons, organizations and locations in free text; later this
task has been expanded to obtain other entities like dates and numeric
expressions. The recent interest in emerging areas like bioinformatics allows to expand
this recognition task to proteins, drugs and chemical names. Name Entity
extractors are useful for a number of NLP tasks as question answering, textual
entailment and coreference resolution, among others. While early studies were
mostly based on handcrafted rules, most recent ones use supervised machine
learning as a way to automatically induce rule-based systems or sequence
labeling algorithms starting from a collection of training examples. Often, corpus
processing tools include some text handling facilities to perform simple Name
Entity detection. Some of them are based in language speci c peculiarities such
as initial upper case letters together with some heuristics for name entities placed
at the beginning of the sentence. This is the case of the tool used for this
experiment [
On the one hand, hashtags are sequences of characters and numbers pre xed
by a '#'. Social networks largely use this facility to indicate relevant words,
groups or topics. From December 2008 to October 2009 hashtags became widely
adopted on the site of Twitter. As [
\A hashtag is the speci c name for a tag in Twitter. Hashtags derive their name from the fact that they are preceded by the symbol # also known as a hash mark, e.g., #nowplaying."
Hashtags can be considered as one of the main topics of the tweet. There are
several studies on this new type of tagging, which highlight the purpose of these
elements. For example, [
On the other hand, Twitter is a relevant social network used, as mentioned in the introduction, for INEX competition. In this case, users' accounts are designated by a user name pre xed by '@'. Therefore, this string may be considered as a name entity that could be considered the author of the tweet, or another user related with the tweet.
Both types of information, hashtags and users accounts, will be used in our experiments to improve the queries used to retrieve documents from Wikipedia using Indri and, therefore, to improve the nal summary. 2.3
REG [
Cortex [
In this section, the two di erent methodologies used for the task in English and the task in Spanish are explained. 3.1
The methodology for the task in English includes three stages:
1) Automatic reformulations of the initial queries provided for the task, that is the title of the tweeets, are performed. In this research, we consider that word sequences that agree with the typical terminological patterns, name entities, hashtags and Twitter users accounts are representative elements of tweet topics. Therefore, in order to reformulate and build the query we extract automatically two types of information from the tweets:
a) Terms and name entities. First, we POS tag the tweets le, obtaining
proper nouns info, which we consider as name entities. Second, we carry out
terminological pattern extraction, by using an already existent module of the
YATE term extraction tool [
b) Hashtags and users accounts. Usually, hashtags and Twitter accounts are not fully arbitrary strings but very often the actual entity may be discovered just analyzing the string.
First, we extract elements tagged with hashtags (#) and users accounts (@) in each tweet. Second, we check if these elements are included in WP. If they are included, we maintain them for the query string for Indri. If they are not included, the system segments the letters of hashtags and the user accounts, in order to nd all the possible combinations of real words (with a maximum of two words). We apply this strategy taking into account that many times hashtags and user accounts include no spaces, so two or more di erent words can be joined. For our purpose, we need to have the words separated, in order to check if they are included in Wikipedia. For example: { The user account @BarbraStreisand is not present as such in Wikipedia.
Nevertheless, Wikipedia includes \Barbra Streisand", that will be included in our query. { The hashtag #EUbudget is not present as a single unit in Wikipedia. However, Wikipedia includes both \EU" and \budget", therefore these elements separately will be included in our query. { The hashtag #NASA Astronauts is not included as a single unit in Wikipedia.
Although, Wikipedia contains \NASA" and \Astronauts" separately. We include both elements in our query.
If the system does not nd any real word after applying all possible segmentations in hashtags and users accounts, we do not include these elements in the query string for Indri.
In order to enrich the queries, we use a local installation of a Wikipedia dump1 to expand the terms with redirection information from such Wikipedia info. Some care has been taken to keep track of multiword sequences as indicated by the Indri query language speci cation (see examples below). For example, for the tweet:
Behind the scenes on Charlie Brooker's Black Mirror nale we obtain the following query: scenes,finale,#1(Black Mirror),#1(Charlie Brooker)
2) Related documents (speci cally 50 texts) are retrieved from WP with the search engine Indri, using the reformulated queries. The resulting set of WP pages has been splitted in several documents. Each document contains the pages relevant to the query.
3) The obtained documents are summarized by using REG and CORTEX summarization systems, in order to provide the nal summary associated to each query.
Figure 1 illustrates the methodology for the task in English. 3.2
Our strategy for Spanish includes neither the search engine Indri nor the summarization systems REG and Cortex. This strategy includes two main stages: 1) To carry out automatic reformulations of the original queries provided for the task. First, name entities, hashtags and Twitter users accounts are extracted from the tweet. Second, the system checks if they have a main page in WP: if they have a page in WP, they are included in the nal query; if they do not have a page in WP, they are eliminated. Third, if after this process, the query includes 1 or 0 elements, terminological patterns are used to extract more units from the tweet, in order to include in the query those which have a main entry in WP. For example, for the tweet:
Vaya asco de arbitros hay en la liga BBVA, ya solo por los arbitros que hay, nadie tendr a que decir que es la mejor liga del mundo...
we obtain the following query:
1 This resource has been otained using [
Some lters are applied before obtaining the nal version of the summary: a) ambiguous WP pages are eliminated, b) passages in brackets are erased, c) only passages included at the beginning of the WP page are selected (text appearing after the rst subtitle of the WP page is not considered), and d) lists of elements are eliminated.
Figure 1 illustrates the methodology for the task in Spanish. 4
In this section, the experiments and results for the task in English and the task in Spanish are explained. 4.1
For the task in English, we have applied our strategy using the 598 tweets that have been collected by the organizers from Twitter. From these tweets, 529 included di erent hashtags and/or users accounts. From this set of elements, 391 were found in Wikipedia.
The evaluation of all the participant systems in the INEX 2011 QA Track
involves two aspects: informativity and readability. On the one hand, to
evaluate informativity, the automatic FRESA [FRamework for Evaluating
Summaries Automatically] [
Informativity has been evaluated based on three overlapping references: { prior set of relevant pages selected by organizers while building the 2013 topics (40 tweets, 380 passages, 11 523 tokens), { pool selection of most relevant passages from participant submissions for tweets selected by organizers (45 tweets, 1 760 passages, 58 035 tokens), { all relevant text merged together with an extra selection of relevant passages from a random pool of ten tweets (70 tweets, 2 378 passages, 77 043 tokens) Table 1 include the results about informativity. In this table, we have only included the best run of each di erent team. The complete table can be retrieved in the o cial website of INEX 2013 (https://inex.mmci.uni-saarland.de/).
Informativity results show that the impact of the summarization system in our strategy is not high. The results using Cortex (run 262) are slightly better than the results using REG (run 255).
On the other hand, readability has been evaluated by organizers over the ten tweets having the largest text references (t-rels). For these tweets, summaries are expected to have almost 500 words since the reference is much larger. For each participant summary, organizers have checked the number of words over 500 in passages that are: { Relevant (T) i.e. clearly related to the tweet. { Sound (A) i.e. no issues about resolving references to earlier or later items in the discourse. { Non redundant (R) with previous passages. { Syntactically (S) correct.
Non relevant passages have also been considered non sound, redundant and syntactically incorrect. Runs are ranked according to mean average scores per summary over Soundness, Non redundancy and Syntactically correctness among Relevant passages. Tabla 2 shows the results about readability.
Readability results show that the impact of the summarization system in our strategy is higher than in the case of informativity results: the results using REG (run 255) obtain a higher score than the results using Cortex (run 262). Anyway, this di erence is not signi cant, since none of the systems appears in the best positions.
Since the process is fully automatic, we nd some limitations. The main limitation that we have found is that Indri does not retrieve relevant documents. For example, one of the queries includes the name entity "Jennifer Hudson" and only one of the 50 documents retrieved by Indri includes this name entity. This fact limits our results: if the input given to the summarization systems is not relevant, the summary will be not relevant either.
Another limitation of our strategy is the kind of summarization systems that we have used. Both REG and Cortex are mono-document summarization systems. We o er to these systems a single text including the 50 documents retrieved by Indri, and they produce an extractive summary. This fact could produce a lack of correference and coherence, which can a ect the nal results. 4.2
For the task in Spanish, we have applied our strategy using the 354 tweets that have been collected by the organizers from Twitter. Nowadays, we do not have the nal o cial results evaluation in the INEX 2013 Tweet Contextualization Track. We will include the results in a more complete version of this paper. In this paper, our strategy for the INEX 2013 Tweet Contextualization Track are presented, both for the task in English and in Spanish. In our work, as explained, a reformulation of the initial queries (tweets) is carried out by using terminological patterns, named entities, hashtags and Twitter users accounts. For the task in English, the automatic summarization systems REG and Cortex are used to summarize the 50 documents obtained by Indri using these queries. We have proved that the use of di erent summarizers has not a signi cant impact over the results. Also, we have detected that the main problem is related with the documents retrieved by Indri. Therefore, for the pilot task for Spanish, we have not used Indri. We hope to improve the results in this way.
As usual, ambiguous terms represent a serious problem in a procedure like the proposed in this experiment. In this work, ambiguous terms (that is, terms linked to a WP disambiguation page) have not been considered, in order to prioritize precision. In the future, we would like to experiment with systems such as those proposed by [16] or [17] in order to perform some disambiguation and therefore improve the selection of the relevant elements for the queries. Other improvement could be achieved by clustering the WP pages provided by Indri with the aim to obtain a better WP pages selection and then summarize them.
This work has been partially nanced by the Spanish projects RICOTERM 4 (FFI2010-21365-C03-01) and APLE 2 (FFI2012-37260), and a Juan de la Cierva grant (JCI-2011-09665). 15. Lin, C.-Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries.
Proceedings of Text Summarization Branches Out: ACL-04 Workshop. 74-81. 16. Mihalcea, R. (2007). Wikify! Linking Documents to Encyclopedic Knowledge. Proceedings of the sixteenth ACM conference on Conference on information and knowledge management: CIKM '07 Pages 233-242. New York. 17. Ferragina, P.; Scaiella U. (2010). TAGME: On-the- y Annotation of Short Text Fragments (byWikipedia Entities). Proceedings of the 19th ACM international conference on Information and knowledge management: CIKM '10 Pages 1625-1628. New York.