According to the organizers, the objective of the 2013 INEX Tweet Contextualization Task is: \...The Tweet Contextualization aims at providing automatically information - a summary that explains the tweet. This requires combining multiple types of processing from information retrieval to multi-document summarization including entity linking." We present the Cortex summarizer applied to the INEX 2013 task. Cortex summarizer uses several sentence selection metrics and an optimal decision module to score sentences from a document source. The results show that Cortex system performs well on INEX task.
Automatic text summarization is indispensable to cope with ever increasing
volumes of valuable information. An abstract is by far the most concrete and
most recognized kind of text condensation [
An open domain Question-Answering system (QA) has to precisely answer a
question expressed in natural language. QA systems are confronted with a ne
and di cult task because they are expected to supply speci c information and
3 CORTEX es Otro Resumidor de TEXtos (CORTEX is anotheR TEXt summarizer).
not whole documents. At present there exists a strong demand for this kind of
text processing systems on the Internet. A QA system comprises, a priori, the
following stages [
Most research e orts in summarization emphasize generic summarization
[11{13]. User query terms are commonly used in information retrieval tasks.
However, there are few papers in literature that propose to employ this approach
in summarization systems [14{16]. In the systems described in [
This paper is organized as follows. In Section 2 we explain the INEX 2013 Tweet Contextualization Track. In Section 3 we explain the methodology of our work. Experimental settings and results obtained with Cortex summarizer are presented in Section 4. Section 6 exposes the conclusions of the paper and the future work. 2
The Initiative for the Evaluation of XML Retrieval (INEX) is an established
evaluation forum for XML information retrieval (IR) [
Like in Question Answering track of INEX 2011 and 2012, the present task is about contextualizing tweets, i.e. answering questions of the form "What is 4 https://inex.mmci.uni-saarland.de/tracks/qa/ this tweet about?" using a recent cleaned dump of the Wikipedia5. As organizers claim, the general process involves three steps: { Tweet analysis. { Passage and/or XML elements retrieval. { Construction of the answer.
Then, a relevant passage segment contains: { Relevant information but { As few non-relevant information as possible (the result is speci c to the question). 2.1
The corpus has been constructed from a dump of the English Wikipedia from November 2012. All notes and bibliographic references were removed to facilite the extraction of plain text answers. (Notes and bibliographic references are di cult to handle). Resulting documents contains a title, an abstract and section. Each section has a sub-title. Abstract end sections are made of paragraphs and each paragraph can have entities that refer to Wikipedia pages. 2.2
598 tweets in English were collected by the organizers from Twitter6 for the Track 2013.. Tweets were selected and checked among informative accounts (for example, @CNN, @TennisTweets, @PeopleMag, @science...), in order to avoid purely personal tweets that could not be contextualized. Information such as the user name, tags or URLs will be provided. 3
Cortex [
The INEX 2013 Tweet Contextualization Track evaluation is a real-world complex question (called long query) answering, in which the answer is a summary constructed from a set of relevant documents. The documents are parsed to create a corpus composed of the query and the the multi-document retrieved 5 See the o cial INEX 2013 Tweet Contextualization Track Website: https://inex.
mmci.uni-saarland.de/tracks/qa/. 6 www.tweeter.com by a Perl program supplied by INEX organizers7. This program is coupled to Indri system8 to obtain for each query, 50 documents from the whole corpus.
The idea is to represent the text in an appropriate vectorial space and
apply numeric treatments to it. In order to reduce complexity, a preprocessing is
performed to the question and the document: words are ltered, lemmatized
and stemmed. The Cortex system uses 11 metrics (see [
By example, the topic-sentence overlap measure assigns a higher ranking for the sentences containing question words and makes selected sentences more relevant. The overlap is de ned as the normalized cardinality of the intersection between the query word set T and the sentence word set S.
Overlap(T; S) = card(S \ T ) card(T )
The system scores each sentence with a decision algorithm that relies on the normalized metrics. Before combining the votes of the metrics, these are partitioned into two sets: one set contains every metric i > 0:5, while the other set contains every metric i < 0:5 (values equal to 0.5 are ignored). We then calculate two values and , which give the sum of distances (positive for and negative for ) to the threshold 0.5 (the number of metrics is , which is 11 in our experiment): (1) (2) (3) (4) = X( i
i=1 = X(0:5 i=1 0:5);
i > 0:5 i); i < 0:5 The value given to each sentence s given a query q is calculated with: if( > ) then Score(s; q) = 0:5 + else Score(s; q) = 0:5
The Cortex system is applied to each document of a topic and the summary is generated by concatenating higher score sentences. 7 See: http://qa.termwatch.es/data/getINEX2011corpus.pl.gz 8 Indri is a search engine from the Lemur project, a cooperative work between the University of Massachusetts and Carnegie Mellon University in order to build language modelling information retrieval tools. See: http://www.lemurproject.org/indri/
In Automatic Text Summarization, preprocessing is an important phase to
reduce the space of textual representation. Classically, stemming and
lemmatization have been widely used for normalizing words. However, even using
normalization on large texts, the curse of dimensionality can disturb the performance
of summarizers. The main idea of ultra stemming[
In this study, we used the document sets made available during the Initiative for the Evaluation of XML retrieval (INEX)9, in particular on the INEX 2012 Tweet Contextualization Track.
The strategy of Cortex system to deal multi-document summary problem is quite simple: rst, a long single document D is formed by concatenation of all i = 1; :::; n relevant documents provided by Indri engine: d1; d2; :::dn. The rst line of this multi-document D is the tweet T . Cortex summarizer extracts of D the most relevant sentences following T . Then, this subset of sentences is sorted by the date of documents di. The summarizer adds sentences into the summary until the word limit is reached. To evaluate the performance of Cortex system on INEX tweet contextualization track, we used the online package available from INEX website10. 5
9 https://inex.mmci.uni-saarland.de/ 10 http://qa.termwatch.es/data/ In this paper we have presented the Cortex summarization system applied on INEX 2013 Tweet Contextualization Track. The rst one is based on the fusion process of several di erent sentence selection metrics. The decision algorithm obtains good scores on the INEX 2013 Tweet Contextualization Track (the decision process is a good strategy without training corpus). The second one is based on the divergence between summary and the source document.
Cortex summarizer using 4-ultra stemming as normalization has obtained very good results in informativeness evaluations. However, Cortex using stemming normalization outperforms Cortex using lemmatization and 4-ultra stemming. A module of compound words may improves the performance of our summarizer. We show that a simple statistical summarizer show good performances in this complex task.