According to the organizers, the objective of the 2014 CLEFINEX Tweet Contextualization Task is: \...The Tweet Contextualization aims at providing automatically information - a summary that explains the tweet. This requires combining multiple types of processing from information retrieval to multi-document summarization including entity linking." We present three statistical summarizer systems applied to the CLEF-INEX 2014 task. Cortex summarizer uses several sentence selection metrics and an optimal decision module to score sentences from a document source. Artex summarizer uses a simple inner product among the topic-vector and the pseudo-word vector. Reg summarizer is a performant graph-based summarizer. The results show that our systems performed well on CLEF-INEX task. Our three systems have obtained the rst rank in the INEX manual evaluation.
Automatic text summarization is indispensable to cope with ever increasing
volumes of valuable information. An abstract is by far the most concrete and most
recognized kind of text condensation [
An open domain Question-Answering system (QA) has to exactly answer a
question expressed in natural language. QA systems are confronted with a ne
and di cult task because they are expected to supply speci c information and
not whole documents. Currently there exists a strong demand for this kind of
text processing systems on the Internet. A QA system comprises, a priori, the
following stages [
Most research e orts in summarization emphasize generic summarization
[11{13]. User query terms are commonly used in information retrieval tasks.
However, there are few papers in literature that propose to employ this approach
in summarization systems [14{16]. In the systems described in [
This paper is organized as follows. In Section 2 we explain the CLEF-INEX 2014 Tweet Contextualization Track. In Section 3.1 we explain the methodology of our work. Experimental settings and results obtained with Cortex summarizer are presented in Section 5. Section 7 exposes the conclusions of the paper and the future work. 2
The Initiative for the Evaluation of XML Retrieval (INEX) is an established
evaluation forum for XML information retrieval (IR) [
Like in iNEX Question Answering track 2011, 2012 and 2013, the present task is about contextualizing tweets, i.e. answering questions of the form "What is this tweet about?" using a recent cleaned dump of the Wikipedia5. As organizers claim, the general process involves three steps: { Tweet analysis. { Passage and/or XML elements retrieval. { Construction of the answer.
Then, a relevant passage segment contains: { Relevant information but { As few non-relevant information as possible (the result is speci c to the question). 2.1
The corpus has been rebuilt in 2013 from a dump of the English Wikipedia from November 2012. All notes and bibliographic references were removed to facilitate the extraction of plain text answers. (Notes and bibliographic references are di cult to handle). Organizers kept only non empty Wikipedia pages (pages having at least one section). 2.2
For the Track 2014, a set of 240 tweets in English have been selected by the organizers from CLEF RepLab 2013 together with their related entity. The tweets have >= 80 characters and do not contain urls in order to focus on content analysis.
In the CLEF-INEXorganizers words: \RepLab provides several annotations for tweets, we selected three types of them: the category (4 distinct), an entity name from the wikipedia (61 distinct) and a manual topic label (235 distinct). The entity name should be used as an entry point into wikipedia or DbPedia and gives the contextual perspective. The usefulness of topic labels for this automatic task is an open question at this moment because of their variety". 3 3.1
Cortex [
The INEX 2014 Tweet Contextualization Track evaluation is a real-world complex question (called long query) answering, in which the answer is a summary constructed from a set of relevant documents. The documents are parsed to create a corpus composed of the query and the the multi-document retrieved by a Perl program supplied by INEX organizers6. This program is coupled to Indri system7 to obtain for each query, 50 documents from the whole corpus.
The idea is to represent the text in an appropriate vectorial space and
apply numeric treatments to it. In order to reduce complexity, a preprocessing is
performed to the question and the document: words are ltered, lemmatized
and stemmed. The Cortex system uses 11 metrics (see [
By example, the topic-sentence overlap measure assigns a higher ranking for the sentences containing question words and makes selected sentences more relevant. The overlap is de ned as the normalized cardinality of the intersection between the query word set T and the sentence word set S.
Overlap(T; S) = card(S \ T ) card(T )
The system scores each sentence with a decision algorithm that relies on the normalized metrics. Before combining the votes of the metrics, these have been split into two sets: one set contains every metric i > 0:5, while the other set contains every metric i < 0:5 (values equal to 0.5 are ignored). We then compute two values and , which give the sum of distances (positive for and negative for ) to the threshold 0.5 (the number of metrics is , which is 11 in our experiment): = X( i
i=1 = X(0:5 i=1 0:5);
i > 0:5 i); i < 0:5 The value given to each sentence s given a query q is calculated with: if( > ) then Score(s; q) = 0:5 + else Score(s; q) = 0:5 (1) (2) (3) (4)
The Cortex system is applied to each document of a topic and the summary is generated by concatenating higher score sentences. 6 See: http://qa.termwatch.es/data/getINEX2011corpus.pl.gz 7 Indri is a search engine from the Lemur project, a cooperative work between the University of Massachusetts and Carnegie Mellon University in order to build language modelling information retrieval tools. See: http://www.lemurproject.org/indri/
Artex8 computes the score of each sentence by calculating the inner product between a sentence vector, an average pseudo-sentence vector (the \global topic") and an average pseudo-word vector (the \lexical weight"). The summary is generated concatenating the sentences with the highest scores.
An average document vector represents the \global topic" of all sentences
vectors is constructed. The \lexical weight" for each sentence, i.e. the number
of words in the sentence, is obtained. A score for each sentence is calculated
using their proximity with the \global topic" and their \lexical weight". Let
s = (s ;1; s ;2; : : : ; s ;N ) be a vector of the sentence = 1; 2; : : : ; . The average
pseudo-word vector a = [a ], was de ned as the average number of occurrences
of N words used in the sentence s :
and the average pseudo-sentence vector b = [bj ] as the average number of
occurrences of each word j used through the sentences:
a =
We create a graph G = (V; A) where S vertices represent sentences and A the set
of edges. An edge between two vertices is created if the corresponding sentences
have at least one word in common. An adjacency matrix is constructed from
the matrix S[P=sentences N=words] as follows: If the element Si;k = 1 of S matrix
(in the phrase i the word k is present), we check the k column. If the element
Sj;k = 1 we put 1 in ai;j of the adjacency matrix A, which means that i and j
sentences share the word k. To extract the heaviest sentence, a variant of tree
problem maximum weight has been proposed. The weights are on the vertices;
not on the edges. We have built an algorithm inspired on the Kruskal's algorithm
[
The solution is based on a calculation greedy search paths. The algorithm Reg performs the following steps: 1. Select the vertex heavy v0, and put it in T . It will be called root. The root is chosen whose degree is >= 2. 2. Add to T the heavy neighbor of v0. It will choose among those who are not part of T . 3. Repeat 2 until k have the required vertices. 4. Return the path T . 5
In this study, we used the document sets made available during the Initiative for the Evaluation of XML retrieval (INEX)9, in particular on the INEX 2012 Tweet Contextualization Track.
The strategy of our three summarizer systems to deal multi-document summary problem is quite simple: rst, a long single document D is formed by concatenation of all i = 1; :::; n relevant documents provided by Indri engine: d1; d2; :::dn. The rst line of this multi-document D is the tweet T . Each summarizer extracts of D the most relevant sentences following T . Then, this subset of sentences is sorted by the date of documents di. The summarizer adds sentences into the summary until the word limit is reached. To evaluate the performance of eaxh system on INEX tweet contextualization track, we used the online package available from CLEF-INEX website10. 6
{ Readable: % of passages considered as readable (Non trash) { Syntax % of passages without syntax or grammatical errors { Diversity % of non redundant passages { Structure % of non breaking anaphora passages 7
In this paper we have presented three statistical summarizer systems applied on CLEF-INEX 2014 Tweet Contextualization Track. The rst one, Cortex is based on the fusion process of several di erent sentence selection metrics. The decision algorithm obtains good scores on the INEX 2014 Tweet Contextualization Track (the decision process is a good strategy without training corpus). The second one, Artex is based on the inner product of main topic and pseudo-words vectors. The third system is Reg, a graph-based summarizer. Our three summarizers have obtained very good results in manual evaluations. Reg is the better system in terms of readability, syntax, diversity and structure manual evaluations. We show that a simple statistical summarizers without knowledge obtains good performances in this complex summarization and tweet contextualization task. Rank System (Run) Readability Syntax Diversity Structure Average 1 Reg (358) 94.82% 87.31% 72.17% 93.10% 86.85% 2 Cortex (356) 95.24% 85.19% 70.31% 92.40% 85.78% 3 Artex (357) 94.88% 82.53% 71.34% 91.58% 85.08%