According to the organizers, the objective of the 2012 INEX Tweet Contextualization Task is: \...given a tweet, the system must provide some context about the subject of the tweet, in order to help the reader to understand it. This context should take the form of a readable (and short) summary, composed of passages from [...] Wikipedia." We present summarizers Cortex and KL-summ applied to the INEX 2012 task. Cortex summarizer uses several sentence selection metrics and an optimal decision module to score sentences from a document source. KLsumm is a new statistical summarizer based on Kullback-Leibler divergence (the same used by INEX organizers) to score sentences. The results show that Cortex system (using original tweets) outperforms KL-summ on INEX task.
Automatic text summarization is indispensable to cope with ever increasing
volumes of valuable information. An abstract is by far the most concrete and
most recognized kind of text condensation [
An open domain Question-Answering system (QA) has to precisely answer a
question expressed in natural language. QA systems are confronted with a ne
and di cult task because they are expected to supply speci c information and
not whole documents. At present there exists a strong demand for this kind of
text processing systems on the Internet. A QA system comprises, a priori, the
following stages [
Most research e orts in summarization emphasize generic summarization
[11{13]. User query terms are commonly used in information retrieval tasks.
However, there are few papers in literature that propose to employ this approach
in summarization systems [14{16]. In the systems described in [
This paper is organized as follows. In Section 2 we explain the INEX 2012 Tweet Contextualization Track. In Section 3 we explain the methodology of our work. Experimental settings and results obtained with Cortex and KL-summ summarizers are presented in Section 4. Section 5 exposes the conclusions of the paper and the future work. 2
The Initiative for the Evaluation of XML Retrieval (INEX) is an established
evaluation forum for XML information retrieval (IR) [
Like in Question Answering track of INEX 2011, the present task is about contextualizing tweets, i.e. answering questions of the form "What is this tweet about?" using a recent cleaned dump of the Wikipedia6. As organizers claim, the general process involves three steps: { Tweet analysis. { Passage and/or XML elements retrieval. { Construction of the answer.
Then, a relevant passage segment contains: { Relevant information but { As few non-relevant information as possible (the result is speci c to the question). 2.1
The corpus has been constructed from a dump of the English Wikipedia from November 2011. All notes and bibliographic references were removed to facilite the extraction of plain text answers. (Notes and bibliographic references are di cult to handle). Resulting documents contains a title, an abstract and section. Each section has a sub-title. Abstract end sections are made of paragraphs and each paragraph can have entities that refer to Wikipedia pages. 2.2
The committee of INEX has de ned about 1000 tweets for the Track 2012. 1133 tweets in English were collected by the organizers from Twitter7. Tweets were selected and checked among informative accounts (for example, @CNN, @TennisTweets, @PeopleMag, @science...), in order to avoid purely personal tweets that could not be contextualized. Information such as the user name, tags or URLs will be provided. 3 3.1
Cortex [
The INEX 2012 Tweet Contextualization Track evaluation is a real-world complex question (called long query) answering, in which the answer is a summary constructed from a set of relevant documents. The documents are parsed 6 See the o cial INEX 2012 Tweet Contextualization Track Website: https://inex.
mmci.uni-saarland.de/tracks/qa/. 7 www.tweeter.com to create a corpus composed of the query and the the multi-document retrieved by a Perl program supplied by INEX organizers8. This program is coupled to Indri system9 to obtain for each query, 50 documents from the whole corpus.
The idea is to represent the text in an appropriate vectorial space and apply numeric treatments to it. In order to reduce complexity, a preprocessing is performed to the question and the document: words are ltered, lemmatized and stemmed.
The Cortex system uses 11 metrics (see [
By example, the topic-sentence overlap measure assigns a higher ranking for the sentences containing question words and makes selected sentences more relevant. The overlap is de ned as the normalized cardinality of the intersection between the query word set T and the sentence word set S.
Overlap(T; S) = card(S \ T ) card(T )
The system scores each sentence with a decision algorithm that relies on the normalized metrics. Before combining the votes of the metrics, these are partitioned into two sets: one set contains every metric i > 0:5, while the other set contains every metric i < 0:5 (values equal to 0.5 are ignored). We then calculate two values and , which give the sum of distances (positive for and negative for ) to the threshold 0.5 (the number of metrics is , which is 11 in our experiment): = X( i
i=1 = X(0:5 i=1 0:5);
i > 0:5 i); i < 0:5
The value given to each sentence s given a query q is calculated with: 8 See: http://qa.termwatch.es/data/getINEX2011corpus.pl.gz 9 Indri is a search engine from the Lemur project, a cooperative work between the University of Massachusetts and Carnegie Mellon University in order to build language modelling information retrieval tools. See: http://www.lemurproject.org/indri/ (1) (2) (3)
The Cortex system is applied to each document of a topic and the summary is generated by concatenating higher score sentences. 3.2
The main idea of KL-summarizer is to weight the sentences of a document, by minimizing the divergence of each sentence from document source. This idea is quite simple. Several divergence measures can be utilized: Jensen-Shannon, Kullback-Leibler, etc. However, in order to obtain a good summarizer on this speci c task, we decide of implement the same measure of evaluation proposed by the INEX' organizers.
Fresa measure [
Fresa simply considered absolute log-di between frequencies. Let T be the set of terms in the source. For every t 2 T , we denote by CtT its occurrences in the source and by CtS its occurrences in the summary. The Fresa package computed the divergence between source and summaries as: if( > ) then Score(s; q) = 0:5 + else Score(s; q) = 0:5 (4) (5) D(T jjS) = X log t2T
CT
t + 1 jT j log
CS
t + 1 jSj
To score each sentence, several automatic measures were computed: { Fresa1: Uni-grams of single stems after removing stop-words. { Fresa2: Bi-grams of pairs of consecutive stems (in the same sentence). { FresaSU4: Bi-grams with 2-gaps also made of pairs of consecutive stems but allowing the insertion between them of a maximum of two stems.
All Fresa scores, Fresa1 = 1 D(T jjS) are normalized between 0 and 1. High values mean a less divergence of summary from source document. In other words, lower divergences (High Fresa scores) shows a more quantity of content of summary.
So, the relevant sentences will be selected as having the less divergence values. The two rst modules are based on the Cortex system10. Finally, the third 10 See section 3.1. module generates summaries by displaying and concatenating of the relevant sentences.
At rst, the rst 50 documents of the cluster are concatenated into a single multi-document in chronological order. Placing the tweet q (enriched or not) like the title of this long document. The divergence between each sentence among the all others is computed using equation 5. 4
In this study, we used the document sets made available during the Initiative for the Evaluation of XML retrieval (INEX)11, in particular on the INEX 2012 Tweet Contextualization Track.
The strategy of Cortex and KL-summ systems to deal multi-document summary problem is quite simple: rst, a long single document D is formed by concatenation of all i = 1; :::; n relevant documents provided by Indri engine: d1; d2; :::dn. The rst line of this multi-document D is the tweet T . Both summarizers systems extract of D the most relevant sentences following T . Then, this subset of sentences is sorted by the date of documents di. The summarizers add sentences into the summary until the word limit is reached. To evaluate the performance of Cortex and KL-summ systems on INEX tweet contextualization track, we used the online package available from INEX website12.
Two di erent strategies were employed to generate 1133 queries from tweets: 1. No pre-processing of tweet. 2. Enrichment of each tweet by semi-automatic synonyms of 100 heavy terms (their weights were calculated using the tf).
1) No pre-processing or modi cation was applied on queries set. Summarizers use the query as a title of a big multi-document retrieved by Indri engine.
2) Enrichment of tweet. The query has been semi-manually enriched as following. Firstly, a list T terms from the set of tweets was extracted then sorted by their term frequency. Liste T was manually inspected to extract the 100 rst "relevant" terms. These 100 relevant terms were injected into tweets to enriched them.
Table 1 shows an example of the results obtained by Cortex and KL-summ systems using the 50 rst documents retreived by Indri as input. The tweet that the summary should contextualiser in this case was the number 169231181181747200: <topic id="169231181181747200"> <tweet> 11 https://inex.mmci.uni-saarland.de/ 12 http://qa.termwatch.es/data/ CNNLive View stake-out camera funeral home WhitneyHouston body expected arrive New Jersey Watch live </tweet>
The tweets were enriched automatically using the 100 rst terms (sorted by their tf weight) obtained from the words contained in the tweets set.
For example, for the tweet 169231181181747200, strategy 2 produce the following list of synonyms: -- funeral : cremation -- home : domicile -- new : recent -- watch : observe -- live : exist
Then, query 169231181181747200 is enriched as show: q = "cnnlive view stakeout camera funeral cremation home domicile whitneyhouston body expected arrive new recent jersey watch observe live exist"
Compound words are not detected in this phase. Since, \New Jersey" was separated in \New" and \Jersey", then \New" was enriched by \recent". This criterion is simple but it seems well enrich some tweets too shorts.
Table 1 presents Cortex and KL-summ results (queries enriched or not; tweet=167355997915058176 see Appendix) in comparison with the INEX baseline (Baseline summary), and three baselines, that is, summaries including random n-grams (Random uni-grams) and 5-grams (Random 5-grams) and empty baseline. In this particular example, we observe that KL-summ outperforms all summarizers.
Figure 1 shows the o cial results of participants of INEX 2012 contextualization task. The performances (rank) of our summarizers are: Cortex=11/33, Cortex with enriched tweets=21/33 and KL-summ=16/33. In this gure, rank axes represents the SU4-bigrams values.
# " " " " " " " "
! $ % In this paper we have presented the Cortex and KL-summ summarization systems applied on INEX 2012 Tweet Contextualization Track. The rst one is based on the fusion process of several di erent sentence selection metrics. The decision algorithm obtains good scores on the INEX 2012 Tweet Contextualization Track (the decision process is a good strategy without training corpus). The second one is based on the divergence between summary and the source document.
Cortex summarizer using original tweets as inputs has obtained very good results in the automatic FRESA evaluations. In fact, semi-automatic tweet enrichment has disappointed in this task. Cortex using original tweets outperforms Cortex using enriched queries. We think that the strategy of enrichment, without compound words detection, was a very simple process. A module of compound words may improves the performance of this strategy. In other hands, the KLsumm summarizer is slightly less good than Cortex system. In fact, function 5 is not additive. Since, a simple sort based on Fresa scores (1-divergence calculated in local form) is not enough to maximize the global score over the whole document. A more complex strategy of optimization must be implemented in order to deal this problem. We show that simple statistical summarizers show good performances in this complex task.
We present the Cortex summary of tweet 167355997915058176: \Scientists claim 42000-year-old paintings of seals by Neanderthals found in Spanish cave CIZyRotb". (Numbers in bold indicate the weight of each sentence). 0.861 Therefore, as human populations slowly increased, the cave bear faced a shrinking pool of suitable caves, and slowly faded away to extinction, as both Neanderthals and anatomically modern humans sought out caves as living quarters, depriving the cave bear of vital habitat. 0.880 The evidence found in archeological sites suggests these early humans were Nomadic, that they lived in caves, and acquired sustenance by hunting wild boar, red deer, mountain goat s, fallow deer, and horse s, competing with other predators like the leopard, the brown bear, and the wolf. 0.902 Late Oldowan/Early Acheulean humans such as "Homo ergaster/Homo erectus" may have been the rst people to invent central campsites or home bases and incorporate them into their foraging and hunting strategies