INEX Tweet Contextualization Track at CLEF 2012: Query Reformulation using Terminological Patterns and Automatic Summarization

Abstract

The tweet contextualization INEX task at CLEF 2012 consists of the developing of a system that, given a tweet, can provide some context about the subject of the tweet, in order to help the reader to understand it. This context should take the form of a readable summary, not exceeding 500 words, composed of passages from a provided Wikipedia corpus. Our general approach to get this objective is the following: we perform some automatic reformulations of the initial tweets provided for the task (obtaining a list of terms related with the main topic of all them using terminological patterns). Then, using these reformulated tweets, we obtain related documents with the search engine Indri. Finally, we use REG, an automatic extractive summarization system based on graphs, to summarize these documents and provide the summary associated to each tweet.

Keywords

• INEX
• CLEF
• Tweets
• Terms
• Named Entities
• Wikipedia
• Automatic Summarization
• REG

-

The tweet contextualization INEX (Initiative for the Evaluation of XML Retrieval) task at CLEF 2012 (Conference and Labs of the Evaluation Forum) consists of the developing of a system that, given a tweet, can provide some context about the subject of the tweet, in order to help the reader to understand it. This context should take the form of a readable summary, not exceeding 500 words, composed of passages from a provided Wikipedia corpus. Like in the Question-Answering (QA) of INEX 2011, the task to be performed by the participating groups is contextualizing tweets, that is answering questions of the form \what is this tweet about?" using a recent cleaned dump of the Wikipedia. The general process involves: tweet analysis, passage and/or XML elements retrieval and construction of the answer. Relevant passages would be segments containing relevant information and also containing as little non-relevant information as possible (the result is speci c to the question).

The test data are about 1000 tweets in English collected by the organizers of the task from Twitter. They were selected among informative accounts (for example, @CNN, @TennisTweets, @PeopleMag, @science...), in order to avoid purely personal tweets that could not be contextualized. Information such as the user name, tags or URLs is provided. The document collection for all the participants, that is the corpus, has been rebuilt based on a dump of the English Wikipedia from November 2011. Resulting documents are made of a title, an abstract and sections with sub-titles.

We consider that automatic extractive summarization systems could be useful in this QA task, taking into account that a summary can be de ned as \a condensed version of a source document having a recognizable genre and a very speci c purpose: to give the reader an exact and concise idea of the contents of the source" [ 1 ]. Summaries can be divided into \extracts", if they contain the most important sentences extracted from the original text (ex. [ 2 ], [ 3 ], [ 4 ], [ 5 ], [ 6 ], [ 7 ]), and \abstracts", if these sentences are re-written or paraphrased, generating a new text (ex. [ 8 ], [ 9 ], [ 10 ]). Most of the current automatic summarization systems are extractive.

Our general approach is the following: we perform some automatic reformulations of the initial queries provided for the task (obtaining a list of terms related with the main topic of all the tweets using terminological patterns). Then, using these reformulated queries, we obtain related documents with the search engine Indri1. Finally, we use REG ([ 11 ], [ 12 ]), an automatic extractive summarization system based on graphs, to summarize these documents and provide the nal summary associated to each query.

This approach is similar to the one used at QA@INEX track 2010 (see [ 13 ]) and 2011 (see [ 14 ]), since the same summarization system is employed. Nevertheless, in our past participations, the system was semi-automatic, while in this work the system is totally automatic, from the reformulation of the queries using terminological patterns, until the multi-document summarization of all the retrieved documents.

The evaluation of the participant systems involves two aspects: informativeness and readability. Informativeness evaluation is automatic, using the automatic evaluation system FRESA (FRamework for Evaluating Summaries Automatically) ([ 15 ], [ 16 ], [ 17 ]), and readability evaluation is carried out manually (evaluating syntactic incoherence, unsolved anaphora, redundancy, etc.).

Following this introduction, the paper is organized as follows. In Section 2, the summarization system REG is shown. In Section 3, some information about terminology and terminological patterns is given. In Section 4, the methodol1 Indri is a search engine from the Lemur project, a cooperative work between the University of Massachusetts and Carnegie Mellon University in order to build language modelling information retrieval tools: http://www.lemurproject.org/indri/ ogy is explained. In Section 5, experimental settings and results are presented. Finally, in Section 6, conclusions are exposed. 2 2.1

State-of-the-art and Resources

Term Extraction

The notion of term that we have adopted in this work is based on the \Communicative Theory of Terminology" [ 18 ]: a term is a lexical unit (single/multiple word) that activates a specialized meaning in a thematically restricted domain. Terms detection implies the distinction between domain-speci c terms and general vocabulary. Its results are useful for any NLP task containing a domain speci c component such as: ontology and (terminological) dictionary building, text indexing, automatic translation and summarization systems, among others. In spite of its large application eld, its reliable and practical recognition still constitutes a bottleneck for many applications.

As shown in [ 19 ], [ 20 ] and [ 21 ] among others, there are several methods to obtain the terms from a corpus. On the one hand, there are methods based on linguistic knowledge, like Ecode [ 22 ]. On the other hand, there are methods based on single statistical measures, such as ANA [ 23 ] or a combination of them, such as EXTERMINATOR [ 24 ]. Some tools combine both linguistic knowledge and statistically based methods, such as TermoStat [ 25 ], the algorithm shown in [ 26 ] or the bilingual extractors by [ 27 ] and [ 28 ]. However, none of these tools uses any kind of semantic knowledge. Notable exceptions are Metamap [ 29 ], Trucks [ 30 ] and YATE [ 31 ], among others. Also Wikipedia must be considered, since it is a very promising resource that is increasingly being used for both monolingual ([ 32 ], [ 33 ]) and multilingual term extraction [ 34 ].

Most of the tools, in particular those including an important linguistic component, takes into consideration the fact that terms usually follow a small number of POS patterns. In [ 35 ] it was shown that three patterns (noun, noun-adjective and noun-preposition-noun) cover more that 90% of the entries found in medical terminological dictionaries. Many of the above mentioned tools make some use of this fact. Nevertheless, some researchers like in [ 36 ] dynamically calculate the list of patterns found in terminological resources. 2.2

Named Entities Extraction

Named Entity Recognition (NER) may be de ned as the task to identify names referring to persons, organizations and locations in free text; later this task has been expanded to obtain other entities like dates and numeric expressions. This task was originally introduced as possible types of llers in Information Extraction systems at the 6th Message Understanding Conference [ 37 ]. Although initially this task was limited to identify such expressions, later it has been expanded to their labeling with one entity type label (\person", \organization", etc.). Note that an entity (such as \Stanford", the American university at the U.S.) can be referenced using several surface forms (e.g., \Stanford University" and \Stanford") and a single surface form (e.g., \Stanford") can refer to several entities (the university but also an American nancer, several places in the UK or a nancial group). See [ 38 ] for an interesting review.

NER has proved to be a task useful for a number of NLP tasks as question answering, textual entailment and coreference resolution, among others. The recent interest in emerging areas like bioinformatics allows to expand this recognition task to proteins, drugs and chemical names. While early studies were mostly based on handcrafted rules, most recent ones use supervised machine learning as a way to automatically induce rule-based systems or sequence labeling algorithms starting from a collection of training examples.

Often, corpus processing tools include some text handling facilities to perform simple NER detection for facilitating later processing. Some of them are based in language speci c peculiarities such as initial upper case letters together with some heuristics for name entities placed at the beginning of the sentence. This is the case of the tool used for this experiment (see a description in [ 39 ]). 2.3

The REG System

REG ([ 11 ], [ 12 ]) is an Enhanced Graph summarizer (REG) for extract summarization, using a graph approach. The strategy of this system has two main stages: a) to carry out an adequate representation of the document and b) to give a weight to each sentence of the document. In the rst stage, the system makes a vectorial representation of the document. In the second stage, the system uses a greedy optimization algorithm. The summary generation is done with the concatenation of the most relevant sentences (previously scored in the optimization stage).

REG algorithm contains three modules. The rst one carries out the vectorial transformation of the text with ltering, lemmatization/stemming and normalization processes. The second one applies the greedy algorithm and calculates the adjacency matrix. We obtain the score of the sentences directly from the algorithm. Therefore, sentences with a higher score are selected as the most relevant. Finally, the third module generates the summary, selecting and concatenating the relevant sentences. The rst and second modules use CORTEX [ 6 ], a system that carries out an unsupervised extraction of the relevant sentences of a document using several numerical measures and a decision algorithm. 3

Methodology

A main point in this research is to consider that named entities as well as words sequences that agree with the typical terminological patterns (see section 2.1) are representative of the tweets' topic. To test this assertion, we design a methodology to automatically retrieve all signi cant sequences from the tweets that satisfy the above mentioned criteria.

The rst step is to POS tag the tweets le. As a matter of fact, and in order to keep the process fully automatic, a minimal manipulation of the tweets le has been done. It includes only a minor modi cation to allow the text handling tool to keep the tweet id connected to the tweet itself.

The next step, terminological patterns extraction, has been done using an already existent module of the YATE term extraction tool [ 31 ]. This information, together with the POS tagged tweet (to obtain proper nouns info) is used to build the query string for Indri.

Some care has been taken to keep track of multiword sequences as indicated by the Indri query language speci cation (see examples below).

In order to enrich the queries, we use a local installation of a Wikipedia dump2 to expand the terms with redirection information from such Wikipedia info. In this way, a query term like \Falklands" may be searched in the Wikipedia to nd that it can be also referenced as \Falkland Islands"; therefore, the nal query term is rewritten as: #syn(Falklands #1(Falkland Islands))

This strategy is also useful to nd acronyms expansion as \USGS" and \United States Geological Survey" resulting in the following query: #syn("USGS" #1(United States Geological Survey)) Moreover, it allows to nd words with di erent spellings as: #syn(#1(Christine de Pisan) #1(Christine de Pizan))

The resulting query has been delivered to Indri, using track organizer's script, to obtain the Wikipedia pages relevant to every query. The following is an example of a full tweet:

Increasingly, central banks, especially in emerging markets, have been the marginal buyers of gold http://t.co/9mftD5ju via WSJ. and its corresponding query string: #1(marginal buyers of gold),#1(emerging markets), #1(central banks),#syn("WSJ" #1(The Wall Street Journal)) The resulting set of Wikipedia pages has been split in several documents. Each document contains the pages relevant to the query. Such document is the input to the REG summarization system (see section 2.3), which builds a summary with the signi cant passages. 2 This resource has been otained using [ 40 ].

Experiments Settings and Results

As mentioned in section 3, the process is fully automatic. No human intervention has taken place; therefore, errors and/or mistakes in the process may have a multiplicative e ect. Most of such issues are exempli ed as follows: 1. Tweet itself. The tweets le (including 1000 tweets) prepared by the organization includes several errors like: mispelling, joined words, foreign language, etc. Consider the following examples: { 169657757870456833: \Lakers now 17-12 on the season & 12-2 at home. @paugasol 20pts 13rebs 4blks. Bynum 15pts 15rebs. @0goudelock 10pts, two 3 PTers." { 169904294642978816: \@ranaoboy @Utcheychy @Jhpiego Thx for the #wiwchat RTs! Great conversation!" { 169655717538701312: \METTA. WORLD. PEACE." { 170175722449670145: \http://t.co/amQ6IShA" { 170207412366745600: \RT @MexicanProblms: #41. When you're eating junk food y tu mom te dice que no comas "chucherias." #MexicanProblems".

Please note that, in some cases, it results in an empty query string or the resulting sentence is too short, causing POS tagging errors due to lack of context. 2. POS tagging. The output of most of the tools used for tagging (TreeTagger in this case) has some error rate. Unfortunately, errors mentioned above as well as extremely short sentences have a negative in uence in the tagger performance. 3. Wikipedia expansion. It may happen that information added through Wikipedia expansion is not fully useful. This may be the case the only added information is the change of the case of some letters of the query term. 4. Indri query system. As shown in [ 41 ], this retrieval system has its own limits. 5. REG summarization system. The retrieval system issues a number of Wikipedia pages; therefore, it would be necessary to use a multidocument summarization system. As a matter of fact, REG is a single document summarizer, so some redundance may appear in the summaries.

Some of the above issues may cause unusual results in the terminological patterns extraction tool. Therefore, in such cases, the pages retrieved by Indri may not correspond to the information available in Wikipedia about tweets' topics.

The evaluation of all the participant systems in the tweet contextualization INEX task at CLEF 2012 involves two aspects: informativeness and readability. On the one hand, as mentioned, to evaluate the informativeness the automatic FRESA package is used. This evaluation framework includes document-based summary evaluation measures based on probabilities distribution, speci cally, the Kullback-Leibler (KL) divergence and the Jensen-Shannon (JS) divergence. As in the ROUGE package [ 42 ], FRESA supports di erent n-grams and skip ngrams probability distributions. FRESA environment has been used in the evaluation of summaries produced in several European languages (English, French, Spanish and Catalan), and it integrates ltering and lemmatization in the treatment of summaries and documents.

Table 1 includes the o cial results of the informativeness evaluation in the the tweet contextualization INEX task at CLEF 2012. This table presents the scores of the 33 participant runs.

As shown in Table 1, our run (165) obtains the position 22 in the rank. Exactly, it obtains 0.8818 using unigrams, 0.9630 using bigrams and 0.9634 using skip bigrams. The best run in the ranking (178) obtains 0.7734, 0.8616 and 0.8623, respectively.

On the other hand, readability is evaluated manually. Evaluators are asked to evaluate several aspects related to syntactic incoherence, unsolved anaphora, redundancy, etc. The speci c orders given to evaluators are: { Syntax S: \Tick the box is the passage contains a syntactic problem (bad segmentation for example)". { Anaphora A: \Tick the box if the passage contains an unsolved anaphora". { Redundancy R: \Tick the box if the passage contains a redundant information, i.e. an information that have already been given in a previous passage". { Trash T: \Tick the box if the passage does not make any sense in its context (i.e. after reading the previous passages). These passages must then be considered as trashed, and readability of following passages must be assessed as if these passages were not present".

The score is the average normalized number of words in valid passages, and participants are ranked according to this score. Summary word numbers are normalized to 500 words each.

Table 2 includes the nal results of readability evaluation in the tweet contextualization INEX task at CLEF 2012. Estimated average scores are available for: { Relevance: proportion of text that makes sense in context. { Syntax: proportion of text without syntax problems. { Structure: proportion of text without broken anaphora and avoiding redundancy.

These measures were estimated on the same pool of tweets as for previously released informativeness evaluation by organizers.

Runs that failed to provide at least 6 consistent summaries in this pool have been kept apart because the estimates were too uncertain for inclusion in the o cial results. Because of this reason, in Table 2 only 27 runs are shown.

As shown in Table 2, our run (165) obtains the position 7 in the rank. Exactly, it obtains 0.5936 using unigrams, 0.6049 using bigrams and 0.5442 using skip bigrams. The best run in the ranking (185) obtains 0.7728, 0.7452 and 0.6446, respectively.

These results show that the performance of our system is not so good regarding informativeness, but it is much better regarding readability. This di erence between informativeness and readability is also shown by other systems (see for example the best runs in both categories, 178 and 185). In our case, we consider that the mentioned mistakes in the tweets and the fact that the terminology extraction is totally automatic can cause that the pages retrieved by Indri are not as relevant as expected. Nevertheless, using an automatic summarization system, we can guarantee that the quality of readability is acceptable. In this paper, our strategy and results for the tweet contextualization INEX task at CLEF 2012 are presented. The task consists of the developing of a system that, given a tweet, can provide some context about the subject of the tweet, in order to help the reader to understand it. This context should take the form of a readable summary, not exceeding 500 words, composed of passages from a provided Wikipedia corpus. The test data are about 1000 tweets in English collected by the organizers of the task from Twitter.

Our system performs some automatic reformulations of the initial tweets provided for the task (obtaining a list of terms related with their main topic using terminological patterns). Then, using these reformulated tweets, we obtain related documents with the search engine Indri. Finally, we use REG to summarize these documents and provide the nal summary associated to each tweet.

The results show that, comparing to the other participants, the performance of our system is not so good regarding informativeness (probably due to mistakes in the tweets and problems in the terminology extraction process), but it is much better regarding readability (probably due to the fact of using a summarization system).

In the future we plan to follow several parallel lines: i) to improve term selection and its expansion to re ne the queries and therefore to improve the pertinence of the Wikipedia pages retrieved by Indri; ii) to further investigate the actual pertinence of the Wikipedia retrieved pages to the query; and iii) to check the actual weight of summarization process in the full task by testing other summarization systems.

References

1. Saggion , H.; Lapalme, G. ( 2002 ). Generating Indicative-Informative Summaries with SumUM . Computational Linguistics 28 ( 4 ). 497 - 526 .

2. Edmunson , H. P. ( 1969 ). New Methods in Automatic Extraction . Journal of the Association for Computing Machinery 16 . 264 - 285 .

3. Nanba , H.; Okumura, M. ( 2000 ). Producing More Readable Extracts by Revising Them . In Proceedings of the 18th Int. Conference on Computational Linguistics (COLING-2000). Saarbrucken . 1071 - 1075 .

4. Gaizauskas , R.; Herring, P. ; Oakes , M. ; Beaulieu , M. ; Willett , P. ; Fowkes , H. ; Jonsson , A. ( 2001 ). Intelligent access to text: Integrating information extraction technology into text browsers . In Proceedings of the Human Language Technology Conference. San Diego . 189- 193 .

5. Lal , P. ; Reger , S. ( 2002 ). Extract-based Summarization with Simplication . In Proceedings of the 2nd Document Understanding Conference at the 40th Meeting of the Association for Computational Linguistics . 90 - 96 .

6. Torres-Moreno , J-M. ; Velazquez-Morales , P. ; Meunier , J. G. ( 2002 ). Condenses de textes par des methodes numeriques . In Proceedings of the 6th Int. Conference on the Statistical Analysis of Textual Data (JADT) . St. Malo . 723 - 734 .

7. da Cunha, I.; Fernandez, S. ; Velazquez, P. ; Vivaldi , J.; SanJuan, E.; Torres-Moreno , J-M. ( 2007 ). A new hybrid summarizer based on Vector Space Model , Statistical Physics and Linguistics. Lecture Notes in Computer Science 4827 . 872 - 882 .

8. Ono , K. ; Sumita , K. ; Miike , S. ( 1994 ). Abstract generation based on rhetorical structure extraction . In Proceedings of the Int. Conference on Computational Linguistics. Kyoto . 344 - 348 .

9. Paice , C. D. ( 1990 ). Constructing literature abstracts by computer: Techniques and prospects . Information Processing and Management 26 . 171 - 186 .

10. Radev , D. ( 1999 ). Language Reuse and Regeneration: Generating Natural Language Summaries from Multiple On-Line Sources . PhD Thesis . New York, Columbia University.

11. Torres-Moreno , J-M. ; Ram rez , J. ( 2010 ). REG : un algorithme glouton applique au resume automatique de texte . Proceedings of the 10th Int. Conference on the Statistical Analysis of Textual . Roma, Italia.

12. Torres-Moreno , J-M. ; Ram rez , J.; da Cunha, I. ( 2010 ). Un resumeur a base de graphes, independant de la langue . In Proceedings of the Int. Workshop African HLT 2010. Djibouti.

13. Vivaldi , J.; da Cunha, I.; Ram rez , J. ( 2011 ). The REG summarization system with question reformulation at QA@INEX track 2010 . In Geva, S. et al. (eds.). INEX 2010, Lecture Notes in Computer Science 6932 . 295 - 302 . Berl n: Springer.

14. Vivaldi , J.; da Cunha, I. ( 2012 ). QA@INEX Track 2011 : Question Expansion and Reformulation Using the REG Summarization System . Lecture Notes in Computer Science (LNCS) 7424 . 257 - 268 . Berlin: Springer.

15. Saggion , H.; Torres-Moreno, J-M. ; da Cunha, I.; SanJuan , E.; Velazquez-Morales , P. ; SanJuan, E. ( 2010 ). Multilingual Summarization Evaluation without Human Models . In Proceedings of the 23rd Int. Conference on Computational Linguistics (COLING 2010 ). Pekin.

16. Torres-Moreno , J-M. ; Saggion , H.; da Cunha, I.; SanJuan , E.; Velazquez-Morales , P. ( 2010 ). Summary Evaluation With and Without References . Polibitis: Research journal on Computer science and computer engineering with applications 42.

17. Torres-Moreno , J-M. ; Saggion , H.; da Cunha, I.; Velazquez-Morales , P. ; SanJuan, E. ( 2010 ). Ealuation automatique de resumes avec et sans reference . In Proceedings of the 17e Conference sur le Traitement Automatique des Langues Naturelles (TALN) . Montreal: Univ. de Montreal et Ecole Polytechnique de Montreal.

18. Cabre , M. T. ( 1999 ). La terminolog a. Representacion y comunicacion . Barcelona: IULA.

19. Cabre , M. T.; Estopa , R. ; Vivaldi, J. ( 2001 ). Automatic term detection. A review of current systems . Recent Advances in Computational Terminology 2 . 53 - 87 .

20. Pazienza , M. T.; Pennacchiotti , M. ; Zanzotto , F.M. ( 2005 ). Terminology Extraction: An Analysis of Linguistic and Statistical Approaches . Studies in Fuzziness and Soft Computing 185 . 255 - 279 .

21. Ahrenberg , L. ( 2009 ). Term Extraction: A Review. (Unpublished draft) .

22. Alarcon , R.; Sierra, G. ; Bach, C. ( 2008 ). ECODE: A Pattern Based Approach for De nitional Knowledge Extraction . In Proceedings of the XIII EURALEX Int. Congress. Barcelona: IULA , UPF , Documenta Universitaria. 923 - 928 .

23. Enguehard , C. ; Pantera , L. ( 1994 ). Automatic Natural Acquisition of a Terminology . Journal of Quantitative Linguistics 2 ( 1 ). 27 - 32 .

24. Patry , A. ; Langlais , P. ( 2005 ). Corpus-based terminology extraction . In Proceedings of 7th Int. Conference on Terminology and Knowledge Engineering . Copenhagen.

25. Drouin , P. ( 2002 ). Acquisition automatique des termes: l'utilisation des pivots lexicaux specialises . Ph.D. Thesis . Montreal (Canada): Universite de Montreal.

26. Frantzi , K. T.; Ananiadou , S. ; Tsujii , J. ( 2009 ). Erdmann, M. ; Nakayama , K. ; Hara , T. ; Nishio , S. ( 2009 ). The C-value/NC-value Method of Automatic Recognition for Multi-word Terms . Lecture Notes in Computer Science 1513 . 585 - 604

27. Vintar , S. ( 2010 ). Bilingual term recognition revisited: The bag-of-equivalents term alignment approach and its evaluation . Terminology 16 ( 2 ). 141 - 158 .

28. Gomez Guinovart , X. ( 2012 ). A Hybrid Corpus-Based Approach to Bilingual Terminology Extraction . In I. Moskowich and B. Crespo (eds.) . Encoding the Past, Decoding The Future: Corpora in the 21st Century. Cambridge Scholar Publishing: Newcastle upon Tyne . 147 - 175 .

29. Aronson , A. ; Lang , F. ( 2010 ). An overview of MetaMap: historical perspective and recent advances . Journal of the American Medical Informatics Association 17 ( 3 ). 229 - 236 .

30. Maynard , D. ( 1999 ). Term Recognition Using Combined Knowledge Sources . Ph.D. Thesis . Manchester Metropolitan University. Manchester (UK).

31. Vivaldi , J. ( 2001 ). Extraccion de candidatos a termino mediante combinacion de estrategias heterogeneas . Ph.D. thesis . Universitat Politecnica de Catalunya. Barcelona (Spain).

32. Vivaldi , J.; Rodr guez, H. ( 2010 ). Using Wikipedia for term extraction in the biomedical domain: rst experiences . Procesamiento del Lenguaje Natural 45 . 251 - 254 .

33. Cabrera-Diego , L.; Sierra , G. ; Vivaldi , J. ; Pozzi, M. ( 2011 ). Using Wikipedia to Validate Term Candidates for the Mexican Basic Scienti c Vocabulary . In Proceedings of LaRC 2011: First Int. Conference on Terminology, Languages, and Content Resources . Seoul. 76- 85 .

34. Erdmann , M. ; Nakayama , K. ; Hara , T. ; Nishio , S. ( 2009 ). Improving the Extraction of Bilingual Terminology from Wikipedia . ACM Transactions on Multimedia Computing, Communications and Applications 5 ( 4 ). 31 . 1 - 31 . 16 .

35. Estopa , R. ( 1999 ). Extracting Lexical Semantic Knowledge from Wikipedia and Wiktionary . Ph.D. Thesis . Pompeu Fabra University. Barcelona (Spain).

36. Nazar , R.; Cabre, M. T. ( 2012 ). Supervised Learning Algorithms Applied to Terminoloy Extraction . In 10th Terminology and Knowledge Engineering Conference.

37. Grishman , R.; Sundheim, B. ( 1996 ). Message Understanding Conference - 6: A Brief History . In Proceedings of the 16th Int. Conference on Computational Linguistics . 466 - 471 .

38. Nadeau , D. ; Sekine , S. ( 2007 ). A survey of named entity recognition and classi - cation . Journal of Linguisticae Investigationes 30 ( 1 ). 3 - 26 .

39. Mart nez, H.; Vivaldi , J. ; Villegas, M. ( 2010 ). Text handling as a Web Service for the IULA processing pipeline . In Proceedings of the 7th conference on International Language Resources and Evaluation (LREC'10) . 22 - 29 .

40. Zesch , T. ; Muller, C. , Gurevych , I. ( 2008 ). Extracting Lexical Semantic Knowledge from Wikipedia and Wiktionary . In 6th LREC Conference Proceedings. 1646-1652.

41. Strohman , Trevor; Metzler, Donald; Turtle, Howard; Croft, Bruce ( 2005 ). Indri: A language-model based search engine for complex queries . University of Massachusetts Amherst. CIIR Technical Report IR-407.

42. Lin , C-Y. ( 2004 ). ROUGE: A Package for Automatic Evaluation of Summaries . In Proceedings of Text Summarization Branches Out: ACL-04 Workshop . 74- 81 .