The paper presents IRIT's approach used at INEX Tweet Contextualization Track 2014. Systems had to provide a context to a tweet from the perspective of the entity. This year we further modified our approach presented at INEX 2011, 2012 and 2013 underlain by the product of different measures based on smoothing from local context, named entity recognition, part-ofspeech weighting and sentence quality analysis. We introduced two ways to link an entity and a tweet, namely (1) concatenation of the entity and the tweet and (2) usage of the results obtained for the entity as a restriction to filter results retrieved for the tweet. Besides, we examined the influence of topic-comment relationship on contextualization.
Millions of tweets are published every day. Twitter is an online social network and
microblogging that enables to send and read text messages up to 140 characters [
The context has to explain the relationship between a tweet and an entity. It should be a readable summary up to 500 words extracted from a dump of the Wikipedia from November 2012.
This paper presents IRIT’s approach used at INEX Tweet Contextualization Track 2014. Since the task introduced the notion of entities associated to tweets, we include this new feature and propose two ways to link an entity and a tweet:
Moreover, we analyzed the influence of topic-comment relationship within a sentence in contextualization task.
As in previous years, we consider tweet contextualization task as multi-document
extractive summarization [
The paper is organized as follows. The Section 2 presents our method by recalling the principles of the 2011-2013 system and describing the modifications we made. The Section 3 discusses the obtained results. The Section 4 concludes the paper and provides some perspectives. 2 2.1
Firstly, we performed query preprocessing which differs over the runs: 1. In order to link an entity and a tweet we combined the fields entity, topic and content into a single search query. 2. The second way is to process fields entity and content as separate queries and then use the results obtained for the entity as a restriction to filter results retrieved for the tweet. Thus, the document retrieved by using the field content as a query are rejected if they do not coincide with top-ranked documents retrieved by using the field entity.
The queries are encoded by ASCII (characters are normalized). An entity is treated as a single phrase, i.e. a document has to contain all words expressing the entity.
Document retrieval was performed by the Terrier platform [
The next step is to parse tweets and retrieved documents by Stanford CoreNLP
which integrates such tools as POS tagger [
Then, we merged annotations obtained by parsers and Wikipedia tagging. 2.2
We modified the extraction component developed for INEX 2011-2013. As in previous years, the general idea is to compute similarity between the query and sentences and to retrieve the most similar passages.
We model a sentence as a set of vectors: Unigram vector represents the lemmas associated with tokens occurred within the sentence. For unigram vectors we compute cosine similarity measure. A lemma possesses the following features: POS, frequency and IDF. Functional words, such as conjunctions, prepositions and determiners, are not taken into account. POS, frequency and IDF represents vectors of weights for the unigram vector. We used generalized POS (e.g. we merge regular adverbs, superlative and comparative into a single adverb group). NE vector. NE vectors are treated in the following way: = + +1 ℎ (1) where ℎ is floating point parameter given by a user (by default it is equal to 1.0), is the number of NE appearing in both query and sentence, is the number of NE appearing in the query.
Each sentence has a set of attributes, e.g. which section it belongs to, whether it is a title or header, whether it has personal verbs etc. We assumed that relevant sentences come from relevant documents therefore we multiply sentence score by document relevance or/and by inverted document rank. These characteristics are used for sentence weighting.
We introduced an algorithm for smoothing from the local context. We assumed that the importance of the context reduces as the distance increases. Thus, the nearest sentences should produce more effect on the target sentence sense than others. For sentences with the distance greater than k this coefficient was zero. The total of all weights should be equal to one. The system allows taking into account k neighboring sentences with the weights depending on their remoteness from the target sentence. Last year we added smoothing from document beginning. Wikipedia abstracts contain the summary of the entire paper; therefore they can be also used for smoothing. However, this parameter did not improve results. Therefore we didn’t use it this year.
As in 2013, we did not apply anaphora resolution. Neither we used redundancy treatment nor sentence reordering since the analysis of previous results showed that their impact is small.
In 2013 we introduced sentence quality measure based on the product of the Flesch
reading ease test [
Linguistics establishes the difference between the clause-level topic and the discourse-level topic. However, within the bound of this paper we are interested in clause-level topic only. The topic (or theme) is the phrase in a clause that the rest of the clause is understood to be about, and the comment (also called rheme or focus) is what is being said about the topic. In simple English clause the topic usually coincides with the subject, however it is not a case of the passive voice. In most languages the common means to mark topic-comment relation are word order and intonation. Moreover, there exist special constructions to introduce the comment. However, the tendency is to use so-called topic fronting, i.e. to place topic at the beginning of a clause.
We hypothesize that topic-comment relationship identification is useful for contextualization. Quick query analysis provides evidence that an entity is considered as a topic, while tweet content refers rather to comment, i.e. what is said about the entity. Moreover, we assume that providing the context to an entity implies that this context should be about the entity, i.e. the entity is the topic, while the retrieved context presents the comment.
We used these assumptions for candidate sentence scoring. We double the weight of sentences in which the topic contains the entity under consideration.
Topic identification is performed under assumption of topic fronting. We simplify this hypothesis by assuming that topic should be place at the sentence beginning. Sentence beginning is viewed as the first half of the sentence. 3
Summaries were evaluated according to their informativeness and readability. Informativeness was estimated as the overlap of a summary with the pool of relevant passages.
As in previous years, the lexical overlap between a summary and a pool was estimated in three terms: Unigrams, Bigrams and Skip bigrams representing the proportion of shared unigrams, bigrams and bigrams with gaps of two tokens respectively. Official ranking was based on decreasing score of divergence with the gold standard estimated by skip bigrams.
The organizers used 2 gold standards: pool of relevant sentences per topic; pool of noun phrases extracted from these sentences together with the corresponding Wikipedia entry.
The gold standard thorough is a manual run on 1/5 of the 2014 topics.
We submitted 3 runs: 1. The first run (ETC) was performed by the system 2013. As a query three fields entity, topic and content were treated. An entity was treated as a single phrase. 2. The second run (ETC_ENTITY) differed from ETC by double weight for sentences where the entity represented the topic. 3. Unlike ETC, the third run (ETC_RESTR_NOENT) was based on document set restricted by entities (see the subsection 2.1 Preprocessing).
Table 1 and Table 2 provide evaluation results. The evaluation results presented in the Table 1 was based on the pool of relevant sentences, while the results obtained on the pool of noun phrases are given in the Table 2.
ref2013 and ref2012 are the baselines generated using 2013 and 2012 corpus. They are using the same system and index. However, they seem to be artificial. Therefore, we believe that they can be ignored in ranking.
According to the evaluation performed on the pool of sentences, our runs ETC, ETC_ENTITY and ETC_RESTR_NOENT were classified 3-rd, 4-nd and 6-th; while according to the evaluation based on noun phrases, they got slightly better ranks, namely 2, 3 and 5 respectively.
Thus, the best results among our runs were obtained by the system that merges fields entity, topic and content into a single query. The run #360 is better than our runs according to sentence evaluation; nevertheless, it showed worse results according to noun phrase evaluation. Our system is targeted on the nouns and especially named entities. This could provoke the differences in ranking with respect to sentences and noun phrases.
The worst results were showed by the run based on entity restriction. This could be explained by the fact that filtering out the documents that are considered irrelevant to the entity may cause the big loss of relevant documents if they are not top-ranked according to entities. ETC_RESTR_NOENT demonstrated the worst results among our runs even in the case of noun phrases. We believe that this is caused by loss in recall since the importance of noun phrases is not evaluated, but filtering out some documents could have negative effect on noun phrase recall.
The results of ETC and ETC_ENTITY are very close. However, topic-subject identification slightly decreased the performance of the system. Yet we believe that finer topic-comment identification procedure may ameliorate the results.
359 ETC_RESTR_NOENT 0.705 0.7528 0.7632 0.782 0.8112 0.814 0.8022 0.8152 Rank 1 2 3 4 5 6 7 8 9 10 11 12 13 14
Run ref2013 ref2012 361 ETC
Readability evaluation was performed by one assessor over a pool of 12 summaries per run. Readability was estimated as mean average scores per summary over soundness, structure (no unresolved anaphora), non-redundancy (diversity) and syntactical correctness.
The readability results are given in the Table 3. In general we can see that informativeness results are opposite to readability ones. However, our runs kept the same relative order. We received very low score for diversity and structure. This may be related to the fact that we decide not to treat this problem since in previous years their impact was small. Despite we retrieved the entire sentences from the Wikipedia, unexpectedly we received quite low score for syntactical correctness.
ETC_ENTITY demonstrated slightly higher results according to all readability measures except diversity. The differences of readability scores between ETC_RESTR_NOENT and ETC are very small since these runs are very similar. The only difference is the documents used as sources of the retrieved sentences. However, all readability scores of ETC_RESTR_NOENT are lower. This can be caused by lower quality of the documents or the influence of the informativeness on the assessor perception of readability. k n a R n u
R 358 356 357 364 360 ref2013 ref2012 359 363 362 361
ETC
Conclusion y t i l b a d a e
R 94.82% 95.24% 94.88% 88.05% 92.60% 91.74% 91.39% 93.03% 83.68% 83.67% 93.23% 90.88% 91.23% 90.10% x a t n y
S 87.31% 85.19% 82.53% 69.94% 70.35% 69.82% 69.58% 70.64% 67.92% 68.00% 70.41% 68.89% 69.47% 68.30% y t i s r e v i
D 72.17% 70.31% 71.34% 63.91% 58.84% 60.52% 60.67% 53.53% 61.13% 60.81% 50.12% 56.59% 54.93% 53.83% e r u t c u r t
S 93.10% 92.40% 91.58% 86.92% 86.33% 85.80% 85.56% 86.34% 87.55% 87.59% 85.97% 80.88% 81.56% 80.70% e g a r e v
A 86.85% 85.78% 85.08% 77.20% 77.03% 76.97% 76.80% 75.88% 75.07% 75.02% 74.93% 74.31% 74.30% 73.23%
This year we further modified our approach presented at INEX 2011, 2012 and 2013 underlain by the product of different measures based on smoothing from local context, named entity recognition, part-of-speech weighting and sentence quality analysis. We introduced two ways to link an entity and a tweet, namely (1) concatenation of the entity and the tweet and (2) usage of the results obtained for the entity as a restriction to filter results retrieved for the tweet. Besides, we examined the influence of topic-comment relationship on contextualization. Despite these modifications did not improve results, we believe that small changes in implementation may produce positive effect on the system performance.