For the participation of Dublin City University (DCU) in the INEX-2012 tweet contextualization task, we investigated sentence retrieval methodologies. The task requires providing the context to an ad-hoc real-life tweet. This context is to be constructed from Wikipedia articles. Our approach involves indexing the passages in Wikipedia articles as separate retrievable units, extracting sentences from the top ranked passages, computing the sentence selection score for each such sentence with respect to the query, and then returning the top most similar ones. The simple sentence selection strategy performed quite well in the task. Our best run has ranked rst from the readability perspective and ranked eighth as ordered by informativeness out of 33 o cial runs.
The tweet contextualization task was rst introduced at INEX in 2011. The task requires construction of a short summary so as to explain the context associated with a given tweet. This context information has to be constructed from Wikipedia articles. As an example, for the CNN tweet \RT @CNNLive: View stake-out camera at funeral home where #WhitneyHouston body is expected to arrive in New Jersey. Watch live: http://t.co/nyqT4PUa", the system is expected to provide such expository information as who is Whitney Houston, what is she famous for, how did she die etc.
The task being di erent from standard ad-hoc IR poses with its own set of challenges. Firstly, the tweet text is very di erent from keyword based queries of ad-hoc search or web search. This necessiatates applying pre-processing steps on the tweet texts to get an appropriate query string. For example, the tweet hash-tags do not exist in Wikipedia articles and needs to be appropriately processed to get a useful query term. Secondly, a standard passage retrieval may not be suitable for the task because of the restriction on the length in the reported summary. The text in a passage itself may surpass the length theshold requirement of the summary. It thus makes sense to decompose passages into smaller units, i.e. sentences, and collate them together.
Previous approaches to INEX-QA have mostly used passage retrieval coupled
with a summarizer. Sentence retrieval on the other hand has widely been
employed in TREC-QA tasks for both factoid and de nition question answering [
In this section, we describe our system details. After describing the document and query processing, and retrieval, we focus on to our working methodologies for sentence selection. 2.1
We used a modi ed version of the SMART1 system for the experiments at INEX
2012. Each paragraph from the Wikipedia corpus2 was indexed as a retrievable
document unit. The beginning of a passage is marked by the XML tag <p>.
This resulted in a total of over 26M passages to retrieve from. Extracted
portions of documents, namely text under the <title>, <p>, <h>, <t> tags, were
indexed using single terms and a controlled vocabulary (or pre-de ned set) of
statistical phrases following Salton's blueprint for automatic indexing [
The tweet texts were pre-processed to produce queries to retrieve against the indexed collection. The pre-processing steps are described as follows. The URLs from the tweets were removed employing a regular expression based pattern matcher. Medial capital words, i.e. words with inner uppercase letters, were split into separate words e.g. the word \WhitneyHouston" was decomposed into \Whitney" and \Houston". Tweet hash-tags were split up into the pre x # character followed by the word, e.g. \#Whitney" was decomposed into # and \Whitney".
The following word breaking rules were applied to split hashtags starting with \#" and usernames starting with \@": A break between the last and current character is employed if: 1 ftp://ftp.cs.cornell.edu/pub/smart/ 2 http://dev.termwatch.es/esj/Term2IR/2012/data/tweetcontext2012corpus.
xml.gz 3 http://www.d.umn.edu/~tpederse/nsp.html i) the last character is lower case and the current character is upper case or digit (e.g. \OccupyWallStreet" ->\Occupy Wall Street"); ii) the last character is upper case and the last character of a valid acronym, the current character is also upper case or a digit (e.g. \CNNNews" -> \CNN News"; iii) the last character and the current character have di erent case and the resulting word would be longer than 3 characters. 2.3
The context for each tweet was constructed in two passes as follows. In the rst
pass, we retrieved N passages using language modelling (LM) [
Language Modelling Similarity. The most simple sentence scoring technique is that of scoring a sentence S by its LM score computed with respect to the query i.e. the pre-processed tweet text. This is done as shown in Equation 1.
P (SjQ) / Y q2Q
P (qjS) + (1
)P (q)
Note that the smooting parameter used in Equation 1 is di erent from
was used for retrieving the passages as discussed in Section 2.3.
Relevance Model Similarity. The second sentence selection strategy which
we use is derived from relevance model (RLM) term scores [
P (wjqi) /
R X P (wjDj )P (qijDj ) j=1 (1) which (2) We can easily extend this notion of relevance model weighting of terms to whole sentences by simply aggregating over the constituent words of a sentence. This is shown in Equation 3 which we use to score every sentence and select the top-scoring ones in the returned summary.
P (SjR) = Y P (wjR)
w2S We then use the topic smoothed document models in the estimation of RLM i.e. we use the de nition of P (wjD) as obtained from Equation 4 in 2 to obtain an extended RLM sentence selection methodology which we name topical relevance model (TRLM) similarity.
R P (wjqi) / X
K
X P (wjzk; )P (zkjDj ; ) P (qijDj )
j=1 k=1
Topical Relevance Model Similarity. This sentence selection score is based
on an extended version of relevance model (RLM) similarity. In our extended
relevance model, we compute the probabilities P (wjD)s by marginalizing them
over a set of latent topics. Firstly, we estimate the topic distribution over the set
of top ranked passages retrieved in the initial step by latent Dirichlet allocation
(LDA) [
P (wjD) =
K X P (wjzk; )P (zkjD; ) k=1 (3) (4) (5) 3
We submitted three o cial runs (run ids: 185, 186 and 187) for the INEX-2012 Tweet contextualization task. The rst pass passage retrieval for each of the three runs is identical and follows the description of Sections 2.1, 2.2 and 2.3. The sentence retrieval strategies of each of these runs is di erent. Run 185 used simple language modelling (LM) similarity, run 186 used RLM similarity, whereas run 187 used TRLM similarity to score sentences. The number of top documents used for the (T)RLM estimation was set to 20. For TRLM, the additional parameter K, i.e. the number of topics, was set to 5. Our submissions did not use any automatic summarization techniques for sentence selection. We rather relied on pure IR-based approaches to generate the twweet contexts. 4
The tweet contexts were evaluated with two measures: a) informativeness, which
measures the closeness of the answer string with a golden reference with the
help of KL divergence between the two; and b) readability, which measures the
syntactic coherence of the text such as whether it has grammatical errors, has
unresolved anaphora or is redundant etc [
Table 1 reports the o cial results of our three submitted runs. Along with
our runs, the table shows the o cial best run as measured by informativeness
and also the run submitted by the organizers as the baseline. Informativeness
evaluation involves computation of three metrics: the KL divergence between
the golden summary and the returned summary for uni-grams, bi-grams, and
bi-grams with two allowable gaps in between [
It can be seen that the most simple sentence retrieval technique using LM similarity fairly well, achieving rank eight, as measured by informativeness. This run in fact achieves the best readability result.
Our other runs, i.e. the (T)RLM based sentence selection strategies, have not performed well in the o cial evaluation. The release of o cial relevance assessments namely the reference summary context for each tweet would enable us to tune the parameters of the two other sentence selection strategies in order to achieve an improved performance. 5
In our rst participation at the INEX Tweet contextualization task, we applied sentence retrieval to construct answer fragments for each tweet. Three di erent sentence selection methodologies were used: i) language modelling (LM) score, ii) relevance modelling (RLM) scoring of a sentence by accumulating over the RLM scores of its constituent terms, and iii) topical relevance modelling (TRLM) scoring of a sentence by accumulating over the topic smoothed RLM scores of its constituent terms.
The results con rm that simple IR-based sentence selection techniques can perform fairly well on both the informativeness and the readability metrics, without the application of any complex NLP techniques. The main advantage of the sentence retrieval methodologies is that these are very fast in contrast to computationally intensive NLP methods.
This research is supported by the Science Foundation Ireland (Grant 07/CE/I1142) as part of the Centre for Next Generation Localisation (CNGL) project.