User-generated content on social websites such as Twitter is known to be an important source of real-time information on signi cant events as they occur, for example natural disasters. Our participation in the FIRE 2016 Microblog track, seeks to exploit WordNet as an external resource for synonym-based query expansion to support improved matching between search topics and the target Tweet collection. The results of our participation in this task show that this is an e ective method for use with a standard BM25 based information retrieval system for this task.
User-generated content on social media websites such as
Twitter is known to be an important real-time source of
information on various events as they occur, including disaster
events like oods, earthquakes and terrorist attacks. If
information relevant to these events can be reliably identi ed
automatically, there is huge potential to exploit it in the
management of the response to these events by disaster and
relief agencies. This raises the challenge of developing
methods to identify the true relevant information from among the
vast volume of content posted to mainstream social media
channels. The FIRE 2016 Microblog task [
Our analysis of the task data showed that a signi cant problem in addressing this task is the di culty in matching the words present in each search topic and those used in the very short microblog documents. Such di erences relate both to di erent choice of vocabulary in the topics and documents, and also to di ering levels of speci city in the words used. In order to address this challenge, we investigate the use of synonym-based query expansion using WordNet for each search topic. The motivation for this approach is to expand the topic to enhance the chance of matching it with relevant tweets in the target collection. The risk of adopting this strategy using resources such as WordNet without taking into account the context within the topic, is that apart from matching with relevant items, we will also match large numbers of non-relevant items. In this case our objective of increasing recall of relevant tweets, will be tempered by low precision arising from retrieval of non-relevant tweets.
In following section, we rst describe the task in overview, we then introduce our method and the experiments that we carried out using it, including details of the dataset and the external resources that we used, nally we present the results that we obtained and draw conclusions. 2.
The FIRE 2016 Microblog track [
The main challenges for this ad-hoc search task are: Identifying speci c keywords relevant to each broad topic within each tweet which only contains a few words (140 characters at most), most of which do not contain speci c keywords relating to the disaster even the tweet itself is relevant to the search topic.
Dealing with noise in the content of the short tweet documents which are often written in an informal style using abbreviations, colloquial terms, etc; 3.
We begin this section by summarising details of the dataset, and then describe our experiments and the results obtained. Listing 1: Json Parser Code import j s o n i f t a r g e t = open ( ' . t x t ' , 'w ' ) f o r i in a D i c t : l i n e = s t r ( i ) + ' '+ a D i c t [ i ] t a r g e t . w r i t e ( l i n e ) t a r g e t . w r i t e ( ' nn ' ) print i t a r g e t . c l o s e ( )
In order to obtain the dataset of tweets for the task, we followed the instruction provided by the task organizers. They provided: a text le of 50,068 tweetids; a Python script, along with the libraries that are required by this script, to crawl the tweets.
We used their instructions to download the listed tweets arising from the Nepal earthquake in April 2015. A total of 49,894 tweets were downloaded and written into a Json le. We then prepared a Json parser to decode and extract the information that we needed which consisted of only the tweet id and content of each tweet. Listing 1 shows the code.
The provided query set contained 7 topics in TREC format, each of which contains three parts: title, brief description, and a more detailed narrative on what type of tweets will be considered relevant to the topic. Listing 2 presents an example of the TREC format topic:
Listing 2: TREC Topic Example <top> <num> Number : FMT1 < t i t l e > WHAT RESOURCES WERE AVAILABLE <d e s c > D e s c r i p t i o n : I d e n t i f y t h e m e s s a g e s which d e s c r i b e t h e a v a i l a b i l i t y o f some r e s o u r c e s . <n a r r > N a r r a t i v e : A r e l e v a n t message must mention t h e a v a i l a b i l i t y o f some r e s o u r c e l i k e food , d r i n k i n g water , s h e l t e r , c l o t h e s , b l a n k e t s , human r e s o u r c e s l i k e v o l u n t e e r s r e s o u r c e s t o b u i l d o r s u p p o r t i n f r a s t r u c t u r e l i k e t e n t s , water f i l t e r , power s u p p l y and s o on . Messages i n f o r m i n g t h e a v a i l a b i l i t y o f t r a n s p o r t v e h i c l e s f o r a s s i s t i n g t h e r e s o u r c e d i s t r i b u t i o n p r o c e s s would a l s o be r e l e v a n t . However , g e n e r a l i z e d s t a t e m e n t s w i t h o u t r e f e r e n c e t o any r e s o u r c e o r m e s s a g e s a s k i n g f o r d o n a t i o n o f money would not be r e l e v a n t . </top> 3.2
Based on our observation of the probable query-document mismatch problems arising from the short length of the tweets and the di ering use of vocabulary in the topics and the tweet, we explore the use of WordNet1 to improve the reliability of query-document marching. We used WordNet to generate synonyms for the terms in each topic. Two experiments were conducted based on the WordNet. In these experiments, Lucene was used to index the tweet set and to carry out the IR. The indexing process followed the following steps: 1. entries from a list of 655 stop words were removed; 2. Porter stemmer was used for stemming the words; 3. BM25 was used for indexing with k 1=1.2, b=0.75. 3.2.1
WordNet is an electronic lexical database and is regarded
as one of the most important resources available to researchers
in computational linguistics, text analysis, and many related
areas. Its design is inspired by current psycholinguistic and
computational theories of human lexical memory. English
nouns, verbs, adjectives, and adverbs are organized into
synonym sets, each representing one underlying lexicalized
concept. Di erent relations link the synonym sets [
WordNet has long been regarded as a potentially useful
resource for query expansion in IR. However, it has met with
1https://wordnet.princeton.edu/
limited success due to its tendency to include contextually
unrelated synonyms for query words which are unrelated.
One of the successful applications of WordNet in IR is found
in [
In this experiment, we also use WordNet to carry out query expansion. WordNet is used as external resource to generate the synonyms for each topic. We limited the number of synonyms for each topic term to 20 maximum, some terms received less synonyms. 3.2.2
Our rst run (named dcu fmt16 1) uses our rst automatic method using WordNet query expansion. In this run, the following 4 steps were applied: 1. remove stop words from each topic; 2. use WordNet to generate the synonyms for each item in every topic; 3. these synonyms are used as expand terms and add them back to each topic; 4. use the expanded topics as new topic to search again (BM25 retrieval model is used for retrieval).
We use the combination of title and narrative elds of the topic in combination as the original topic. An example of an original topic and its extend version are shown in the Appendix. 3.2.3
Our second run (named dcu fmt16 2) is an semi-automatic run which means that the manual selection is involved. This run was carried out using the following steps: 1. use the original topic to search and obtain a rank list; 2. go through top 30 tweets from rank list to select 1-2 relevant tweets and to do query expansion. Number 30 is selected to promise we could nd at least one relevant tweet for some topics. 3. remove the stopwords and duplicate terms from the select tweets, add the rest term to original topic; 4. then, applied WordNet again on the expanded topics and nd synonyms for these terms; 5. nally, add the synonyms to each expanded topic to generate new topics and use them as query to search again to obtain the nal search results. 3.2.4
Since the aim of this track is to identify a set of tweets that are relevant to each topic, set-based evaluation metrics of precision, recall, and MAP are used for evaluation. The gold standard, against which the set of tweets identi ed by the participants are matched, is generated using a \manual run" where human assessors were given the same set of tweets and topics, and asked to identify all possible relevant tweets using a search engine (Indri). While judging the participants' runs, the track organizers arranged for a second round of assessments to judge the relevance of tweets that are identi ed by the participants but were not identi ed during the rst round of human assessment.
Results of our two runs are shown in Table 1. The table shows results for 4 runs, two of them are automatic and the other two are semi automatic. In the automatic runs listed, our submissions were placed third. The Precision@20 of the best automatic run result is 0.4357 where ours is 0.3786. However our automatic run achieved the best MAP@1000 value of 0.1103, which is an increase of 27.93% relative to the best run. Our overall MAP is lower because we only submitted the top 1000 tweets for each topic while other participants submitted more. We received rst place for the semi-automatic method where our Precision@20 is 33.35% higher than the second place run. These numbers show that using WordNet to generate synonyms for topic terms is a positive way to carry out query expansion for this Microblog task. 4.
For our submissions to the FIRE 2016 Microblog Track, we employed WordNet as an external resource to carry out query expansion by retrieving the synonyms of each topic term and using them as the additional query terms to reformulate each topic. We conducted two runs using this method, an automatic run and a semi-automatic run. The semi-automatic involved manual selection of relevant tweets from a rst run and application of WordNet in a subsequent retrieval stage. Our automatic run received the third place among submission, however with the best MAP value. Our semi-automatic run obtained the overall rst place. These positive results show that when a topic is too general and does not contain the necessary terms to match with relevant documents, using WordNet as an external resource to generate synonyms is a good way to make them more e ective. Potentially, using WordNet to retrieve hypernym or hyponym for each topic term maybe another method worth attempt for this task. 5. 6.
This research is supported by Science Foundation Ireland in the ADAPT Centre (Grant 13/RC/2106) (www. adaptcentre.ie) at Dublin City University. MAP@1000 0.0869 0.1103 0.0815 0.0670
MAP 0.1125 0.1103 0.0815 0.0827
Original topic: <num> Number: FMT6 <title> WHAT WERE THE ACTIVITIES OF VARIOUS NGOs / GOVERNMENT ORGANIZATIONS <narr> Narrative: A relevant message must contain information about relief-related activities of di erent NGOs and Government organizations in rescue and relief operation. Messages that contain information about the volunteers visiting di erent geographical locations would also be relevant. However, messages that do not contain the name of any NGO / Government organization would not be relevant. Expanded topic: <num> Number: FMT1 <narr>were activities assorted respective several diverse versatile various NGOs government organization organisation arrangement system administration governance governing body establishment brass constitution formation organizations a relevant message mustiness moldiness must incorporate comprise hold bear carry control hold in check curb moderate take turn back arrest stop hold back contain information about relief related activities di erent organization organisation arrangement system administration governance governing body establishment brass constitution formation organizations indium atomic number four9 indiana hoosier state inwards inward in deliverance delivery saving deliver rescue relief operation. message content subject matter substance messages that incorporate comprise hold bear carry control hold in check curb moderate take turn back arrest stop hold back contain information about volunteers visit see travel call in call inspect in ict bring down impose chew fat shoot breeze chat confabulate confab chitchat chit chat chatter cha er natter gossip jaw claver visiting di erent geographical location placement locating position positioning emplacement localization localisation x locations would besides too likewise well also relevant. message content subject matter substance messages that do not incorporate comprise hold bear carry control hold in check curb moderate take turn back arrest stop hold back contain name whatever whatsoever any NGO government organization would not relevant