This paper describes our semi-automatic keyword based approach for the four topics of Information Extraction from Microblogs Posted during Disasters task at Forum for Information Retrieval Evaluation (FIRE) 2016. The approach consists three phases;
Machine
It is undeniable that microblogging sites have become key
resources of significant information during disaster event [
However, dealing with tweets and identifying specific keywords
are challenging work due to the nature of Twitter. The small,
noisy and fragmented tweets mean they have very simple
discourse and pragmatic structure, issues which still challenge
state-of-the-art NLP systems [
Task description: The aim is to retrieve a number tweets relevant to each topic provided with high precision as well as high recall. The titles of topics are provided in TREC format as the following:
What infrastructure damage or restoration were reported Dataset: Approximately 50,000 tweets that posted during Nepal earthquake disaster were given in JSON format. A main feature of the task is that a gold standard dataset was not provided. In terms of our approach, we propose to achieve the first four topics using keywords extraction with manual work and classification methods.
This paper is organized as follows. First, the components of
approach are described separately. Then, the result analysis and
conclusion are presented. Our work is submitted in FIRE 2016
Microblog track [
Our approach for the Microblog track comprises three phases, Keyword extraction, Retrieval, and Classification (see Figure 1). In the first phase, we extracted all relief resources (keywords) that were available or required. Using those keywords and Terrier1 search engine, we retrieved a number of tweets that each tweet includes a keyword at least in the middle phase. In the last phase, the retrieved tweets are classified into the first and second topics using Support Vector Machine (SVM).
In order to extract keywords, we used separately the following
two methods with manual work. The keywords that we first
extracted is provided in the topic descriptions, such as food,
water, volunteer, money, medicine and transportation. The
quantitative results explored in this phase is presented in Table 1.
Since tweets are usually written in an informal style, the most of
NLP tools show poor performance on Twitter datasets. So we
tried to exploit specific NLP tools which are [
Chunking and Wordnet: One of the basic technique for
information extraction, chunking, is used to identify keywords in
our approach. We defined some chunk grammar, for example,
“CHUNK: {<NNP.?>*<VB.?>+<DT>?<JJ>*<NN|NNS>+}”
based on tagging by POS in the previous step. Next, the nouns
were filtered by Wordnet [
Once we had a bunch of keywords extracted in the previous phase, we retrieved all tweets (around 8620 tweets) that include at least one keyword using those keywords on the Terrier. There are few open search engines however, we chose Terrier taking some of its advantages into consideration. In term of scoring model, we employed BM25 which is based on probabilistic retrieval framework. The rank and scores are used to compute the relevance of a tweet to a topic in further.
The most of tweets that retrieved in previous phase can be significantly related to the first two topics while some of them cannot. For instance, Even though the following two tweets both includes water (a keyword), the former one is related to the first topic what resources were available, whereas the latter tweet is not related to any topic.
Anyone in need of drinking water contact me. Have some can donate #earthquake #Nepal #bhaktapur #ShameOnYou #nepalgov Rs 20 water cost Rs 40 #earthquakenepal #earthquake #Nepal #fuckoff don't #donate #unknown #website Therefore we classified the tweets into three classes – available, required and other. In order to do that, first, we annotated 1000 tweets manually. In preprocessing of classification, all URL, usertag and some symbols were removed. Then we employed three classifiers with basic features such as unigram, bag-of-words and some twitter specific features on WEKA2 open source machine learning software. The best result was executed by SVM (see Table 2).
In term of the third and fourth topic, “What medical resources were available” and “What medical resources were required”, we retrieved the relevant tweets from the tweets of the first topic and second topic using medical relief resources respectively. It is impossible to compare our results to other participants results because we submitted the attempts of only three topics to the organizers. However, the results estimated by the organizers was reasonable, which brought us encourage to complete our work. The result is presented in Table 3. 0,8500 0,4988 0,2204
Overall MAP 0,2420 4
In this paper, we have presented our keyword based approach for the four topics of FIRE2016 Microblog Track. Our system is semi-automatic, which includes manual work in the keyword extraction phase. Moreover, the phases are not integrated with each other.
Next, we plan to improve our system to become automatic and to use advanced methods. 5