The micro blogging sites contain the emotion and expression of the public in raw format. The data can be used to extract much meaningful information that could be used to develop technologies for future use. There are numerous micro blogging sites available these days that are used in different contexts. Some are used basically for conversation, some for image and video sharing, and some for formal and official purposes. Twitter is one of the most outspoken platform for sharing the emotions and comments on almost every topic starting from sport to entertainment, religion to politics and many more. The paper attempts to extract information from the database of the tweets collected from Twitter. The task is to develop methodologies for extracting tweets that are relevant to each topic with high precision. This paper presents the nita_nitmz team participation in FIRE 2016 Microblog track.
The faster growth of Internet in current period provides new
sources of information. Now a day’s people prefer to express
themselves more often on social sites than any print media. The
idea of Information Extraction from Microblogs Posted during
Disasters was introduced by Sarah Vieweg et.al., in 2010
Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems [
The problem of Information Extraction from Microblogs Posted
during Disasters is researched for a couple of years starting from
2010 by Sarah Vieweg et.al. [
A large set of microblogs (tweets) posted during a recent disaster event was be made available, along with a set of topics (in TREC format). Each ‘topic’ identified a broad information need during a disaster, such as – what resources are needed by the population in the disaster affected area, what resources are available, what resources are required / available in which geographical region, and so on. Specifically, each topic contained a title, a brief description, and a more detailed narrative on what type of tweets will be considered relevant to the topic. The participants are required to develop methodologies for extracting tweets that are relevant to each topic with high precision as well as high recall.
• •
Around 50,000 microblogs (tweets) from Twitter, those were posted during the Nepal earthquake in April 2015.
with a script that was be used to download the tweets using the Twitter API. Out of 50,000 tweets only 48845 could be downloaded on my experimental setup. A set of 5 – 8 topics in TREC format, each containing a title, a brief description, and a more detailed narrative.
For the given task we created the required searching configuration on Apache Nutch 0.9 which is a highly extensible and scalable open source web crawler software project. The implementation of the said task is done in two steps. First is creating the searching environment, Secondly apply appropriate test queries to search the results from the previously configured Nutch using Tomcat server.
The python script provided by the organizers helped to get the tweets. But file generated was of json type. So another script had to be developed to extract the tweets from the json file. The json file contained the tweets, tweet_id and many more metadata. First problem arises when after extracting tweets from the json file, we found out that only 48845 tweets were downloaded via the given script. As per the norms of the Apache Nutch, the tweets had to separated into different files. Since the task was to extract relevant tweets , the files containing the tweets had to be named according to their tweet_ids. We developed two files one containing the tweets only and other containing the tweet_ids. After that we wrote a code to take the tweet_id from one file , create a text file with that name , take a tweet from another file and store it in the newly created file. But due to some unavoidable errors, only 48815 file could be created, each having the name as tweet_id and containing the corresponding tweet inside.
We used another code to store the addresses of the different file in the urls.txt file. We started the crawling part using the nutch. It works in following steps: Injecter: All the URLs are taken by the injector from the seed.txt (here urls.txt) file, compares urls with regexurlfiler regex and update crawldb with supported urls.. The crawldb maintains information on all known URLs (fetch schedule, fetch status, metadata, …).
Generator: Based on the data of crawldb, the generator selects best-scoring urls due for fetch and the segments directory is created.
Fetcher and CrawlDb Update: Next, the fetcher fetches the remote pages of the URLs on the fetch list and updates it to the segment directory. This step takes a lot of time.
Parser: The contents of each web page are parsed. If the crawl produces another extension to an already existing one, the updater adds the new data to the crawldb.
Inverting: The links need to be inverted before indexing. The fact that the number of incoming links is more valuable than the outgoing links is taken care off, similar to how Google PageRank works. The inverted links are saved in the linkdb.
Indexing, Deduplicating and Merging: Using data from crawldb, linkdb and segments, the indexer creates an index and saves it. The Lucene library is used for
Indexing.
Now, we can search for tweets regarding the crawled database. The searching of the test query takes place in the following steps: Stop Word Removal: From the given query the stop words have to be removed, because they do not contribute much to the searching procedure.
Query Segmentation: for a given set of words in a query, the search engine may not give proper results, so it searched for every combination of the words contained in the search query.
Merging: As discussed in the Query segmentation, the results for a given query may not be available; so the results obtained from the different combination of the words of the query text is merged to get one of the probable results of the search. 5
We submitted 37 results of the four query subtexts. The run submission was accepted in the category of semi automatic run. The results from the organizers after judging the submitted run is provided in Table 1. The results are not encouraging, but considering the fact that we started from the scratch, we have much to learn. The different participating teams have employed different algorithms to extract the results. We would try to enhance our methodology for future research. 6