This paper describes the work of HLJIT-IRMIDIS for the Information Retrieval from Microblogs during Disasters. This track is divided into two sub-tasks. Task 1 is to solve the identification problem of need-tweets and availabilitytweets during the disaster. Task 2 is to solve the matching problem between need-tweets and availability-tweets. For Task 1, the identification of need-tweets and availabilitytweets is formalized into a classification problem. This paper presents a classification method for distinguishing the needtweets and availability-tweets. For Task 2, the match of need-tweets and availability-tweets is formalized into a retrieve problem. This paper proposes a matching method based on language model. The evaluation shows the performance of our approach, which achieved 0.0687 on MAP in Task 1 and 0.1671 on F-Score in Task 2.
Microblogging sites such as Twitter have become important
sources of situational information during disaster events [
For Task 1 is considered as a classification problem in
this paper. We selected three classifiers, AdaBoost [
Intuitively, Task 1 can be viewed as a two-category classification. If we formalize Task 1 of recognition tweet as a classification problem, our objectives focus on answering the following two questions: (1) Which classification-based methods can effectively be applied to the recognition tweet, and (2) which features should be used in the classifier. 2.1
For classification tasks D = {(x1, y1), (x2, y2), ⋯ , ( xm, ym) }, yiϵ{0,1}, where xi is a feature vector and yi is a feature label. IRMiDis Fire2017 submitted three groups of run. We use AdaBoost, SVM-L and SVM-NL classifiers to predict need-tweets and availability-tweets, respectively.
For task1_1, we use the SVM-L classification model. The principle of the model is to classify the data using the hyperplane. The distance from the positive sample point to the hyperplane as the sorting result.
For task1_2, we use Adaboost, which is a family of algorithms that can enhance weak learners to strong learners. The working mechanism of the classifier is to start from the initial training set training at a base learner, according to the performance of the base learner to the training sample distribution of new adjustments. In the previous course, the training samples of the wrong learners received more attention in the follow-up, and then the next-based learner was trained based on the adjusted sample distribution. A probability value with a positive probability greater than 0.5 is used as the sorting result.
For task1_3, we
use SVM-NL. The classification principle is to use the inner product kernel function instead of the high-dimensional space to the non-linear mapping of positive and negative examples of separation. During the test, the classifier generates a prediction probability for the positive case. We use the probability value as the sorting result. 2.2
Content-based microblogging filtering method, affecting a microblogging is need-tweets or availability-tweets factors are the features of the microblogging. For content-based filtering
methods, words are natural features. For the Fire2017 task, we applied the logistic regression model to select 1116 disaster-related words as microblogging features. Feature words can filter out the noise word, but also improve the classification efficiency of the classifier. In this paper, the weight of the feature in the feature library is updated by the
method of gradient descending. Using the gradient descent method, select the appropriate feature learning rate to ensure the appropriate learning rate. Table 1 shows the top 20 features. people's living security items. The extracted words can represent information about the microblogging in the disaster. 3
According to the description of Matching need-tweets and availability-tweets, we formalize the problem as follows. Denote a retrieval problem as IR = ( , , , ( , )) , where Q is need-tweet and D is availability-tweet, F is the rule that satisfies the relevance sorting model, ( , ) for query and document relevance.
Where and are predicted need-tweet and
availabilitytweet in Task 1. The open source retrieval tool indri1 is used
in Task 2. We use the language model based on the Dirichlet
[
( | ) = ( , ) + ( )
| | + where ( ) is language model and μ is a smoothing parameter. 4
We begin this section by summarising details of the dataset, performance
measures, experimental settings, and then describe our experiments result.
This section describes the dataset provided to the shared task participants. 20000 training data with answer and 50000 testing data was provided by the organizers during the Nepal earth-quake in April 2015. 4.2
For Task 1, evaluation is Mean Average Precision (MAP) considering the retrieved ranked list. For Task 2, evaluation is
F-Score. F-Score = 2 *
Recall / tweet that is correctly identified. Recall, i.e., what fraction of overall need-tweets could be correctly matched by at least one availability-tweet. 4.3
Pre-processing: remove punctuation, URL and mention. Parameter selection of feature selection: learning rate = (1) (2) 0.004. Parameter settings for the classifier: the parameters of each classifier are shown in Tables 2.
4.4
n_estimators=100, algorithm=SAMME.R, LearningRate=1.0 kernel=rbf, gamma=auto, probability=true, classweight=12
No 1 2 3 Run ID
0.5400 0.2300
0.1878 0.1276 0.1633 0.0905 0.0493
0.3500 0.0200
0.1405 0.0913 0.1194 0.0468 0.0468 0.0079 0.0687 0.0633 0.0286
From the experimental results, we can see that the Run2 achieves higher Precision@100 than others. For Run2, we submitted 73 Need-Tweets and 216 Need-Tweets, so Recall@1000 is lowest. However, too many negative examples may lead to Recall@1000 of three groups result is too low in the training model. 4.5
Table 4 shows the experimental results of Task 2.
0.1819 0.2051
0.1546 0.1405 0.0913
0.1671 0.1662 0.1264
From the experimental results, we can see that the results of Run1 and Run3 are similar on the F-score. 5
We have described our approach to all of the tasks in the context of IRMiDis fire2017 competition. The evaluation shows the performance of our approach, which achieved Map (0.0687) in Task 1 and F-Score (0.1671) in Task 2. As a future work, we work like to explore deep learning to text matching and information retrieval of the tweets. Meanwhile, also includes finding new filtering techniques and parameters to tackle such informally written documents like tweets.
This work is supported by the Social Science Fund of Heilongjiang Province of China (No. 16XWB02)