We present text processing framework for discovering emergency related events via analysis of information sources such as social networks. The framework performs focused crawling of messages, text parsing, information extraction, detection of messages related to emergencies, as well as automatic novel event discovering and matching them across different information sources. For detection of emergencyrelated messages, we use CNN and word embeddings. For discovering novel events and matching them across different sources, we propose a multimodal topic model enriched with spatial information and a method based on Jensen-Shannon divergence. The components of the framework are experimentally evaluated on Twitter and Facebook data.
Recent research showed that Twitter, Facebook, and other social networks have valuable applications in emergency situations. Since large-scale emergency events give rise to a massive publication activity in social networks [35], these resources accumulate information about situation in affected areas, infrastructure damage, casualties, requests and proposals for help. They have already been used for enhancing situation awareness of affected people and emergency response teams [3, 21, 15], as well as for online detecting and monitoring emergency events like earthquakes [27, 29]. Advanced information retrieval techniques can detect emergencies in text streams automatically so direct appeals to the rescue services through the standard channels may not be needed.
This research continues the previous studies
presented in [
The task set in this work has a global spatial restriction. In particular, we are interested primarily in the events and messages from the Arctic zone. This restriction brings additional difficulties due to sparseness of data, lack of ready-to-use software, methods, and linguistic resources needed for text processing.
In this work, we evaluate several models for detection of emergency related messages based on various types of embeddings and classification techniques including deep learning. We present a multimodal topic model for event discovering that leverages spatial information, as well as describe approaches to assessing event novelty and matching events from different information sources. The experimental evaluations on collections of messages from Twitter and Facebook show that our methods outperform the baselines.
The rest of the paper is structured as follows. Section 2 reviews the related work on methods for novel topic/event detection in text streams. Section 3 describes the natural language pipeline of our system including the subsystem for extraction of emergency related messages. Section 4 presents the developed method for novel emergency event discovering and matching across information sources. Section 5 describes the experimental evaluation of methods. Section 6 concludes and outlines the future work. The work related to our current research includes publications considering the tasks of event detection in microblogs, topic evolution tracking, as well as emerging topic detection. Most of the approaches to these problems can be divided into two major groups.
The first group of methods for emerging event detection and tracking primarily relies on topic models adopted to temporal aspects of the task. They are based on different modifications of PLSA models [13] (often LDA [6]). One of the fundamental works in this area is [5]. It proposes several dynamic topic models that align topics across time steps with logistic normal distribution, train with approximation based on variational Kalman filters and perform inference with the help of wavelet regression. Another fundamental model named “topics over time” is presented in [30]. Authors propose a method for jointly modelling both word co-occurrences and localization in continuous time without employing Markov assumption. Another topic model that takes into account temporal dimension is on-line LDA presented in [1]. In this approach, distributions generated on the previous time steps are used as priors for word generation on the current step. For each topic, the method builds transformation matrix that captures the evolution of the topic over time. Authors consider a topic as emerging if it is significantly different from topics in the same time period or from all topics seen before. For topic comparison, Kullback-Leibler divergence is used. In [31], researchers instead of creating monolith Bayesian model propose to learn a topic model and a transition matrix to shift distributions over discrete time steps. They formulate the problem of model learning as minimizing the least square error between predicted topic distribution using transformation and the actual topic distribution of new documents. The proposed approach provides the ability to predict topic trends in the future. Other notable related work on topic models for emerging topic detection in microblog data include Twitter-LDA [12], BBTM (bursty biterm topic model) [34], and TopicSketch [33].
The second group of methods is based on detection of emerging features like terms, keywords, or token segments, and clustering of them. In [7], to define emerging terms authors use two metrics named “nutrition” and “energy function” (biology metaphor). Nutrition of a term is calculated as a sum of modified term frequency in a tweet multiplied by author importance (calculated via PageRank) summed through all tweets in a time period. The energy function of a term is proportional to the difference of its current nutrition and its nutrition in the previous time intervals. Authors declare a term as emerging if its energy value is more than “critical drop” value, which is proportional to the average energy of all terms in the current time period. Using cooccurrence of terms, authors build a graph with edges that correspond to the strongest relationships between terms. The emerging terms become seeds of strongly connected components that finally represent emerging topics. Authors of [32] use wavelet analysis for detection of emerging keywords. They consider frequencies of words as signals and decode these signals with wavelet analysis. Some trivial words are filtered away by analyzing their corresponding signal autocorrelations. The remaining words are then clustered to form events with a modularity-based graph partitioning technique. In [8], a real-time framework for detecting hot emerging topics for organizations in social media context is presented. Authors discover emerging topics and extract emerging features from both the organization and topic perspectives. They extract emerging terms by leveraging chi-square test for foreground and background distributions of terms. Topics are discovered by incremental k-means type clustering algorithm. To perform timely identification of hot emerging topics, authors proposed two semisupervised classifiers (based on co-training and selflearning). Authors engineered several features that incorporate an authority of a source, importance of keywords, number of retweets, and some other aspects. In [28], the emerging keywords are identified using significance measure based on outlier detection algorithm. More specifically, authors used exponentially weighted average of terms and co-occurring terms. For detection of novel events, in [20], researchers propose to use instead of single unigrams so called “event segments” – key phrases for an event that possibly refer to named entities or semantically meaningful information units. They cluster event segments into events considering both their frequency distribution and content similarity. Emerging segments are detected by abnormal frequency distribution of the tweet and user frequencies of the segments. Importance of an event is also determined by Wikipedia. Authors consider segments that frequently appear as anchors in Wikipedia more favorable. This approach is intended for finding the most realistic events and to derive the most newsworthy segments to describe the identified events.
The method presented in [14] combines two aforementioned approaches: it uses topic modelling in conjunction with models for emerging terms detection. Topic models are used to detect topic distributions in each time interval. Term novelty is estimated by local weighted linear regression. In order to advance from detection of term novelty to detection of topic novelty, authors solve optimization problem. The solution gives novelty and fading probabilities for a topic. Based on these two probabilities, topic evolution operations are defined subsequently to identify emerging topics from the large number of latent ones and track how these topics evolve over time. To compare topics, authors use Jensen-Shannon distance.
Another approach to emergency event detection employ dictionary learning method [17]. The dictionary contains topics, which are consist of atoms (numerical vectors). Vector representation of documents can be approximated with a linear combination of such atoms. The method consists of two steps: determining novel documents in a text stream and identifying a cluster structure among the novel documents. In the first step, the method checks whether a new document can be represented as a sparse linear combination of known atoms with low error. If it is not the case, the document is considered novel. Such documents are used to learn a new dictionary of novel topics. On the second step, the learned dictionary is used to build clusters of similar novel messages. These clusters are considered as emerging topics.
Our approach to novel event discovering is based on multimodal topic modeling and takes into account spatial information. Its key benefits compared to the previous work are the following. ● It allows to separate similar emergency events happened in different locations (for example, storms or typhoons). ● It provides an obvious way to match messages from different sources (social networks) taking into account location information. ● It can help to reveal location information of an event from a set of scattered messages. 3
Our method for event discovering needs complex preprocessing of natural language texts. We perform basic linguistic analysis, named entity recognition, time recognition, and detection of emergency related texts.
The final results of the natural language processing pipeline are used for three tasks: focused crawling, enriching information about events, creating modularities for topic models. 3.1
The basic linguistic analysis includes tokenization, sentence splitting, pos-tagging, lemmatization, and syntax parsing. The pipeline is implemented via IsaNLP19 – a library that organizes various NLP components for English and Russian. In this paper, we perform experiments only with English texts, therefore, the constructed pipeline contains only components for parsing English.
Tokenization, sentence splitting, postagging, and lemmatization are performed by components based on NLTK toolkit [4]. The syntax parsing is performed by SyntaxNet McParseface [2]. 3.2
We perform extraction of the following types of objects: person’s names, organizations, geographical locations, and ship names. For basic NER extraction, we use Polyglot framework. This system uses distant supervision on Wikipedia for learning underlying model and is able to perform named entity recognition for 40 languages. However, we note that performance of such an approach is not suitable for location extraction due to lack of recall. High recall of spatial information is needed to perform filtering of the text stream and topic modelling. Wikipedia lacks many miscellaneous locations, therefore, there is not enough data for training a good model. Polyglot also lacks the ability to normalize locations.
To improve the recall of location extraction and
achieve the ability to normalize extracted textual
information into geographic coordinates, in the previous
work, we implemented a rule- and dictionary-based
module [
To extract and normalize temporal expressions, we use a combination of two tools: spaCy21 (NLP framework based on deep learning) and a datetimeparser22 (a library based on a set of hand-crafted rules).
For extraction of ship names, in the previous work [11], we implemented a hybrid approach. On the basis of a database of ship names, we implemented a gazetteer that has high recall but low precision due to the fact that many generic words appear to be ship names. To mitigate this problem, we also trained a neural network based on C-LSTM architecture [36]. The network filters out erroneous cases generated by the gazetteer and drastically improves precision and overall F1-score of ship name detection. 3.3
For detection of emergency related tweets, in the previous work, we also used a combination of a gazetteer and a neural network based on C-LSTM architecture. The gazetteer is based on the CrisisLex lexicon, proposed in [23]. This gazetteer generates many false positives that are filtered out by the neural network. To create this solution, in the previous work, we collected a corpus of tweets and trained a neural network on it. In this work, we improve the module for detection of emergency related messages by incorporating more labeled data from CrisisLex corpora [24] and by exploring: ● Various embeddings: word-level: fastText [16] (trained on our own corpus / pre-trained on English Wikipedia), GloVe [26] (Common Crawl with dimension 300 / Twitter with dimension 200), Word2Vec [22], sentence-level: InferSent [9]. ● Various types of models: logistic regression (from scikit-learn), random forest (from scikit-learn), gradient boosting on decision trees (LigthGBM algorithm [18]), fully-connected network (FCN), convolutional neural network (CNN), and C-LSTM as before. 19 https://github.com/IINemo/isanlp 20 http://www.geonames.org/ 21 https://spacy.io/ 22 https://github.com/scrapinghub/dateparser
Facebook Topic crawling of
Facebook Multimodal topic modeling
Generate queries for Facebook Natural language processing
Information extraction (locations, objects, etc.)
Detect emergency messages Linguistic processing Twitter topic crawling Twitter
Filter background topics
Find novel events
New emergency messages among crawled tweets using methods described in section
In the second step, we train multimodal topic model to identify emergency events described by messages and then determine novel events among them by comparing term
distributions of the events from adjacent time periods.
In the third step, we use event-related and locationrelated lexis from the obtained topics to crawl messages
Linguistic processing Information extraction (locations, objects, etc.)
Detect emergency messages Check topic similarity from other sources (Facebook in particular). Then, we apply emergency detection method again and filter out all irrelevant posts. The trained topic model is used to check whether the remaining messages are topically similar to the events extracted from Twitter.
In the first step, we discretize the timeline into small time periods (one day in the experiments). In each time period, multimodal topic model with additive regularization [37, 38] is trained.
Let be a collection of tweets from a time period, let Def be a default modality (regular event-related lexis) and let Loc be a modality devoted to location of events. The main reason to use such modalities is to separate similar events happened in different places in one period of time. We consider each message ∈ tokens, related to those modalities = as a set of ∪ .
The goal of the topic modeling is to find factorization for matrix of empirical probabilities for documents and tokens: ̂( | ) ≈ ( | ) = ∑∈ ( | ) ( | ) = ∑∈ , ∀ ∈ .
( 1) (2)
This problem could be solved by maximizing the weighted sum of the following log-likelihoods with additive regularizers: ( , ) ∈
→ , . = ∈
∈ ∈ ( ) + ( ) + ( ) + earlier similar topics in a predefined time window. modalities, Φ is a matrix of token probabilities for topics, and Θ is a matrix of topic probabilities for documents. As in [37], we apply smooth-sparse regularizers to achieve smooth term distributions in topics and sparse topic distributions in messages: ( ) =
( || ), ( ) = −
( || ), ∈ ∈ (3) (4) where distributions.
and are sampled from some predefined
We apply decorrelation regularizer only for location modality to be able to detect similar events happened in different places at the same time: ( ) = − .
(5) ,∈ ∈
We use BigARTM library [39] to train multimodal models. The result is Φ and Θ matrices for each time period.
4.2
In the second step, we determine whether the extracted events were discussed before. We aggregate several adjacent periods of time to “time windows”. Consider we have topics s and t in the same time window. Denote vectors of token distributions for these topics as Φ and Φ . As in [14], we use Jensen–Shannon divergence between token probabilities for the topics to estimate topic similarity: ( || ) =
( || ) 1 2 1 2 = ( + ).
+ ( || ) , (6)
A topic is denoted as a “new event” if there is no 23 https://github.com/jeanphix/Ghost.py 4.3
In the third step, we match messages related to the same event from different sources, which can be various types of social networks or mass media sites. In experiments, we enriched messages from Twitter related to novel emergency events with Facebook public posts. For each novel event, we construct a search query as a combination of default and location tokens with the highest weights. To crawl Facebook, we use Ghost.py23 library.
We filter obtained posts (leaving only emergency related messages) as described in section 3.3 and extract named entities and locations from them. We infer topicprobabilities matrix Θ for remaining posts using the pretrained
model for the event. Then, we filter all messages, which are not topically similar to the event. Due to the use of multimodal models, information about locations is also taken into account when assessing the similarity of posts. 5 5.1
Dataset and Pre-processing For evaluation of method for detection of emergency related messages, we use the CrisisLexT6 dataset. The dataset consists of 60,000 tweets related to 6 major crisis situations. Emergency related tweets are labeled as “ontopic” and others are labeled as “off-topic”. The preprocessing procedure included elimination of the special characters, as well as conversion of hashtags, emojis, and
Hyperparameters
regression.
Random Forest. Number of estimators: 1,000. No limits to maximum number of features and tree depth. Split quality measure: Gini impurity. Min number of samples per split: 2. Min number of samples per leaf: 1.
Gradient boosting. Maximum tree depth: 20. Number of leaves: 11. Learning rate: 0.05. Feature fraction 0.9. Bagging fraction: 0.8. Min frequency: 5. Number of estimators: 4,000 with early stopping for 200.
FCN. Size of hidden layer: 256. Dropout: 0.5. Number of epochs: 10. Loss: cross entropy. Optimization algorithm: Adam. Learning rate: 0.0001. Weight decay: 0. Batch size: 256.
CNN. Kernel size: [3, 4, 5]. Number of filters: 512. Dropout: 0.5. Optimization algorithm: Adam. Learning rate: 0.0001. Loss: binary cross entropy. Batch size: 128. Vocabulary size: 10,001. Number of epochs: 10 with early stopping for 3 epochs.
Results and Discussion We use 5-fold cross-validation for evaluation. Results are presented in Table 1. We discovered several insights into problems with processing and analyzing crisis and Twitter specific lexicon: ● Sentence-level embeddings are better than averaging word vectors. Averaging embeddings of all words in a tweet blur the real meaning of text. InferSent embedding model, which is constructed using NLI data and BiLSTM encoders, treats sentence as a single entity and performs more general projection process. But the higher dimensionality (required to make accurate projections) makes it harder to use several classification algorithms. ● GloVe embeddings pretrained on a Common Crawl corpus show better results than Twitter specific embeddings. Sentence-level embeddings, pretrained on non-specific natural language inference data, also show superior results. It seems reasonable that crisis-related lexicon differs from common Twitter lexicon and tends to be closer to common lexicon. However, we should note that there is a lack of publicly available Twitter data for training. GloVe Twitter corpus contains only 27 billion words, which is much less compared to Common Crawl corpus size of 840 billion words. ● All neural network models have lower standard deviation of F1-score compared to other machine learning algorithms (except GBDT). Therefore, the quality of neural networks could be much stable on unseen data and less sensitive to the context. ● Our best classifier (CNN for text classification + fastText, trained on our dataset) outperforms models presented in the related work [40, 41, 42]. 5.2
Dataset and Pre-processing We crawled 60k Twitter messages from April 1, 2018 to April 12, 2018 using the focused crawler presented in [11]. With the help of CNN neural network, we filtered out messages that are not related to emergency events, which reduced the number of tweets in the dataset to 5,200. The remaining tweets were analyzed with the natural language processing pipeline and with the event discovery method. After that, we also crawled Facebook posts for each extracted event. Using the developed method, we filtered out posts that were considered irrelevant to events extracted from Twitter. After filtering, 1k Facebook posts left. In our experiments, we applied grid search to tune weights of the regularizers for topic models. A criterion for the search was a weighted sum of model perplexity, model’s matrices sparsity and model’s pointwise mutual information.
Results and Discussion Since the experiments were conducted on open data, we estimated only precision of models. The results are presented in Table 2. The experiment shows that the proposed approach outperforms baseline LDA models. This confirms the importance of using information about the locations in the framework. One can note relatively low precision for the events matching. We believe this is due to substantial lag of time between the message crawling and the event matching experiments. Thus, true event-related posts may be treated by Facebook’s search as less actual than others. We considered several problems related to monitoring of social networks: detection of messages related to emergencies, extraction of novel events, and matching events reflected in different text sources. For detection of emergency-related messages, we use CNN and word embeddings. For extraction of novel events and matching them across different sources, we propose a multimodal topic modelling enriched with spatial information and Jensen–Shannon divergence.
We investigated the performance of different algorithms and embeddings for emergency-related message detection on CrisisLexT6 dataset and found that the best solution is given by CNN with fastText embeddings. We also compared the proposed multimodal topic model and the LDA baseline. The experimental results are promising and show that the proposed framework could be useful for monitoring emergency events via messages in social media.
In the future work, we are going to address the problem of emergency event locating and create visualization tools for presenting them on a geographic map.
Acknowledgments. The project is supported by the Russian Foundation for Basic Research, project numbers: 15-29-06082, 15-29-06045 “ofi_m”.