Microblogs are increasingly gaining attention as an important information source in emergency management. Nevertheless, it is still di cult to reuse this information source during emergency situations, because of the sheer amount of unstructured data. Especially for detecting small scale events like car crashes, there are only small bits of information, thus complicating the detection of relevant information. We present a solution for a real-time identi cation of small scale incidents using microblogs, thereby allowing to increase the situational awareness by harvesting additional information about incidents. Our approach is a machine learning algorithm combining text classi cation and semantic enrichment of microblogs. An evaluation based shows that our solution enables the identi cation of small scale incidents with an accuracy of 89% as well as the detection of all incidents published in real-time Linked Open Government Data.
Social media platforms are widely used for sharing information about incidents.
Ushahidi, a social platform used for crowd-based ltering of information [
Current approaches for using social media in emergency management largely focus on large scale incidents like earthquakes. Large-scale incidents are characterized by a large number of social media messages as well as a wide geographic and/or temporal coverage. In contrast, small-scale incidents, such as car crashes or res, usually have a small number of social media messages and only narrow geographic and temporal coverage. This imposes certain challenges on approaches for detecting small scale incidents: large incidents, like earthquakes, with thousands of social media postings, are much easier to detect than smaller incidents with only a dozen of postings, and further processing steps, such as the extraction of factual information, can work on a larger set of texts. For large scale incidents, one can optimize systems for precision, whereas recall is not problematic due to the sheer amount of information on the incident. In contrast, detecting small scale incidents imposes much stricter demands on both precision and recall.
The approach discussed in this paper combines information from the social
and the semantic web. While the social web consists of text, images, and videos,
all of which cannot be processed by intelligent agents easily, the Linked Open
Data (LOD) cloud contains semantically annotated, formally captured
information. Linked Open Data can be used to enhance Web 2.0 contents by semantically
enriching it, as shown, e.g., in [
This paper contributes an approach that leverages information provided in the social web for detection of small scale incidents. The proposed method consists of two steps: (1) automatic classi cation of user-generated content related to small scale incidents, and (2) pre ltering of irrelevant content, based on spatial, temporal, and thematic aspects.
The rest of this paper is structured as follows: In section 2, we review related approaches. Section 3 provides a detailed description of our process, followed by an evaluation in section 4. In section 5, we show an example application. We conclude in Section 6 with a short summary and an outlook on future work. 2
Event and incident detection in social media gained increased attention the last
years. In this case, various machine learning approaches have been proposed for
detecting large scale incidents in microblogs, e.g., in [
All those approaches focus on large scale incidents. On the other side, only
few state-of-the-art approaches focus on the detection of small scale incidents
in microblogs. Agarwal et al. [
Twitcident [
Preprocessing • Pre-filtering • Removing stopwords • Correcting spelling errors • Slang replacement • POS filtering • Temporal mention replacement • Spatial mention replacement
Feature Extraction • Word n-grams • Char n-grams • TF-IDF • Syntactic Features • Spatial/Temporal Features • FeGeLOD Features
Incident
Detection
• Trained classifier for
incident types
• Temporal and spatial
filtering
processed by the semantic enrichment module which includes NER using
DBpedia Spotlight [
Wanichayapong et al. [
Li et al. [
In summary, only few of the mentioned approaches make use of semantic web technologies for detecting small scale incidents. Furthermore, all of the mentioned approaches do not compare with events published in governmental data sources, thus the detection of incidents in real-time remains unevaluated. 3
the preprocessed tweets, several features are extracted and used for classi cation. Second, for real-time incident detection, new tweets are rst ltered based on spatial and temporal ltering. Then the classi er is applied to detect incident related information. For optimizing our classi er, we evaluate the results on incident reports published in Linked Open Government Data. As a result, the optimized pipeline can be used to present valuable information for decision makers in emergency management systems. 3.1
Crawling and Preprocessing We continuously collect tweets using the Twitter Search API1. In this case, we restrict our search to English tweets of certain cities. Every collected tweet is then preprocessed. First, we remove all retweets as these are just duplicates of other tweets and do not provide additional information. Second, @-mentions in the tweet message are removed as we assume that they are not relevant for detection of incident related tweets. Third, very frequent words like stop words are removed as they are not valuable as features for a machine learning algorithm. Fourth, abbreviations are resolved using a dictionary compiled from www.noslang.com. Furthermore, as tweets contain spelling errors, we apply the Google Spellchecking API2 to identify and replace them if possible.
We use our temporal detection approach (see below) to detect temporal expressions and replace them with two annotations @DATE and @TIME, so we prevent over tting the classi cation model for temporal values from the training dataset. Likewise, we use our spatial detection approach (see below) to detect place and location mentions and replace them with two annotations @LOC and @PLC.
Before extracting features, we normalize the words using the Stanford lemmatization function3. Furthermore, we apply the Stanford POS tagger4. This enables us to lter out some word categories, which are not useful for our approach. E.g., during our evaluation we found out that using only nouns and proper nouns for classi cation improve the accuracy of the classi cation (cf. Section 4.3) Feature Extraction After nishing the initial preprocessing steps, we extract several features from the tweets that will be used for training a classi er: Word unigram extraction: A tweet is represented as a set of words. We use two approaches: a vector with the frequency of words, and a vector with the occurrence of words (as binary values).
Character n-grams: A string of three respective four consecutive characters in a tweet message is used as a feature. For example, if a tweet is: \Today is so hot. I feel tired" then the following trigrams are extracted: \tod", \oda", 1 https://dev.twitter.com/docs/api/1.1/get/search/tweets 2 https://code.google.com/p/google-api-spelling-java/ 3 http://nlp.stanford.edu/software/corenlp.shtml 4 http://nlp.stanford.edu/software/tagger.shtml \day", \ay ", \y I", etc.. To construct the trigram respective fourgram list, all the special characters, which are not a letter, a space character, or a number, are removed.
TF-IDF: For every document we calculate an accumulated tf-idf score [
Linked Open Data features: FeGeLOD [
Classi cation The di erent features are combined and evaluated using three
classi ers. For classi cation, the machine learning library Weka [
For real-time incident detection, we rst apply spatial and temporal ltering before applying the classi er.
Temporal Extraction Using the creation date of a tweet is not always su cient for detecting events, as people also report on incidents that occurred in the past or events that will happen in the future. During our evaluations we found out, that around 18% of all incident related tweets contain temporal information. Thus, a mechanism can be applied to lter out tweets that are not temporally related to a speci c event.
For identifying what the temporal relation of a tweet is, we adapted the
HeidelTime [
As discussed above, we also use our adaptation to replace time mentions in microblogs with the annotations @DATE and @TIME to use temporal mentions as additional features. Furthermore, compared to other approaches we are able to retrieve precise temporal information about a small scale incident, which enables the real-time detection of current incidents.
Spatial Extraction Besides a temporal ltering, a spatial ltering is also applied. As only 1% of all tweets retrieved from the Twitter Search API are geotagged, location mentions in tweet messages or the user's pro le information have to be identi ed.
For location extraction, we use a threefold approach. First, location mentions are identi ed using Stanford NER5. As we are only interested in recognizing location mentions, which includes streets, highways, landmarks, blocks, or zones, we retrained the Stanford NER model on a set of tweets, using a set of 1250 manually labeled tweets. We use only two classes for named entities: LOCATION and PLACE. All location mentions, for which we can extract accurate geo coordinates, such as cities, streets and landmarks, are labeled with LOCATION. The words of the tweets that are used as to abstractly describe where the event took place, such as \home", \o ce", \school" etc., are labeled with PLACE. The customized NER model has precision of 95.5% and 91.29% recall. As discussed above, we also use our adaptation to annotate the text so that a spatial mention can be used as an additional feature. The following tweet may be the result of this approach: \Several people are injured in car crash on <LOC>5th Ave</LOC> <LOC>Seattle</LOC>"
Second, to relate the location mention to a point where the event happened,
we geocode the location strings. In this case, we create a set of word unigrams,
bigrams, and trigrams. These are sent to the geographical database GeoNames6
to identify city names in each of the n-grams and to extract geocoordinates. As
city names are ambiguous around the world, we choose the longest n-gram as
the most probable city. If there is no city mention in the tweet, we try to extract
the city from the location eld in the user pro le (see [
Third, for ne-grained geolocalization, on street or building level, we are using the MapQuest Nominatim API7, which is based on OpenStreetMap data. Using this approach we are able to extract precise location information for 87% of the tweets. Compared to other approaches, which rely on city level precision, we are able to precisely geolocalize small scale events on street level. Classi cation After temporal and spatial ltering, we apply our classi er to identify incident related tweets. For further re nement and the veri cation that our approach enables the real-time detection of tweets, we compare our results with the o cial incident information published in Linked Open Government Data like the data that can be retrieved from data.seattle.gov. Finally, the detected relevant information can be presented to decision makers in incident management systems (cf. Section 5). 4
We conduct an evaluation of our method on the publicly available Twitter feed. First, we evaluate the performance of the FeGeLOD features. Second, we measure the performance using all features and third, we compare our results to real-time incident reports. 4.1
Training Dataset For building a training dataset, we collected 6 million public tweets using the Twitter Search API from November 19th, 2012 to December 19th, 2012 in a 15km radius around the city centers of Seattle, WA and Memphis, TN. For labeling the tweets, we rst extracted tweets containing incident related keywords. We retrieved all incident types using the \Seattle Real Time Fire 911 Calls" dataset from seattle.data.gov and de ned one general keyword set with keywords that are used in all types of incidents like \incident", \injury", \police" etc. For each incident type we further identi ed speci c keywords, e.g., for the incident type \Motor Vehicle Accident Freeway" we use the keywords \vehicle", \accident", and \road". Based on these words, we use WordNet8 to extend this set by adding the direct hyponyms. For instance, the keyword \accident" was extended with \collision", \crash", \wreck", \injury", \fatal accident", and \casualty".
For building our training set for identifying car crashes, we used the general and the speci c keyword set for the incident types \Motor Vehicle Accident", \Motor Vehicle Accident Freeway", \Car Fire", and \Car Fire Freeway" to extract tweets that might be related to car crashes. We randomly selected 10k tweets from this set to manually label the tweets in two classes \car accident related" and \not car incident related". The tweets were labeled by scienti c members of our departments. The nal training set consists of 993 car accident related and 993 not car accident related tweets.
Test Dataset To show that the resulting model using the training dataset is not over tted to the events in the period when the training data was collected, we collected an additional 1.5 million tweets in the period from February 1st, 2013 to February 7th, 2013 from the same cities. We also used the keyword extraction approach and manually labeled test dataset, resulting in 320 car accident related tweets and 320 not car accident related tweets.
Socrata Dataset For evaluating our approach with real-time Linked Open Government Data, we use the \Seattle Real Time Fire 911 Calls" dataset from seattle.data.gov. In the period from February 5th, 2013 to February 7th, 2013, we collected information about 15 incidents related to car crashes and 830 related to other incident types. 4.2
Our classi cation results are calculated using strati ed 10-fold cross validation on the training set, and evaluating a model trained on the training dataset on the test dataset. To measure the performance of the classi cation approaches, we report the following metrics: { Accuracy (Acc): Number of the correctly classi ed tweets divided by total number of tweets. { Averaged Precision (Prec): Calculated based on the Precision of each class (how many of our predictions for a class are correct). { Averaged Recall (Rec): Calculated based on the Recall of each class (how many tweets of a class are correctly classi ed as this class).
{ F-Measure (F): Weighted average of the precision and recall. 4.3
Evaluation of FeGeLOD Features We used DBPedia Spotlight to detect named entities in the tweet messages9. These annotations were used to generate additional features with FeGeLOD, as discussed above. Table 1 shows the classi cation accuracy achieved using only features generated with FeGeLOD from the training dataset. We used FeGeLOD to create three models with different features. The rst one contains only types of the extracted entities, the second one contains only Wikipedia categories of the extracted entities, and the third one contains both categories and types of the extracted entities from the tweets. The best results with accuracy of 67.1% are achieved when using categories and types. Furthermore, analyzing the results from JRip, we get rules using categories as \Accidents", \Injuries", or \Road infrastructure" and Types as \Road104096066" or \AdministrativeArea". This shows that the features generated by FeGeLOD are actually meaningful. 9 The parameters used were: Con dence=0.2; Contextual score= 0.9; Support = 20; Disambiguator = Document; Spotter=LingPipeSpotter; Only best candidate Evaluation of all features In order to evaluate which machine learning features contribute the most to the accuracy of the classi er, we built di erent classi cation models for all the combinations of the features that we described in Section 3.1. We rst evaluated the models on the training dataset, then we reevaluated the models on the test dataset. Table 2 shows the best classi cation results achieved using di erent combinations of machine learning features.
The tests showed that using word n-grams without POS ltering, TF-IDF accumulate score, and syntactic features provide the best classi cation results for the training dataset. But re-evaluating the same model on the test set that contains data from a di erent time period, the results dropped signi cantly. That leads to conclusion that the model is over tted to the training dataset. Furthermore, the model contains a large number of features and requires a lot of processing performance to be used for predictions in real-time.
Adding the features generated by FeGeLOD did not a ect the classi cation accuracy on the training dataset, but improved the classi cation accuracy on the test dataset. This shows that using semantic features generated from LOD helps to prevent over tting. Additionally, we used POS ltering to lter out some word categories from the tweet to see which word categories contribute to the classi cation performance. The tests showed that using only nouns and proper nouns when generating the word n-grams improve the results on the test set. This approach signi cantly reduced the number of attributes in the model, making it applicable for real-time predictions.
our system can detect compared to governmental emergency systems, we evaluated our predictions with the data from the Socrata test set. We correlated each of the 15 car accidents with the incidents from our system, if a spatial (150m) and temporal (+/-20min) matching applies. Using this approach we were able to detect all of the Socrata incidents with our approach. As the average number of tweets identi ed by our system for each car accident was around ten (with a minimum of only three tweets for one accident), this shows that our approach is capable of detecting incidents with only very few social media posts. Performance For our experiments, we have crawled the Twitter API as discussed above. In the scenario using data from Seattle and Memphis, there were around 100 Tweets per minute. Processing and classifying a bulk of 500 tweets using our trained SVM takes around seven seconds, which is about 14 milliseconds per Tweet. 5
In [
This paper contributes an approach that leverages information provided in microblogs for detection of small scale incidents. We showed how machine learning and semantic web technologies can be combined to identify incident related microblogs. With 89% detection accuracy, we outperform state-of-the-art approaches. Furthermore, our approach is able to precisely localize microblogs in space and time, thus, enabling the real-time detection of incidents.
With the presented approach, we are able to detect valuable information during crisis situations in the huge amount of information published in microblogs. In this case, additional and previously unknown information can be retrieved that could contribute to enhance situational awareness for decision making in daily crisis management. In the future, we aim at re ning our approach, e.g., to use more sophisticated NLP techniques, exploring the capability of our approach to detect other types of events, as well as including larger sets of open government data in the evaluation.
This work has been partly funded by the German Federal Ministry for Education and Research (BMBF, 13N10712, 01jS12024), by the German Science Foundation (DFG, FU 580/2), and by EIT ICT Labs under the activities 12113 Emergent Social Mobility and 13064 CityCrowdSource of the Business Plan 2012.