The rise of social media as an information channel during crisis has become key to community response. However, existing crisis awareness applications, often struggle to identify relevant information among the high volume of data that is generated over social platforms. A wide range of statistical features and machine learning methods have been researched in recent years to automatically classify this information. In this paper we aim to complement previous studies by exploring the use of semantics as additional features to identify relevant crisis information. Our assumption is that entities and concepts tend to have a more consistent correlation with relevant and irrelevant information, and therefore can enhance the discrimination power of classi ers. Our results, so far, show that some classi cation improvements can be obtained when using semantic features, reaching +2.51% when the classi er is applied to a new crisis event (i.e., not in training set).
As per the 2016 World Humanitarian Data and Trends report by UNOCHA,1 there were around 102 million people, from 114 countries, a ected by natural disasters in the year of 2015 alone, causing an estimated damage of $90 billion. During such disasters there is normally a surge of real time content across multiple social media platforms. For example, during the 2011 Japan earthquake, there was a 500% increase in the number of tweets.2 All these messages constitute a critical source of information for relief and search teams, communities and individuals.
However, it is almost impossible to manually absorb and process the sheer
volume of social media reports generated during a crisis, and to e ciently
lter any relevant and actionable information [
Much research explored methods for the classi cation of social media data
into crisis-related or unrelated, based on supervised [
We explore the e ectiveness of semantic features by creating and testing classi ers to identify relevant crisis information, as well as by testing these classi ers with previously unseen information from di erent crisis events. The dataset used in our research is a small subset of CrisisLexT26;3 a library of 205K annotated tweets posted during 26 real crisis events in 2012 and 2013. Our subset consists of a balanced related-unrelated set of 3.2K tweets on 9 crisis events (detailed in Section 3.1). Our results show that using semantic information can indeed help to enhance classi cation results, but only by a small margin. When the classi er is applied to a new crisis event, results show that the use of semantic annotations of concepts and entities in itself is e ective, and the use of semantically expanded concepts (i.e., entities and their hypernyms) further improves over it slightly. However, the use of hypernyms also sometimes introduces generic concepts, such as \person", that appear in both, crisis related and non-crisis related posts, and thus e ects the discrimination power of semantic features.
The contributions of this work can be summarised as follows: { Demonstrating the impact of using a variety of semantic features for identifying crisis-related information from social media posts. { Showing that adding semantic features is especially useful when classifying new crisis events that were not seen during the model training phase. { Testing using annotated data from CrisisLexT26 of 9 real crisis events. { Discussing and re ecting on the potential use of semantics to identify crisisrelevant information.
The rest of the paper is structured as follows. Section 2 summarises the related work on processing social media data for identifying crisis related content. Section 3 describes our approach, including the selected semantic features and how they are used to created various types of classi ers. The experiments are results are reported in Section 4. Section 5 discusses the lessons learned from this work, as well as its limitations and the future lines of work. Conclusions are reported in Section 6.
During a crisis, a very large number of messages are often posted on various
social media platforms. Processing all such messages requires substantial time and
e ort to ensure that crisis related messages are e ciently spotted and handled,
since a good percentage of messages posted about a crisis tends to be
irrelevant and unrelated. Olteanu and colleagues observed that crisis reports could be
classi ed into three main categories: related and informative, related but not
informative, and not related [
To identify crisis related messages from social media data, several works have
proposed the use of supervised [
While semantic models have been developed and used to represent and
capture the information that emerge from crisis events (e.g., MOAC - Management
of a Crisis 4, or HXL - Humanitarian eXchange Language5), few works in the
literature have explored the use of semantics to identify and lter crisis-related
information. In [
Classi cation: Identifying Crisis Related Information Our approach for identifying crisis related information among tweets functions by generating binary classi ers to di erentiate crisis-related from non-related posts. In this section, we explain (i) the dataset used in our experiments, (ii) the two set of features (statistical and semantic) that we use to build the classi ers, and (iii) our classi er selection process. 3.1
To conduct our study we selected the CrisisLexT267 dataset [
We merged those tweets labelled as Not Related and Not Applicable under the class Not Related, obtaining a total of 1539 non crisis-related tweets. We also merged those tweets labelled as Related and Informative and Related but not Informative under the class Related, obtaining a total of 7461 crisis related tweets. In line with common practice, we balanced the dataset to remove classi cation bias towards the bigger class Related, by randomly selected 1667 crisis related tweets. This gives us a balanced and annotated dataset of 3206 of Related and Not Related tweets. 3.2
To generate classi ers able to identify crisis-related posts, we explore two
distinct feature sets, statistical and semantic features. Statistical features have been
widely studied in the literature [
To extract the unigrams from social media posts we make use of the Weka data mining software9, and speci cally its StringToWord functionality, including lower case conversion for all tokens, stemming (using Lovins' algorithm)10, stopword removal, and tf*idf transformation. The total number of unigrams, or vocabulary size, for the complete dataset is 10655. To extract the Part of Speech (POS) tags and the statistical features listed above (top ve), we make use of the Stanford Core NLP software.11 Hashtags are identi ed by the use of the # character, and readability is computed using the Gunning fog index. 3.2.2 Semantic Features (SemF) The semantic feature extraction process is summarised in Figure 3.2.2 and consists of three main steps: (i) semantic annotation, (ii) semantic expansion, and (iii) semantic ltering. Each of these three steps generates a di erent set of semantic features that we explore, individually and in combination, when generating binary classi ers to distinguish crisis-related posts from unrelated ones.
annotation) semantic entities are extracted from the posts by using Babelfy.12
This Name Entity Recogniser (NER) identi es the di erent entities that appear
in the text, disambiguates them, and links them to the BabelNet[
10 http://www.mt-archive.info/MT-1968-Lovins.pdf 11 https://stanfordnlp.github.io/CoreNLP/ 12 http://babelfy.org
Semantic Expansion Features (SemEF): In the second step (semantic expansion) the BabelNet knowledge base is used to extract every direct hypernym (distance-1) of these entities. Our hypothesis for considering hypernyms is that, by introducing upper level concepts, we might be able to better encapsulate the semantics of crisis-related tweets. For example, if the entities reman and policeman appear often in crisis related posts. These entities have a common hypernym, defender. As a result, a post with the entity MP (Military Police), is more likely to also be crisis-related since this entity also has the hypernym defender. The semantic expansion process resulted in an additional 7032 unique concepts.
Semantic Filtering Features (SemFF): When semantically expanding
the initially extracted entities, we could sometimes introduce very generic
concepts with low discrimination power. For example, the hypernym Person appears
in both crisis and non-crisis related posts, and thus does not help the classi ers
to identify crisis-related information. Our ltering process aims to discard such
semantic annotations that might be too generic and hence are likely to reduce
the discrimination power of semantics. Our proposed ltering process is based
on the computation of the depth of a concept in the hierarchy of BabelNet.
To determine the depth of concepts, we query iteratively through the hierarchy
of BabelNet. Abstract concepts, i.e., concepts with a lower depth are therefore
removed. To determine the shortest depth of a concept in the hierarchy of
BabelNet, we used nearly 4 million relations extracted by iteratively querying for
hypernyms and generated a Directed Graph. The node with highest betweenness
centrality (SynSetID `bn:00031027n', which relates to the main sense `Entity')
was determined to be the most abstract concept. The NetworkX13 graph library
for Python was used for this task. We then computed the Shortest path between
the node `Entity' and all the extracted hypernyms. The maximum depth found
was 21, where level 0 is assigned to the concept `Entity'. By performing an
em13 https://networkx.github.io/
pirical analysis of the concepts using Information Gain, we observed that the
most informative concepts are those whose depth is between 3 and 7. Those are
therefore the ones selected as features for classi cation. This ltering process
resulted in 574 concepts ltered out from the semantics across 9 events.
When selecting a classi cation method for the problem at hand we considered
the high dimensionality of features, particularly the high number of unigrams
and semantic features, the limited set of labelled data (3,206 posts) and the
importance of avoiding over- tting. Given the large set of features in comparison
with the number of training examples, we opted for selecting the Support Vector
Machine (SVM) classi cation model [
In this section we describe our experimental set up, and particularly the design of our model selection and testing experiments. We report on the obtained results and later discuss how semantic features can help enhancing the performance of classi ers based on statistical features, and especially when the classi er is applied to cross-crisis scenarios. 4.1
We designed two main experiments where we train and test our classi cation models on (i) all 9 crisis events, and (ii) on 8 events, and retest on the 9th event, i.e., cross-crisis testing. 14 http://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf Radial Basis Function (rbf) kernel or a Polynomial Kernel may cause an over- tting problem,hence we opted for a linearly separable hyperplane. { Crisis Classi cation Model: In our rst experiment we compare the performance of classi ers generated with statistical features vs. classi ers enhanced with semantic features and analyse if the use of semantics does indeed help boosting the performance of binary classi ers when identifying crisis-related posts. We compare the performance of four di erent classi ers generated using the complete dataset (see Section 3.1), and tested using 10-fold cross validation. We use the WEKA software (v.3.8)15 to generate the classi ers.
SF: A classi er generated with statistical features; our baseline.
SF+SemAF: A classi er generated with statistical features, and semantic annotation features.
SF+SemAF+SemEF: A classi er generated with statistical features, semantic annotations, and their hypernyms, i.e., the Semantic Expanded Features.
SF+SemFF: A classi er generated with statistical features, and ltered semantic annotations, along with their hypernyms, i.e., the Semantic Filtered Features { Cross-crisis Classi cation: In our second experiment we retest the classi ers above by applying them to a new crisis data, i.e., on data from a new crisis event that was not part of the training set. For this experiment, we generate the same four classi ers described in the previous task. However, rather than using the complete dataset to generate the model, we use 8 out of the 9 crisis events to generate the model, and then apply the models to the remaining event for validation. We therefore generate 36 di erent classi cation models for this experiment. 4.2
The results of our rst experiment (each model statistically evaluated with 10 iterations of 10-fold cross validation) are presented in Table 1. The table presents F-measure(F) value (from 10-fold cross validation), mean of F-measure (Fmean) of 100 results from 10 iterations, standard deviation in F-measure ( ),and the increment of Fmean over the baseline F=F . Precision and Recall values where equal to F in this experiment, and hence were omitted from table. As we can see in this table, the use of semantic features helps to enhance classi cation results in all cases, but almost negligibly (less than 0.6%). However, the use of annotations alone (SF+SemAF) produces slightly better results than the use of annotations and hypernyms (SF+SemAF+SemEF).
To better understand the impact of semantics in this context, we manually analyse some of the tweets that were misclassi ed by the statistical baseline model, but were correctly classi ed when using semantics (see Table 2)In addition, we perform feature selection using Information Gain (IG) over the generated classi ers to determine which are the most discriminative statistical and semantic features when identifying crisis related posts. 15 http://www.cs.waikato.ac.nz/ml/weka/
When applying IG over the attributes of the baseline classi er, the number of hashtags was the most relevant feature. By manually checking some of the tweets, we observe that Not Related posts tend to either have no hashtags (see as example Post2) or contain many hashtags (see Post1). The number of nouns and pronouns is also a high discriminative feature. As we hypothesized, crisis related posts generally contain more nouns and pronouns mentioning persons, resources or locations relevant to the crisis event. When including semantics, we observe that the hypernyms and annotations are among the highly ranked features, based on IG. Apart from highly ranked statistical features, hypernyms such as `Happening ' and `Event ' (which, in BabelNet, are hypernyms of concepts such as `Incident', `Fire', `Crisis',`Disaster',and `Death'), were among the top 10 attributes (out of almost 800 positive IG attributes).
Post3 was misclassi ed when using only statistical features. Although it contains the relevant term burn, it barely appears in the training data. However, the post is correctly classi ed by SF+SemAF, because the term burn returns the concept Fire as part of its semantic annotation. Post 4 was misclassi ed by SF+SemAF, but correctly classi ed when adding semantic expansion (SF+SemAF+SemEF).The original tweet was annotated with the concept Wildre, which has the hypernym Fire; a feature with high IG and strongly associated with the class crisis-related. Therefore, in this case, the use of hypernyms helped to obtain the additional information needed to correctly categorise the post.
Post 5 was misclassi ed by SF+SemAF+SemEF but correctly classi ed by SF+SemFF. Annotations such as Thanks and Meet semantically expanded to hypernyms such as Virtue, and Desire, which have a very low discrimination power, and hence weakens the classi er. We observe that removing such less informative abstract concepts results in increasing the discriminative power of the remaining, more informative, concepts, such as `Volunteer ' and hypernym (of `donor') `Benefactor '. 4.3
The results of this experiment are reported in Table 3. In this experiment, we compile 9 di erent datasets, where in each dataset 1 out of the 9 crisis events is entirely left out of the training sample used to train and test the classi cation model.16 Each row is named after the the crisis event that was left out of the dataset during its creation (see Section 3.1). The data split for each dataset (train on 8 event/test on 9th event) is presented in the second column of the table. For each of these 9 datasets we created the four di erent classi ers described in Section 4.2. The results of each of these models for the 9 di erent datasets are reported in table along with their values of Precision (P), Recall (R), F1-measure (F) and the increment of F measure over the baseline, F=F .
As we can see, the use of semantics enhances classi cation results in all cases. We observe that SF+SemAF improves the classi cation over the baseline SF, in 6 out of 9 case, with an average of 0.9% increase in F-1 measure. As opposed to our previous experiment (10-fold cross-validation), however, the use of hypernyms makes the model more adaptable to unknown data in 6 out of 9 cases, with an average improvement of 1.94% over the baseline(SF). Semantic expansion (SemEF) improves over the annotation model (SemAF) in 5 out of 9 cases. Also, it is worth noting that ltering out the abstract concepts resulted in an improved performance of SF+SemFF over SF+SemAF+SemEF model ( average of 0.6%), in 7 out of 9 cases. This validates the argument (Sec 3.2.2) that certain concepts tend to appear in both, crisis related and non-related tweets, and 16 Each model was tested on the 8 event dataset it was trained on using 10 fold crossvalidation to ensure its accuracy before applying it to the 9th event data. There accuracy drops around 17% on average when applied to new events. therefore introduce noise rather than helping with the classi cation. Filtering out such concepts enhances the classi cation. SF+SemFF model improves over the baseline by an average of 2.51%. 5
Our ndings show potential in mixing statistical and semantic features for classifying crisis-related and unrelated tweets. The highest, and more worthy, improvement is achieved when using this hybrid model to classify data of a new crisis event that the model was not trained on. This is due to the use of semantic knowledge graphs to expand the vocabulary into semantic concepts and hypernyms, and thus capturing the essence of the tweets and their terms. However, we showed that such a semantic expansion could introduce noise in the form of abstract concepts, which requires ltering to maximise bene t.
An issue we encountered was the unsymmetrical mappings of HypernymHyponym relationship in BabelNet, which e ected the hierarchical expansion of semantics and hierarchy generation. As a future work, we plan to refer to more symmetrically mapped resources, such as WordNet17, and extend to the types and categories of semantics through external knowledge base such as DBpedia18.
One of the limitations of this study is the small size of the dataset (3206 annotations) and type of crisis events (5 di erent types), which we plan to expand in future work. We also need to investigate whether the discriminative features di er across the various type of crisis, and languages. Additionally, we will investigate whether adding semantic features incorrectly classi es some tweets that are correctly classi ed by the statistical approach. 6
This work presents an approach to leverage semantic enrichment for classifying unseen crisis Twitter data. The two approaches of semantic enrichment; annotation and semantic expansion, exhibit an improvement in classi cation performance over the statistical features by 0.9%-2.51%. We have also demonstrated empirically that more abstract concepts are less discriminative, and proposed a method that lters the concepts which are less likely to be discriminative. 17 https://wordnet.princeton.edu/ 18 http://wiki.dbpedia.org/