From its start, the so-called Islamic State of Iraq and the Levant (ISIL/ISIS) has been successfully exploiting social media networks, most notoriously Twitter, to promote its propaganda and recruit new members, resulting in thousands of social media users adopting pro-ISIS stance every year. Automatic identification of pro-ISIS users on social media has, thus, become the centre of interest for various governmental and research organisations. In this paper we propose a semantic-based approach for radicalisation detection on Twitter. Unlike most previous works, which mainly rely on the lexical and contextual representation of the content published by Twitter users, our approach extracts and makes use of the underlying semantics of words exhibited by these users to identify their pro/anti-ISIS stances. Our results show that classifiers trained from words' semantics outperform those trained from lexical and network features by 2% on average F1-measure.
The so-called Islamic State of Iraq and the Levant (ISIL/ISIS) is one of the leading terrorists organisation on the use of social media to share their propaganda, raise money and radicalise and recruit individuals. According to a 2015 U.S government report3 this organisation has lured more than 25,000 foreigners to fight in Syria and Iraq, including 4,500 from Europe and North America.
Aiming to hinder ISIS recruiting efforts via social media, researchers, governments
and organisations are actively working on identifying ISIS-linked or ISIS-supporting
social media accounts. Current research works that have aimed to analyse radicalisation
and pro-ISIS stances of social media users mainly rely on features extracted from the
lexical and the contextual representation of words [
TaskForceFinalReport.pdf aforementioned limitation constitutes a problem when trying to discriminate the stance expressed by users in social media. We therefore hypothesise that, by exploiting the latent semantics of words expressed in tweets, we could identify additional pro-ISIS and anti-ISIS signals that will complement and enhance the ones extracted by previous approaches.
Starting from this position, this paper investigates the use of ontologies and knowledge bases to support a conceptual-based analysis of tweets content. Entities are extracted from the tweets of users’ timelines (e.g. “ISIS”, “Syria”, “United Nations”) and expanded with their corresponding semantic concepts (e.g. “Jihadist Group”, “Country”, “Organisation”), by using ontologies like DBpedia. The extracted conceptual semantics of words are then used as features (so-called semantic features in our work) for detecting the radicalisation stances of users on Twitter.
The effectiveness of semantic features to identify pro-ISIS and anti-ISIS stances is compared against two baseline features, particularly unigram features and network features. This comparison is performed by creating classifiers, based on the different sets of features, from a training dataset of 1,132 European Twitter users equally divided in pro-ISIS and anti-ISIS. Our results show how classifiers trained with semantic features outperform the baselines by 2% on average F1-measure, showing a positive impact on the use of semantic information to identify pro and anti ISIS stances. 2
Radicalisation detection of Twitter users can be considered as a text classification
problem where features extracted from the users’ timelines are used to train and build
radicalisation classifiers using machine learning methods. In this work we use a dataset
of 1,132 European Twitter users, equally divided into pro-ISIS and anti-ISIS, along with
their timelines. Users in this dataset are collected and labelled with their radicalised
stance in our previous work [
Note that we perform a feature selection process on all the feature sets to reduce the
size of the classifiers’ feature space. To this end, we use Information Gain (IG) [
Results in all experiments are computed using 10-fold cross validation over 10 runs of
different random splits of the data to test their significance. Statistical significance is done
using Wilcoxon signed-rank test [
Table 3 shows the results of our stance classification using Unigrams, Network, and Semantic features, applied over the 1,132 users in our dataset. The table reports three sets of precision (P), recall (R), and F1-measure (F1), one for anti-ISIS stance identification, one for pro-ISIS stance identification, and the third shows the average of the two. The table also reports the total number of features used for classification under each feature 5 The list of ontologies and knowledge bases used by AlchemyApi is listed under http://www.alchemyapi.com/api/entity/textc.html set. From the results presented in Table 3, we notice that semantic features have the highest impact on the classification performance among all other features. Specifically, semantic features outperform unigrams features by 2.7% and 2.8% in accuracy and average F1 respectively. Also, semantic features improve classification performance by 2.5% in accuracy and by 1.18% in F1 in comparison with the network features. Overall, semantic features increase the classification performance by 2% in F1 in comparison with the average performance of all the baseline features (F1 = 90.7%).
No. of Features
P anti-ISIS
R
F 1
P pro-ISIS
R
F 1
P
Average
R
F 1 UNIGRAMS 41,200 0.885 0.92 0.902 0.917 0.88 0.898 0.901 0.9 0.9 NETWORK 25,532 0.887 0.952 0.918 0.948 0.878 0.912 0.917 0.915 0.915 SEMANTICS 8,429 0.91 0.945 0.927 0.943 0.906 0.924 0.926 0.926 0.926 Table 3: Classification performance of the three feature sets with IG feature selection. The values highlighted in grey correspond to the best results obtained for each feature. Results in average P, R and F1 are statistically significant with < 0:001.
The above results show the effectiveness of using semantic features for radicalisation classification of users on Twitter. 5
In this paper we proposed the use of the conceptual semantics of words for detecting pro-ISIS and anti-ISIS stances of users on social media. We used Twitter as case study of social media platforms, and investigated how named-entities in tweets can be extracted and used, together with their corresponding semantic concepts, as features to train machine learning classifiers for stance detection of Twitter users.
We experimented with semantic features on a Twitter dataset of 1132 pro-ISIS and anti-ISIS users and compared the performance of a NB classifier trained from semantic features against classifiers trained from unigrams, and network features. Results showed that using the semantic features in radicalisation classification improves performance by 2% in F1 over the average performance of all baselines.
This work was supported by the EU-FP7 project SENSE4US (grant no. 611242).