In Internet slang, a troll is someone who posts inflammatory, extraneous, or offtopic messages in an online community, such as an online discussion forum, chat room, or blog, with the primary intent of provoking other users into a desired emotional response or of otherwise disrupting normal on-topic discussion [1].
The amount of social data on the Web is on an infinite uphill and online social networking is becoming one of the most prevalent means of expression worldwide. Websites like Twitter, Youtube and Blogger are providing a tunnel to link different parts of the world and also different classes of global society.
The flipside of the coin, on the other hand, is rather dark, fractious and bizarre. Social web is inherently democratic and user anonymity is gratuitous in this space. Be it real world or virtual social web, existence of malicious faction among inhabitants and users is inevitable.
In social web context, emotional attacks on a person or a group through malicious and vulgar comments in order to provoke response are referred to as 'trolling' and the generator is called 'a troll'. The term was first used in early 1990 and since then a lot of concern has been raised to contain or curb trolls.
This work proposes a technique based on Sentic Computing [2], a novel paradigm for the affective analysis of natural language text, to automatically detect and check web trolls. We present results that are effective in controlling trolls efficiently. To the best of our knowledge this work has no prior.
The structure of the paper is the following: Section 2 argues about the phenomenon of internet trolling, Section 3 presents the state of the art of malicious post detection, Section 4 and Section 5 explain in detail the techniques used within this work, Section 6 illustrates the overall process for filtering trolls, Section 7 demostrates the potential of such process through an evaluation study, and Section 8 comprises concluding remarks and a description of future work.
Trolling is a method of fishing where some baited fishing lines are drawn through the water, usually from a slow-moving boat, with the purpose of hooking unwary fish. An online troll does pretty much the same.
The trend of trolling, where anonymous online users bombard victims with offensive messages or abuse, appears to have spread a lot recently and it is alarming most of the biggest social networking sites since, in extreme cases such as abuse, has led some teenagers to commit suicide. These attacks usually address not only individuals but also entire communities. For example, reports have claimed that a growing number of Facebook tribute pages had been targeted, including those in memory of the Cumbria shootings victims and soldiers who died in Afghanistan.
At present users cannot do much rather than manually delete abusive messages. Current anti-trolling methods, in fact, mainly consist in identifing additional accounts that use the same IP address and blocking fake accounts based on name and anomalous site activity e.g. users who send lots of messages to non-friends or whose friend requests are rejected at a high rate.
In July 2010 Facebook launched an application that gives users a direct link to advice, help and the ability to report cyber problems to the Child Exploitation and Online Protection Centre (CEOP) [3]. Reporting trouble through a link or a button, however, is a too slow process since social networking websites usually cannot react instantly to these alarms. A button, moreover, does not stop users from being emotionally hurt by trolls and it is more likely to be pushed by people who actually do not need help rather than, for instance, children who are being sexually groomed and do not realize it.
For these reasons, we need systems able to automatically analyze semantics and sentics, i.e. cognitive and affective information, associated to natural language in order to filter out inopportune messages and, hence, stop users from 'feeling' the trolls.
A prior analysis of the trustworthiness of statements published on the Web has been presented by Rowe and Butters [4]. Their approach adopts a contextual trust value determined for the person who asserted a statement as the trustworthiness of the statement itself. This study, however, does not focus on the problem of trolling but rather on defining a contextual accountability for the detection of web, email and opinion spam.
Existing approaches in these fields, in particular, can be grouped into three main categories: keyword spotting [5][6], in which text is classified according to the presence of fairly unambiguous spam words, lexical affinity [7] [8], which assigns arbitrary words a probabilistic affinity for spam content, and statistical methods [9][10], which consist in calculating the valence of keywords, punctuation and word co-occurrence frequencies on the base of a large training corpus.
The problem with these approaches is that they mainly rely on parts of text in which web, email and opinion spam is explicitly expressed through spam links, commercial terms or abusive words. But, more generally, spam manifests implicitly through context and domain dependent concepts, which makes keywordbased approaches extremely ineffective.
To overcome this problem we need to use natural language processing (NLP) techniques that rely on semantics rather than syntactics. Within this work, in particular, we exploit two Sentic Computing tools to extract semantics and sentics from web posts and, eventually, process the results in order to detect and filter trolls.
Sentic Computing is a new opinion mining and sentiment analysis paradigm which exploits AI and Semantic Web techniques to better recognize, interpret and process opinions and sentiments in natural language text.
The term Sentic Computing derives from the Latin 'sentire' (the root of words such as sentiment and sensation) and 'sense' (intended as common sense) and concerns a kind of computing that relates to, arises from and influences opinions and sentiments in natural language text.
In Sentic Computing the analysis of text is not based on statistical learning models but rather on common sense reasoning tools [11] and domain-specific ontologies [12]. Differently from statistical classification, which generally requires large inputs and thus cannot appraise texts with satisfactory granularity, Sentic Computing enables the analysis of documents not only on the page or paragraphlevel but also on the sentence-level.
Within this work, in particular, we exploit the combination of two Sentic Computing tools for the extraction of semantics and sentics from web posts i.e. a multi-dimensional vector space of common sense and affective knowledge (Section 4.1) coupled with a novel emotion categorization model born from the idea that our mind consists of four independent emotional spheres, whose different levels of activation make up the total emotional state of the mind (Section 4.2).
AffectiveSpace [13] is a language visualization system which transforms natural language from a linguistic form into a multi-dimensional space. AffectiveSpace is built by blending ConceptNet [14], a semantic network of common sense knowledge, and WordNet-Affect [15], a linguistic resource for the lexical representation of emotions. This alignment operation yields AffectNet: a new dataset in which common sense and affective knowledge coexist i.e. a matrix 14,301 × 117,365 whose rows are concepts (e.g. 'dog' or 'bake cake'), whose columns are either common sense and affective features (e.g. 'isA-pet' or 'hasEmotion-joy'), and whose values indicate truth values of assertions.
Therefore, in AffectNet, each concept is represented by a vector in the space of possible features whose values are positive for features that produce an assertion of positive valence (e.g. 'a penguin is a bird'), negative for features that produce an assertion of negative valence (e.g. 'a penguin cannot fly') and zero when nothing is known about the assertion. The degree of similarity between two concepts, then, is the dot product between their rows in AffectNet. The value of such a dot product increases whenever two concepts are described with the same feature and decreases when they are described by features that are negations of each other. When performed on AffectNet, however, these dot products have very high dimensionality (as many dimensions as there are features) and are difficult to work with. In order to approximate these dot products in a useful way, we project all of the concepts from the space of features into a space with many fewer dimensions i.e. we reduce the dimensionality of AffectNet by means of principal component analysis (PCA). In particular, we perform truncated singular value decomposition (TSVD) [16] on AffectNet and obtain a new matrix, AffectNet*, which forms a low-rank approximation of the original data. This estimation is based on minimizing the Frobenius norm of the difference between AffectNet and AffectNet* under the constraint rank(AffectNet* ) = k and it represents the best approximation of AffectNet in the least-square sense (for the Eckart-Young theorem [17]).
In particular, we choose to discard all but the first 100 principal components and hence obtain AffectiveSpace (Fig. 1), a 100-dimensional space in which different vectors represent different ways of making binary distinctions among concepts and emotions. In AffectiveSpace common sense and affective knowledge are in fact combined, not just concomitant, i.e. everyday life concepts like 'have breakfast', 'meet people' or 'watch tv' are linked to a hierarchy of affective domain labels.
By exploiting the information sharing property of TSVD, concepts with the same affective valence are likely to have similar features i.e. concepts concerning the same opinion tend to fall near each other in the vector space. Concepts and emotions are represented by vectors of 100 coordinates: these coordinates can be seen as describing concepts in terms of 'eigenmoods' that form the axes of AffectiveSpace i.e. the basis e 0 ,...,e 99 of the vector space. For example, the most significant eigenmood, e 0 , represents concepts with positive affective valence. That is, the larger a concept's component in the e 0 direction is, the more affectively positive it is likely to be. Consequently concepts with negative e 0 components have negative affective valence.
This model is a variant of Plutchik's emotion categorization [18] and constitutes an attempt to emulate Marvin Minsky's conception of emotions. Minsky sees the mind as made of thousands of different resources and believes that our emotional states result from turning some set of these resources on and turning another set of them off [19]. Each such selection changes how we think by changing our brain's activities: the state of anger, for example, appears to select a set of resources that help us react with more speed and strength while also suppressing some other resources that usually make us act prudently.
The Hourglass of Emotions (Fig. 2) is specifically designed to recognize, understand and express emotions in the context of human-computer interaction (HCI). In the model, in fact, affective states are not classified, as often happens in the field of emotion analysis, into basic emotional categories, but rather into four concomitant but independent dimensions in order to understand how much respectively:
1. the user is happy with the service provided (Pleasantness) 2. the user is interested in the information supplied (Attention) 3. the user is comfortable with the interface (Sensitivity) 4. the user is disposed to use the application (Aptitude) Each affective dimension is characterized by six levels of activation, called 'sentic levels', which determine the intensity of the expressed/perceived emotion as a float ∈ [-3,3]. These levels are also labelled as a set of 24 basic emotions (six for each of the affective dimensions) in a way that the model can specify the affective information associated to text both in a dimensional and in a discrete form. The dimensional form, in particular, is called 'sentic vector' and it is a four dimensional vector that can potentially express any human emotion in terms of Pleasantness, Attention, Sensitivity and Aptitude. Some particular sets of sentic vectors have special names as they specify well-known compound emotions. For example the set of sentic vectors with a level of Pleasantness ∈ (1,2] ('joy'), a null Attention, a null Sensitivity and a level of Aptitude ∈ (1,2] ('trust') are called 'love sentic vectors' since they specify the compound emotion of 'love'. The main aim of the Troll Detector is to identify malicious contents in natural language text with a certain confidence level. To train the detector, we first identify the concepts most commonly used by trolls (Section 5.1) and then expand the resulting knowledge base with semantically related concepts (Section 5.2). We finally define a method to calculate trollness i.e. the probability for a post to be edited by a troll (Section 5.3).
The technique we use to identify the concepts commonly used by trolls is called CF-IOF [20] (concept frequency -inverse opinion frequency) and it is an approach similar to TF-IDF weighting which evaluates how important a concept is to a set of opinions concerning the same topic.
We first calculate the frequency of a concept c i for a given topic j by counting the occurrences of the concept c i in the set of available j-tagged opinions and divide the result by the sum of occurrences of the same concept in the whole set of opinions concerning j. We then multiply this frequency by the logarithm of the total number of opinions divided by the number of opinions containing the concept c i , that is:
where n i,j is the number of occurrences of the considered concept c i in the opinions tagged with the topic j, |{o : c i ∈ o}| the number of opinions where c i appears and |O| the total number of opinions.
A high weight in CF-IOF is reached by a high concept frequency (in the given opinions) and a low opinion frequency of the concept in the whole collection of opinions. Therefore, thanks to CF-IOF weights, we manage to filter out common concepts and detect relevant concepts that are usually used by trolls to emotionally attack unaware users.
In order to expand the set of concepts previously obtained by applying CF-IOF, we use a technique called spectral association [21] that involves assigning values, or activations, to 'seed concepts' and applying an operation that spreads their values across the ConceptNet graph.
This operation, an approximation of many steps of spreading activation, transfers the most activation to concepts that are connected to the key concepts by short paths or many different paths in common sense knowledge. In particular, we build a matrix C that relates concepts to other concepts, instead of their features, and add up the scores over all relations that relate one concept to another, disregarding direction.
Applying C to a vector containing a single concept spreads that concept's value to its connected concepts. Applying C 2 spreads that value to concepts connected by two links (including back to the concept itself). But what we'd really like is to spread the activation through any number of links, with diminishing returns, so perhaps the operator we want is:
We can calculate this odd operator, e C , because we can factor C. C is already symmetric, so instead of applying Lanczos' method to CC T and getting the SVD, we can apply it directly to C and get the spectral decomposition C = V ΛV T . As before, we can raise this expression to any power and cancel everything but the power of Λ. Therefore, e C = V e Λ V T . This simple twist on the SVD lets us calculate spreading activation over the whole matrix instantly.
As with the SVD, we can truncate these matrices to k axes and therefore save space while generalizing from similar concepts. We can also rescale the matrix so that activation values have a maximum of 1 and do not tend to collect in highly-connected concepts such as 'person', by normalizing the truncated rows of V e Λ/2 to unit vectors, and multiplying that matrix by its transpose to get a rescaled version of V e Λ V T .
In order to calculate the probability for a post to be edited by a troll, we exploit both the semantics and the sentics associated to it.
For each concept contained in the post, the Troll Detector checks if this belongs to the set of 'troll concepts' calculated through spectral association and exploits its relative sentic vector to check if it carries malicious affective charge. By analyzing a set of 1000 offensive phrases extracted from Wordnik [22], in fact, we found that, statistically, a post is likely to be edited by a troll when its average sentic vector has a high absolute value of Sensitivity and a very low polarity. Hence we defined the trollness t i associated to a concept c i as a float ∈ [0, 1] such that:
where s i (float ∈ [0, 1]) is the semantic similarity of c i wrt any of the CF-IOF seed concepts, p i (float ∈ [−1, 1]) is the polarity associated to the concept c i and 5 is the normalization factor (the maximum value of the numerator in fact is given by a similarity of 1, a Sensitivity of 3 or -3 and a polarity equal to -1). In particular, p i is defined [23] as:
where 9 is the normalization factor (since the numerator's maximum value is given by the sentic vectors [3, ±3, 0, 3] and the minimum by [−3, 0, ±3, −3]).
In the formula, Attention and Sensitivity are taken in absolute value since, from the point of view of polarity rather than affection, all of their sentic values represent positive and negative values respectively (e.g. 'anger' is positive in the sense of level of activation of Sensitivity but negative in terms of polarity and 'surprise' is negative in the sense of lack of Attention but positive from a polarity point of view).
Hence, the total trollness of a post containing N concepts is defined as:
This information is stored, together with post type and content plus sender and receiver ID, in an interaction database that keeps trace of all the messages and comments interchanged between users within the same social network.
Posts with a high level of trollness (current threshold has been set, using a trial and error approach, to 60%) are labelled as troll posts and, whenever a specific user addresses more than two troll posts to the same person or community, his/her sender ID is labelled as troll for that particular receiver ID.
All the past troll posts sent to that particular receiver ID by that specific sender ID are then automatically deleted from the website (but kept in the database with the possibility for the receiver to either visualize them in an apposite troll folder and, in case, restore them). Moreover, any new post with a high level of trollness edited by a user labelled as troll for that specific receiver is automatically blocked i.e. saved in the interaction database but never displayed in the social networking website.
The process for filtering trolls (illustrated in Fig. 3) comprises four main components: a NLP module, which performs a first skim of the document, a Semantic Parser, whose aim is to extract concepts from the lemmatized text, AffectiveSpace, for the extraction of sentics from the given concepts, and the Troll Detector, whose aim is to detect and eventually block the troll.
The NLP module interprets all the affective valence indicators usually contained in text such as special punctuation, complete upper-case words, onomatopoeic repetitions, exclamation words, negations, degree adverbs and emoticons, and eventually lemmatizes text.
The Semantic Parser then deconstructs text into concepts and provides, for each of them, the relative frequency, valence and status i.e. the concept's occurrence in the text, its positive or negative connotation, and the degree of intensity with which the concept is expressed.
The AffectiveSpace module projects the retrieved concepts into the vector space, clustered wrt the Hourglass model, and it infers the affective valence of these, in terms of Pleasantness, Attention, Sensitivity and Aptitude, according to the positions they occupy in the space. This information, encoded as a sentic vector, is given as input to the Troll Detector which exploits it, together with the semantic information coming directly from the Semantic Parser, to calculate the post's trollness and, eventually, to detect and block the troll (according to the information stored in the interaction database). As an example of Troll Filtering Process output, we can consider a troll post recently addressed to the Indian author Chetan Bhagat: "You can't write, you illiterate douchebag, so quit trying, I say!!!". In this case we have a very high level of Sensitivity (corresponding sentic level 'rage') and a negative polarity, which give a high percentage of trollness, as shown below:
<Concept: !'write
In order to perform a first evaluation of our system, we considered a set of 500 tweets (most of which fetched from Wordnik) manually annotated as troll and non-troll posts. We considered true positives those posts with both a positive troll-flag and a trollness ∈ [0.6, 1] and those with both a negative troll-flag and a trollness ∈ [0, 0.6). The threshold has been set to 60% based on trial and error over a separate dataset of 50 tweets.
Results show that, by using the Troll Filtering Process, inflammatory and outrageous messages can be identified with good precision (82%) and decorous recall rate (75%). In particular, the F-measure value (78%) is significantly high compared to the corresponding F-measure rates of the baseline methods (53% for keyword spotting, 59% for lexical affinity, 66% for statistical methods).
However, we expect to obtain much better results by evaluating the process at interaction-level rather than just at post-level. In the next future, in fact, we plan to evaluate the Troll Filtering Process by monitoring not just single posts but also users' holistic behaviour within the same social network (i.e. contents and recipients of their interaction) and submit further results elsewhere for publication.
As the Web plays a more and more significant role in people's social lives, it contains more and more information concerning their opinions and feelings. After the explosion of Web 2.0, a lot of users have been exploiting this trend, together with the anonymity of the Web, to attack specific people or communities with inflammatory and outrageous messages and, hence, provoke them into a desired emotional response.
For their fiendish nature, these users have been labelled as trolls. Online community masters have desperately tried to warn users against these mischievous people with messages such as DNFTT (Do Not Feed The Trolls) but so far this has not been enough to stop trolls trolling.
Within this work we exploited Sentic Computing, a new paradigm for the affective analysis of natural language text, to design a process capable to extract semantics and sentics from web-posts and infer from these the truthfulness of user interaction.
The main aim of the Troll Filtering Process, in fact, is to exploit the cognitive and affective information associated to natural language text to define a level of trollness of each post and, according to this, classify users and prevent the malicious ones from emotionally hurting other people or communities within the same social network.
In the next future, we plan to improve the process by using a much bigger dataset for training the Troll Detector and also to perform an evaluation of the system at interaction-level rather than just at post-level, in order to better understand, and hence prevent, trolls' behaviour.
Eventually, we plan to enhance the system by making most of its functionalities available as web-services in a way that the Troll Filtering Process could be easily embedded in any social networking website and, hence, change the meaning of the popular acronym often displayed in these websites, DNFTT, from a shadowy and often ineffective suggestion to a reassuring and deterrent slogan -Do Not Feel The Trolls.