The passage from a read-only to a read-write Web gave people the possibility to freely interact, share and collaborate through social networks, online communities, blogs, wikis and other online collaborative media. The democracy of the Web is what made it so popular in the past decades but such a high degree of freedom of expression also gave birth to negative side e ects { the so called `dark side' of the Web. An example of this is trolling i.e. the exploitation of the anonymity of the Web to post in ammatory and outrageous messages directed to one speci c person or community to provoke them into a desired emotional response. Online community masters usually warn users against trolls with messages such as DNFTT (Do Not Feed The Trolls) but so far this has not been enough to stop trolls trolling. The aim of this work is to use Sentic Computing, a new paradigm for the a ective analysis of natural language text, to detect trolls and hence prevent web-users from being emotionally hurt by malicious posts.
In Internet slang, a troll is someone who posts in ammatory, extraneous, or o
topic messages in an online community, such as an online discussion forum, chat
room, or blog, with the primary intent of provoking other users into a desired
emotional response or of otherwise disrupting normal on-topic discussion [
The amount of social data on the Web is on an in nite uphill and online social networking is becoming one of the most prevalent means of expression worldwide. Websites like Twitter, Youtube and Blogger are providing a tunnel to link di erent parts of the world and also di erent classes of global society.
The ipside of the coin, on the other hand, is rather dark, fractious and bizarre. Social web is inherently democratic and user anonymity is gratuitous in this space. Be it real world or virtual social web, existence of malicious faction among inhabitants and users is inevitable.
In social web context, emotional attacks on a person or a group through malicious and vulgar comments in order to provoke response are referred to as `trolling' and the generator is called `a troll'. The term was rst used in early 1990 and since then a lot of concern has been raised to contain or curb trolls.
This work proposes a technique based on Sentic Computing [
The structure of the paper is the following: Section 2 argues about the phenomenon of internet trolling, Section 3 presents the state of the art of malicious post detection, Section 4 and Section 5 explain in detail the techniques used within this work, Section 6 illustrates the overall process for ltering trolls, Section 7 demostrates the potential of such process through an evaluation study, and Section 8 comprises concluding remarks and a description of future work. 2
Trolling is a method of shing where some baited shing lines are drawn through the water, usually from a slow-moving boat, with the purpose of hooking unwary sh. An online troll does pretty much the same.
The trend of trolling, where anonymous online users bombard victims with o ensive messages or abuse, appears to have spread a lot recently and it is alarming most of the biggest social networking sites since, in extreme cases such as abuse, has led some teenagers to commit suicide. These attacks usually address not only individuals but also entire communities. For example, reports have claimed that a growing number of Facebook tribute pages had been targeted, including those in memory of the Cumbria shootings victims and soldiers who died in Afghanistan.
At present users cannot do much rather than manually delete abusive messages. Current anti-trolling methods, in fact, mainly consist in identi ng additional accounts that use the same IP address and blocking fake accounts based on name and anomalous site activity e.g. users who send lots of messages to non-friends or whose friend requests are rejected at a high rate.
In July 2010 Facebook launched an application that gives users a direct link
to advice, help and the ability to report cyber problems to the Child Exploitation
and Online Protection Centre (CEOP) [
For these reasons, we need systems able to automatically analyze semantics and sentics, i.e. cognitive and a ective information, associated to natural language in order to lter out inopportune messages and, hence, stop users from `feeling' the trolls.
A prior analysis of the trustworthiness of statements published on the Web has
been presented by Rowe and Butters [
Existing approaches in these elds, in particular, can be grouped into three
main categories: keyword spotting [
The problem with these approaches is that they mainly rely on parts of text in which web, email and opinion spam is explicitly expressed through spam links, commercial terms or abusive words. But, more generally, spam manifests implicitly through context and domain dependent concepts, which makes keywordbased approaches extremely ine ective.
To overcome this problem we need to use natural language processing (NLP) techniques that rely on semantics rather than syntactics. Within this work, in particular, we exploit two Sentic Computing tools to extract semantics and sentics from web posts and, eventually, process the results in order to detect and lter trolls. 4
Sentic Computing is a new opinion mining and sentiment analysis paradigm which exploits AI and Semantic Web techniques to better recognize, interpret and process opinions and sentiments in natural language text.
The term Sentic Computing derives from the Latin `sentire' (the root of words such as sentiment and sensation) and `sense' (intended as common sense) and concerns a kind of computing that relates to, arises from and in uences opinions and sentiments in natural language text.
In Sentic Computing the analysis of text is not based on statistical learning
models but rather on common sense reasoning tools [
Within this work, in particular, we exploit the combination of two Sentic Computing tools for the extraction of semantics and sentics from web posts i.e. a multi-dimensional vector space of common sense and a ective knowledge (Section 4.1) coupled with a novel emotion categorization model born from the idea that our mind consists of four independent emotional spheres, whose di erent levels of activation make up the total emotional state of the mind (Section 4.2). 4.1
A ectiveSpace [
Therefore, in A ectNet, each concept is represented by a vector in the space of possible features whose values are positive for features that produce an assertion of positive valence (e.g. `a penguin is a bird'), negative for features that produce an assertion of negative valence (e.g. `a penguin cannot y') and zero when nothing is known about the assertion. The degree of similarity between two concepts, then, is the dot product between their rows in A ectNet. The value of such a dot product increases whenever two concepts are described with the same feature and decreases when they are described by features that are negations of each other.
When performed on A ectNet, however, these dot products have very high
dimensionality (as many dimensions as there are features) and are di cult to
work with. In order to approximate these dot products in a useful way, we project
all of the concepts from the space of features into a space with many fewer
dimensions i.e. we reduce the dimensionality of A ectNet by means of principal
component analysis (PCA). In particular, we perform truncated singular value
decomposition (TSVD) [
In particular, we choose to discard all but the rst 100 principal components and hence obtain A ectiveSpace (Fig. 1), a 100-dimensional space in which di erent vectors represent di erent ways of making binary distinctions among concepts and emotions. In A ectiveSpace common sense and a ective knowledge are in fact combined, not just concomitant, i.e. everyday life concepts like `have breakfast', `meet people' or `watch tv' are linked to a hierarchy of a ective domain labels.
By exploiting the information sharing property of TSVD, concepts with the same a ective valence are likely to have similar features i.e. concepts concerning the same opinion tend to fall near each other in the vector space. Concepts and emotions are represented by vectors of 100 coordinates: these coordinates can be seen as describing concepts in terms of `eigenmoods' that form the axes of A ectiveSpace i.e. the basis e0,...,e99 of the vector space. For example, the most signi cant eigenmood, e0, represents concepts with positive a ective valence. That is, the larger a concept's component in the e0 direction is, the more a ectively positive it is likely to be. Consequently concepts with negative e0 components have negative a ective valence. 4.2
This model is a variant of Plutchik's emotion categorization [
The Hourglass of Emotions (Fig. 2) is speci cally designed to recognize, understand and express emotions in the context of human-computer interaction (HCI). In the model, in fact, a ective states are not classi ed, as often happens in the eld of emotion analysis, into basic emotional categories, but rather into four concomitant but independent dimensions in order to understand how much respectively: 1. the user is happy with the service provided (Pleasantness) 2. the user is interested in the information supplied (Attention) 3. the user is comfortable with the interface (Sensitivity) 4. the user is disposed to use the application (Aptitude)
Each a ective dimension is characterized by six levels of activation, called
`sentic levels', which determine the intensity of the expressed/perceived emotion
as a oat 2 [
The main aim of the Troll Detector is to identify malicious contents in natural language text with a certain con dence level. To train the detector, we rst identify the concepts most commonly used by trolls (Section 5.1) and then expand the resulting knowledge base with semantically related concepts (Section 5.2). We nally de ne a method to calculate trollness i.e. the probability for a post to be edited by a troll (Section 5.3). 5.1
The technique we use to identify the concepts commonly used by trolls is called
CF-IOF [
We rst calculate the frequency of a concept ci for a given topic j by counting the occurrences of the concept ci in the set of available j-tagged opinions and divide the result by the sum of occurrences of the same concept in the whole set of opinions concerning j. We then multiply this frequency by the logarithm of the total number of opinions divided by the number of opinions containing the concept ci, that is: (CF -IOF )i = X j
ni;j Pk nk;j log
jOj jfo : ci 2 ogj where ni;j is the number of occurrences of the considered concept ci in the opinions tagged with the topic j, jfo : ci 2 ogj the number of opinions where ci appears and jOj the total number of opinions.
A high weight in CF-IOF is reached by a high concept frequency (in the given opinions) and a low opinion frequency of the concept in the whole collection of opinions. Therefore, thanks to CF-IOF weights, we manage to lter out common concepts and detect relevant concepts that are usually used by trolls to emotionally attack unaware users. 5.2
In order to expand the set of concepts previously obtained by applying CF-IOF,
we use a technique called spectral association [
This operation, an approximation of many steps of spreading activation, transfers the most activation to concepts that are connected to the key concepts by short paths or many di erent paths in common sense knowledge. In particular, we build a matrix C that relates concepts to other concepts, instead of their features, and add up the scores over all relations that relate one concept to another, disregarding direction.
Applying C to a vector containing a single concept spreads that concept's value to its connected concepts. Applying C2 spreads that value to concepts connected by two links (including back to the concept itself). But what we'd really like is to spread the activation through any number of links, with diminishing returns, so perhaps the operator we want is: 1 + C +
C2 2! +
C3 3! + ::: = eC
We can calculate this odd operator, eC , because we can factor C. C is already symmetric, so instead of applying Lanczos' method to CCT and getting the SVD, we can apply it directly to C and get the spectral decomposition C = V V T . As before, we can raise this expression to any power and cancel everything but the power of . Therefore, eC = V e V T . This simple twist on the SVD lets us calculate spreading activation over the whole matrix instantly.
As with the SVD, we can truncate these matrices to k axes and therefore save space while generalizing from similar concepts. We can also rescale the matrix so that activation values have a maximum of 1 and do not tend to collect in highly-connected concepts such as `person', by normalizing the truncated rows of V e =2 to unit vectors, and multiplying that matrix by its transpose to get a rescaled version of V e V T . 5.3
In order to calculate the probability for a post to be edited by a troll, we exploit both the semantics and the sentics associated to it.
For each concept contained in the post, the Troll Detector checks if this
belongs to the set of `troll concepts' calculated through spectral association and
exploits its relative sentic vector to check if it carries malicious a ective charge.
By analyzing a set of 1000 o ensive phrases extracted from Wordnik [
P lsnt(ci) + jAttnt(ci)j
jSnsit(ci)j + Aptit(ci)
9
where 9 is the normalization factor (since the numerator's maximum value is
given by the sentic vectors [
In the formula, Attention and Sensitivity are taken in absolute value since, from the point of view of polarity rather than a ection, all of their sentic values represent positive and negative values respectively (e.g. `anger' is positive in the sense of level of activation of Sensitivity but negative in terms of polarity and `surprise' is negative in the sense of lack of Attention but positive from a polarity point of view).
Hence, the total trollness of a post containing N concepts is de ned as: t = i=1
N 5 X 9 si(ci) + 10 jSnsit(ci)j 9
N jAttnt(ci)j
This information is stored, together with post type and content plus sender and receiver ID, in an interaction database that keeps trace of all the messages and comments interchanged between users within the same social network.
Posts with a high level of trollness (current threshold has been set, using a trial and error approach, to 60%) are labelled as troll posts and, whenever a speci c user addresses more than two troll posts to the same person or community, his/her sender ID is labelled as troll for that particular receiver ID.
All the past troll posts sent to that particular receiver ID by that speci c sender ID are then automatically deleted from the website (but kept in the database with the possibility for the receiver to either visualize them in an apposite troll folder and, in case, restore them). Moreover, any new post with a high level of trollness edited by a user labelled as troll for that speci c receiver is automatically blocked i.e. saved in the interaction database but never displayed in the social networking website. 6
The process for ltering trolls (illustrated in Fig. 3) comprises four main components: a NLP module, which performs a rst skim of the document, a Semantic Parser, whose aim is to extract concepts from the lemmatized text, A ectiveSpace, for the extraction of sentics from the given concepts, and the Troll Detector, whose aim is to detect and eventually block the troll.
The NLP module interprets all the a ective valence indicators usually contained in text such as special punctuation, complete upper-case words, onomatopoeic repetitions, exclamation words, negations, degree adverbs and emoticons, and eventually lemmatizes text.
The Semantic Parser then deconstructs text into concepts and provides, for each of them, the relative frequency, valence and status i.e. the concept's occurrence in the text, its positive or negative connotation, and the degree of intensity with which the concept is expressed.
The A ectiveSpace module projects the retrieved concepts into the vector space, clustered wrt the Hourglass model, and it infers the a ective valence of these, in terms of Pleasantness, Attention, Sensitivity and Aptitude, according to the positions they occupy in the space.
This information, encoded as a sentic vector, is given as input to the Troll Detector which exploits it, together with the semantic information coming directly from the Semantic Parser, to calculate the post's trollness and, eventually, to detect and block the troll (according to the information stored in the interaction database). As an example of Troll Filtering Process output, we can consider a troll post recently addressed to the Indian author Chetan Bhagat: \You can't write, you illiterate douchebag, so quit trying, I say!!!". In this case we have a very high level of Sensitivity (corresponding sentic level `rage') and a negative polarity, which give a high percentage of trollness, as shown below: <Concept: !`write'> <Concept: `illiterate'> <Concept: `douchebag'> <Concept: `quit try'> <Concept: `say'> Semantics: 0.69 Sentics: [0.0, 0.48, 2.7, -1.22] Polarity: -0.38
Trollness: 0.75 7
In order to perform a rst evaluation of our system, we considered a set of 500 tweets (most of which fetched from Wordnik) manually annotated as troll and non-troll posts. We considered true positives those posts with both a positive troll- ag and a trollness 2 [0.6, 1] and those with both a negative troll- ag and a trollness 2 [0, 0.6). The threshold has been set to 60% based on trial and error over a separate dataset of 50 tweets.
Results show that, by using the Troll Filtering Process, in ammatory and outrageous messages can be identi ed with good precision (82%) and decorous recall rate (75%). In particular, the F-measure value (78%) is signi cantly high compared to the corresponding F-measure rates of the baseline methods (53% for keyword spotting, 59% for lexical a nity, 66% for statistical methods).
However, we expect to obtain much better results by evaluating the process at interaction-level rather than just at post-level. In the next future, in fact, we plan to evaluate the Troll Filtering Process by monitoring not just single posts but also users' holistic behaviour within the same social network (i.e. contents and recipients of their interaction) and submit further results elsewhere for publication. 8
As the Web plays a more and more signi cant role in people's social lives, it contains more and more information concerning their opinions and feelings. After the explosion of Web 2.0, a lot of users have been exploiting this trend, together with the anonymity of the Web, to attack speci c people or communities with in ammatory and outrageous messages and, hence, provoke them into a desired emotional response.
For their endish nature, these users have been labelled as trolls. Online community masters have desperately tried to warn users against these mischievous people with messages such as DNFTT (Do Not Feed The Trolls) but so far this has not been enough to stop trolls trolling.
Within this work we exploited Sentic Computing, a new paradigm for the a ective analysis of natural language text, to design a process capable to extract semantics and sentics from web-posts and infer from these the truthfulness of user interaction.
The main aim of the Troll Filtering Process, in fact, is to exploit the cognitive and a ective information associated to natural language text to de ne a level of trollness of each post and, according to this, classify users and prevent the malicious ones from emotionally hurting other people or communities within the same social network.
In the next future, we plan to improve the process by using a much bigger dataset for training the Troll Detector and also to perform an evaluation of the system at interaction-level rather than just at post-level, in order to better understand, and hence prevent, trolls' behaviour.
Eventually, we plan to enhance the system by making most of its functionalities available as web-services in a way that the Troll Filtering Process could be easily embedded in any social networking website and, hence, change the meaning of the popular acronym often displayed in these websites, DNFTT, from a shadowy and often ine ective suggestion to a reassuring and deterrent slogan { Do Not Feel The Trolls.