English. This paper describes the CoLing Lab system for the participation in the constrained run of the EVALITA 2016 SENTIment POLarity Classification Task (Barbieri et al., 2016). The system extends the approach in (Passaro et al., 2014) with emotive features extracted from ItEM (Passaro et al., 2015; Passaro and Lenci, 2016) and FB-NEWS15 (Passaro et al., 2016).
emoticons, slang, specific terminology,
abbreviations, links and hashtags is higher than in other
domains and platforms. Twitter users post messages
from many different media, including their
smartphones, and they “tweet” about a great variety of
topics, unlike what can be observed in other sites,
which appear to be tailored to a specific group of
topics
The paper is organized as follows: Section 2
describes the architecture of the system, as well
as the pre-processing and the features designed
in
The system extends the approach in
The aim of the preprocessing phase is the identification of the linguistic and nonlinguistic elements in the tweets and their annotation.
While the preprocessing of nonlinguistic elements such as links and emoticons is limited to their identification and classification (cf. section 2.2.4), the treatment of the linguistic material required the development of a dedicated rule-based procedure, whose output is a normalized text that is subsequently feed to a pipeline of generalpurpose linguistic annotation tools. The following rules have been applied in the linguistic preprocessing phase:
Emphasis: tokens presenting repeated characters like bastaaaa “stooooop” are replaced by their most probable standardized forms (i.e. basta “stop”); Links and emoticons: they are identified and removed; Punctuation: linguistically irrelevant punctuation marks are removed; Usernames: the users cited in a tweet are identified and normalized by removing the @ symbol and capitalizing the entity name; Hashtags: they are identified and normalized by simply removing the # symbol;
The output of this phase are
linguisticallystandardized tweets, that are subsequently POS
tagged with the Part-Of-Speech tagger described
in
The inventory of features can be organized into six classes. The five classes of features described in this section have been designed in 2014, the sixth class, described in the next section is referred to the emotive and LDA features.
Lexical features represent the occurrence of bad
words or of words that are either highly emotional
or highly polarized. Relevant lemmas were
identified from two in-house built lexicons (cf. below),
and from Sentix
Bad words lexicon: By exploiting an in house built lexicon of common Italian bad words, we reported, for each tweet, the frequency of bad words belonging to a selected list, as well as the total amount of these lemmas.
Sentix: Sentix (Sentiment Italian Lexicon:
Cscore = Intensity P olarity (1) Depending on their Cscore, the selected lemmas have been organized into several groups: strongly positives: 1 Cscore < 0:25 weakly positives:0:25 Cscore < 0:125 neutrals: 0:125 Cscore 0:125 weakly negatives: 0:125 < Cscore highly negatives: 0:25 < Cscore 1 0:25 Since Sentix relies on WordNet sense distinctions, it is not uncommon for a lemma to be associated with more than one hIntensity,Polarityi pair, and consequently to more than one Cscore.
In order to handle this phenomenon, the
lemmas have been splitted into three different
ambiguity classes: Lemmas with only one
entry or whose entries are all associated with
the same Cscore value, are marked as
“Unambiguous” and associated with their Cscore.
Ambiguous cases were treated by inspecting,
for each lemma, the distribution of the
associated Cscores: Lemmas which had a Majority
Vote (MV) were marked as “Inferable” and
associated with the Cscoreof the MV. If there
was no MV, lemmas were marked as
“Ambiguous” and associated with the mean of the
Cscores. To isolate a reliable set of polarized
words, we focused only on the
Unambiguous or Inferable lemmas and selected only the
250 topmost frequent according to the PAIS
corpus
Other Sentix-based features in the ColingLab model are: the number of tokens for each Cscore group, the Cscore of the first token in the tweet, the Cscore of the last token in the tweet and the count of lemmas that are represented in Sentix.
Negation features have been developed to encode the presence of a negation and the morphosyntactic characteristics of its scope.
The inventory of negative lemmas (e.g. “non”)
and patterns (e.g. “non ... mai”) have been
extracted from
In order to characterize the scope of each
negation, we used the dependency parsed tweets
produced by DeSR
This information has been included in the model by counting the number of negation patterns encountered in each tweet, where a negation pattern is composed by the PoS of the negated element plus the number of negative tokens depending from it and, in case it is covered by Sentix, either its Polarity, its Intensity and its Cscores value.
The linguistic annotation produced in the preprocessing phase has been exploited also in the population of the following morphological statistics: (i) number of sentences in the tweet; (ii) number of linguistic tokens; (iii) proportion of content words (nouns, adjectives, verbs and adverbs); (iv) number of tokens for Part-of-Speech.
This group of features has been developed to describe distinctive characteristics of web communication. The group includes: Emoticons: We used the lexicon LexEmo to mark the most common emoticons, such as :-( and :-), marked with their polarity score: 1 (positive), 1 (negative), 0 (neutral).
LexEmo is used both to identify emoticons and to annotate their polarity.
Emoticon-related features are the total amount of emoticons in the tweet, the polarity of each emoticon in sequential order and the polarity of each emoticon in reversed order. For instance, in the tweet :-(quando ci vediamo? mi manchi anche tu! :*:* “:-(when are we going to meet up? I miss you, too :*:*” there are three emoticons, the first of which (:-() is negative while the others are positive (:*; :*).
Accordingly, the classifier has been fed with the information that the polarity of the first emoticon is 1, that of the second emoticon is 1 and the same goes for the third emoticon. At the same way, another group of feature specifies that the polarity of the last emoticon is 1, as it goes for that of the last but one emoticon, while the last but two has a polarity score of 1.
Links: These features contain a shallow classification of links performed using simple regular expressions applied to URLs, to classify them as following: video, images, social and other. We also use as feature the absolute number of links for each tweet. Emphasis: The features report on the number of emphasized tokens presenting repeated characters like bastaaaa, the average number of repeated characters in the tweet, and the cumulative number of repeated characters in the tweet.
Creative Punctuation: Sequences of contiguous punctuation characters, like !!!, !?!?!?!!?!????! or ......., are identified and classified as a sequence of dots, exclamations marks, question marks or mixed. For each tweet, the features correspond to the number of sequences belonging to each group and their average length in characters.
Quotes: The number of quotations in the tweet.
This group of features describes some Twitterspecific characteristics of the target tweets. Topic: This information marks if a tweet has been retrieved via a specific political hashtag or keywords. It is provided by organizers as an attribute of the tweet; Usernames: The number of @username in the tweet; Hashtags: Hashtags play the role of organizing the tweets around a single topic, so that they are useful to be considered in determing their polarity (i.e. a tweet containing hashtags like and #amore “#love” and #felice “#happy” is expected to be positive and a tweet containing hashtags like #ansia “#anxiety” and #stressato “#stressedout” is expected to be negative. This group of features registers the presence of an hashtag belonging to the list of the hashtags with a frequency higher than 1 in the training corpus.
In order to add emotive features to the CoLing Lab
model, we created an emotive lexicon from the
corpus FB-NEWS15
In order to build the distributional profiles of the words, we extracted the list T of the 30,000 most frequent nouns, verbs and adjectives from FBNEWS15. The lemmas in T were subsequently used as target and contexts in a square matrix of co-occurrences extracted within a five word window ( 2 words, centered on the target lemma). In addition, we extended the matrix to the nouns, adjectives and verbs in the corpus of tweets (i. e. lemmas not belonging to T ).
For each hemotion, PoSi pair we built a centroid
vector from the vectors of the seeds belonging to
that emotion and PoS, obtaining in total 24
centroids1. Starting from these spaces, several groups
1Following the configuration in
the emotive Facebook spaces;
number of emotive words and the total number of words in the tweet;
ing a high (greater than 0:4) emotive score for at least one emotion; Tweet emotive score: Score calculated as the ratio between the number of strongly polarized words and the number of the content words in the tweet (Eq. 2). The feature assumes values in the interval [0, 1]. In absence of strongly emotive words, the default value is 0.
E(T weet) = Count(Strongly emotive words) Count(Content words) (2) Maximum values: The maximum emotive value for each emotion (8 features); Quartiles: The features take into account the distribution of the emotive words in the tweet. For each emotion, the list of the emotive words has been ordered according to the emotive scores and divided into quartiles (e.g. the fourth quartile contains the most emotive words and the first quartile the less emotive ones.). Each feature registers the count of the words belonging to the pair hemotion; quartilei (32 features in total); ItEM seeds: Boolean features registering the presence of words belonging to the words used as seeds to build the vector space models. In particular, the features include the top 4 frequent words for each emotion (32 boolean features in total); Distintive words: 32 features corresponding to the top 4 distinctive words for each emotion. The degree of distinctivity of a word for a given emotion is calculated starting from the VSM normalized using Z-scores. In particular, the feature corresponds to the proportion of the emotion hemotionii against the sum of total emotion score [e1, ..., e8]; so that we opted for the more economical setting, i.e. the multiclass one.
Polarity (count): The number of positive and negative words. The polarity of a word is calculated by applying Eq. 3, in which positive emotions are assumed to be JOY and TRUST, and negative emotions are assumed to be DISGUST, FEAR, ANGER and SADNESS.
P olarity(w) = JOY+T2RUST
DISGUST+FEAR+ANGER+SADNESS
4
(3)
Polarity (values): The polarity (calculated using
Eq. 3) of the emotive words in the tweet.
The maximum number of emotive words is
assumed to be 20;
LDA features: This group of features includes 50
features referred to the topic distribution of
the tweet. The LDA model has been built
on the FB-NEWS15 corpus
We used the same paradigm used in
The classification task proposed by the
organizers could be approached either by building
two separate binary classifiers relying of two
different models (one judging the positiveness of
the tweet, the other judging its negativeness),
or by developing a single multiclass classifier
where the possible outcomes are Positive
Polarity (Task POS:1, Task NEG:0), Negative
Polarity (Task POS:0, Task NEG:1), Mixed Polarity
(Task POS:1, Task NEG:1) and No Polarity (Task
POS:0, Task NEG:0). In Evalita 2014
Although this model is not optimal according to the global ranking, if we focus on the recognition of the negative tweets (i.e. the NEG task), it ranks fifth (F1-score), and first if we consider the class 1 of the NEG task (i.e. NEG, F.sc. 1). Such trend is reversed if we consider the POS task, which is the worst performing class of this system.
Task POS POS POS task NEG NEG NEG task GLOBAL
Class 0 1
Due to the great difference in terms of performance between the results obtained by performing a 10 fold cross validation, we suspected that the system was overfitting the training data, so that we performed different feature ablation experiments, in which we included only the lexical information derived from ItEM and FB-NEWS (i.e. we removed the features relying to Sentix, Negation and Hashtags (cf. table 2). The results demonstrate on one hand that significant improvements can be obtained by using lexical information, especially to recognize negative texts. On the other hand the results highlight the overfitting of the submitted model, probably due to the overlapping between Sentix and the emotive features.
The advantage of using only the lexical features derived from ItEM are the following: i) the emotional values of the words can be easily updated; ii) the VSM can be extended to increase the lexical coverage of the resource; iii) the system is “lean” (it can do more with less).
The Coling Lab system presented in 2014