English. This paper studies the relationship between the valence, one of the psycholinguistic variables in the Italian version of ANEW (Montefinese et al., 2014), and emotive scores calculated by exploiting distributional methods (Passaro et al., 2015). We show two methods to infer valence from fine grained emotions and discuss their evaluation.
Recent years have seen a surge in studies
concerning emotional ratings, both in
psycholinguistics and in affective computing. Traditionally, the
three main behavioral dimensions to measure the
emotional value of a word are valence, arousal and
dominance. Warriner et al. (2013) define valence
as the “pleasantness of the stimulus”, usually
ranging from 1 (very unpleasant) to 9 (very pleasant).
The word dead has a low valence rating, whereas
holiday has a higher one. Arousal is the intensity
of the feeling evoked on a scale from “stimulated”
to “unaroused”. A highly stimulating word is
passion. On the contrary, sleep is not arousing.
Finally, dominance is identified with the degree to
which the stimulus makes the reader feel “in
control”
In the domain of Affective Computing, the goal moves from the identification of such variables to the annotation of the texts with the emotions they express and - for Sentiment Analysis - with their degree of positivity and/or negativity.
The aim of this work is to study the relationship
between the most important psycholinguistic
variables and emotive scores calculated by exploiting
distributional methods. In particular, we will
focus on valence ratings, assuming that, within these
three dimensions, valence is the most highly
related with a positive, negative or neutral emotional
content. In fact, it can be defined as the “the
polarity of emotional activation”
A possible approach to infer the valence of
the words from co-occurrence statistics is the one
adopted by Louwerse and Recchia (2014), who
followed a bootstrapping method to extend the
ANEW lexicon
Experimental results showed, in an indirect
way, that distributional emotive ratings can be
very useful in the implementation of systems for
polarity classification
This paper is organized as follow: in Section 2 we describe the resources used for the creation of the model. Section 3 shows our method and the results obtained. Finally, in Section 4 we evaluate the results and discuss our findings. 2
The main resources we used for our experiments
are the Italian version of the Affective Norms for
English Words
ANEW (Affective Norms for English Words)
We used 2 main regression models to predict the valence from the distributional emotive scores. The first experiment, described in section 3.1 shows a polynomial regression model, and the second one (section 3.2) shows a logistic model in which the valence scores in ANEW have been discretized into two classes representing the positiveness and negativeness of the word.
A simple preprocessing phase has been applied
to align the two resources. ANEW has 1121
words, but 65 of them have multiple POS (e.g.
aereo (plane) can be both a noun and an adjective).
We duplicated each word, extending the dataset
to 1189 elements, and extracted distinct emotive
scores for each <lemma,PoS> pair. In addition,
we replaced word forms like “scorie” (waste), with
their most frequent word type (scoria) in ItaWaC
Due to the bimodal distribution of the data in ANEW, we decided to use a polynomial regression model to predict the valence of the words in ANEW by exploiting their emotive normalized scores in ItEM. Preliminary tests had in fact shown that a simple multiple linear regression model was not able to properly fit the data. The histogram in Figure 1 shows such data distribution, in which most of the ANEW words have a valence score in the ranges 2-3 and 6-8, with a slight bias towards higher values.
To define the most performing degree (Deg) of the polynomial function, we performed 10-fold cross validation for degrees in the range f1:::5g. The results, presented in Table 1, clearly show overfitting for degrees equal or higher than 3. This is due to the fact that, given the number of parameters (#P), the estimated minimum number of observations (Min. Obs.), computed as #P 15, must be at most around the total number of observations. This is true only for polynomial of degree 1 and 2. This finding is in line with Schmidt (1971) and Harrell (2001) who demonstrated that to guarantee the reliability of the prediction, each parameter in the regression model should have a minimum number of observations between 10 and 20.
Given this result, we performed a polynomial interpolation over our parameters with a polynomial of degree 2. Then, we applied a simple multiple linear regression over the new data for predicting the valence. Figure 2 shows the result of the regression fitting. For this model, we obtained a R-Squared (R2) of 0:58, a mean absolute error (MeanAE) of 1:08, a mean squared error (MSE) of 1:81, and a Median absolute error (MedianAE) of 0:95.
For this experiment, we also provide two additional evaluations (the corresponding results are shown in Table 2):
A) the results of prediction by means of a 10fold cross validation; B) the results of prediction by means of split of the data between training (66%) and test (33%).
Method
A B
R2 0:53 0:54
MeanAE 1:13 1:13
MSE 1:99 2:00
MedianAE 0:98 0:93
We would like to notice that our prediction performs better for words with a very high arousal. In fact, emotionally arousing words were more likely to be produced as an emotive prototypical word in the elicitation phase of ItEM. As a consequence, since ItEM’s emotive centroids have been constructed using the vectors of these words (namely the seeds), also their nearest neighbors (i.e., the most emotive words) are assumed to have a high level of arousal. Moreover, the distribution of the data in Figure 3, clearly shows how, in ANEW, high arousal corresponds to very high (or very low) valence ratings, suggesting that highly arousing words tend to be very positive or very negative (i.e. polarized). Building on this evidence, we performed an additional experiment in which we used the portion of the data (573 words) with an arousal rating higher than its median (5:64) for prediction. In such model, in fact, R2 is attested to 0:64.
Given the distribution of the data showed in Figure 2, it is clear that a polynomial regression might not be a perfect fit for valence ratings. Nevertheless, it is very important to focus on MeanAE and MSE values. These errors are relatively low with respect to the scale of the human-rated valences.
This means that, on average, the difference between human-rated valence and predicted valence is between 1 and 2. To prove this point, we also compared the obtained scores with the original human annotations, by exploiting the standard deviation for each valence rating. We found that 73; 5% of our predictions fall into the correct range around the average valence. If we consider a word having (in ANEW) a valence score of around 8 (e.g. pace (peace)) the system will predict a score between 6 and 9, leaving the word around the same (positive) area of the distribution. The same (and opposite) goes for low-valenced words, such as drogato (drug addicted) and feccia (scum). Problems arise in the case of the words with a medium valence. Examples can be corridoio (corridor) and insipido (bland). In this case, the word will have the same chance to be attributed with a high valence score (5-6) or with a low one (3-4). Supposing to discretize valence ratings in two classes, a positive and a negative one, with a cut on the median, predictions will fall in the right class for most of the high (or low) valenced words, and (possibly) in the wrong one for the words of medium valence. In fact, by constructing a shallow mapping of the valence into positive (with valence >= 5:5) and negative class, we found a correlation of 0:73 between predicted and actual data. Building on the last experiment, and supposing a discretization of the valence into the positive and negative class, we also used a logistic regression model to predict this binary valence. The results of this experiment are very promising. We performed 10-fold cross validation to evaluate the effectiveness of the logistic regression over the transformed valence ratings, and obtained an average mean accuracy of 0:80. Detailed results for this evaluation are shown in Table 3.
MicroAVG MacroAVG
Precision 0:806 0:803
Recall 0:803 0:803 The results provided in previous experiments showed both pros and cons of this approach.
The main advantage of exploiting distributional emotive scores to predict the word’s valence is that such scores can be easily obtained in an unsupervised way by means of co-occurrence statistics.
Moreover, predicted data showed a rather good accuracy with respect to the actual distribution, especially considering the logistic regression experiment. In fact, our models reach peak performances by focusing the analysis on the sign of the valence with logistic regression instead of working with continuous values.
On the other hand, the main drawback of our approach derives from the dimension of the ANEW dataset, and in particular from the lack of examples around the medium valence score ratings. It is clear that the ratings distribution in this resource prevented us from obtaining reliable results for continuous values. This might also provide an explanation for the errors concerning the logistic regression experiment. We are confident that having access to a new resource covering the full spectrum of the valence more evenly would have a positive impact on our model. 5
In this work we studied the relationship between valence and distributional emotive scores. We modeled our data with regression in order to predict both a continuous score for valence and its corresponding binarized version (i.e., polarity).
Despite the difficulties of modeling an accurate representation of a continuous valence rating from a small and unbalanced dataset like the Italian ANEW, we can identify a clear relationship between distributional emotional scores and a discrete valence obtained by categorizing the ratings into a positive and a negative class.
In the near future, we plan to improve our
regression models, with the aim of reducing the
impact of the distribution of the data in ANEW,
possibly implementing new strategies able to cope
with non linear data. ANEW is a highly renown
psycholinguistic dataset, but we plan to extend the
present work to predict sentiment polarity scores
taken from SentiWordNet
Moreover, we plan to follow the approach employed in ItEM to create a polarity lexicon for Italian, using ANEW words as seed to build positive and negative polarity centroids. This would also be beneficial for evaluating performances on a emotion-based approach and a polarity-based one.
Finally, we aim at testing the effectiveness of our system for Sentiment Polarity Classification. European Chapter of the Association for Computational Linguistics (EACL06), Trento (Italy). Association for Computational Linguistics.