On the Identi cation of Emotions and Authors' Gender in Facebook Comments on the Basis of their Writing Style

Abstract

In this paper, we propose a method for automatic identifying emotions in written texts in social media with high proliferation such as Facebook. For that task we try to model the way people use the language to express themselves, and also use this model for identifying the gender of the authors. We focused on Spanish due to the lack of studies and resources in that language.

Keywords

• a ective processing
• emotion identi cation
• gender identi cation
• Spanish Facebook

1 Introduction

World is rapidly changing, social media are growing day by day and, in a sense, customers are becoming users looking for new experiences. The emotional aspect of the life is acquiring a growing importance and with it, the need of automatically processing the a ective content of such social media, in order to know what users want and need.

The potentiality o ered by social networking is undoubtful from lots of perspectives like marketing, security or health. But it is also undoubtful that the information users include about themselves, if they include it, may lack credibility. Age, gender, a liation, likes... many users invent them, use linguistic devices such as sarcasm and irony, or simply, they have never reported them. Getting to know the demographic and psychosocial pro le of such users is an opportunity for organizations and companies, and a challenge for natural language processing technologies, due to the fact that the unique certainty we can have is what we can obtain from what the users write and share in such social media.

Studies like [ 11 ] link the use of the language with some traits like the gender of the author, but the vast majority of such investigations are limited to English and traditional media, which should be extended to the (di erent?) use of language in the new technologies and media, and to other languages such as Spanish.

This investigation presents a method for automatically identifying emotions in Facebook and in Spanish language, taking into account another dimension of personality of growing interest in the scienti c community: the author gender3. The main objective is to establish a common framework and a series of resources to investigate the relationship among demographics and emotions, and in the future with personality traits, in social media.

In Section 2 we describe the related work on resources and a ective processing. In Section 3 we present our proposal for modeling the style of the language to automatically identify emotions and gender with a machine learning algorithm. In Section 4 the methodology is described and the results are presented in Section 5. In Section 6 we present the conclusions and future work to achieve our objective. 2

Related Work

Classi cation of related work can be done from two perspectives: the generation of a ective resources and the a ective processing methods. 2.1

Generation of a ective resources

Dictionaries which include the a ective dimension are the most common resources, being pioneers the Lasswell Value Dictionary [ 14 ], where each word is annotated with the existence of dimensions such as wealth, power, rectitude, respect, enlightenment, skill, a ection or wellbeing, and the General Inquirer [ 29 ], where each word is annotated with the existence of dimensions such as active, passive, strong, weak, pleasure, pain, feeling, arousal, virtue, vice, overstated or understated. Both dictionaries use binary tags without considering the degree of occurrence.

Like the previous dictionaries, based on obtaining the existence of certain emotional dimensions, the Clairvoyance A ect Lexicon [ 9 ] labels categories such as anger, joy and fear, and also adds some dimensions as centrality and strength, in order to complete the relationship between the word and its a ective class.

In the line of identifying emotional dimensions, the Dictionary of A ect in Language (DAL) [ 34 ] consists of a set of 8,842 words labeled by their activation and ability to imagine the emotion; or the A ective Norms for English Words (ANEW) [ 1 ] whose objective is to have measured the maximum number of English words in terms of activation, evaluation and control. On the other hand, Strapparava and Valitutti [ 30 ] developed WordNetA ect as a subdomain of Wordnet, where each word is labeled according to its emotional category, evaluation and activation.

In [ 16 ] the authors used Mechanical Turk4 for creating a high-quality, moderatesize, emotion lexicon of about 2,000 terms. They showed how terms related to 3 http://www.uni-weimar.de/medien/webis/research/events/pan-13/pan13web/author-profiling.html 4 https://www.mturk.com/mturk/welcome emotions are among the most common unigrams and bigrams, and also identi ed which emotions tend to be evoked simultaneously by the same term. They used automatically generated word choices to detect and reject erroneous annotations.

Linguistic Inquiry and Word Count (LIWC) is a software for obtaining features from text. It was developed by Pennebaker et al. [ 22 ] Through using text analysis that provides up to 70 dimensions such as the degree of positive and negative emotions, self-references, causal words, and so on.

There are hardly any resources in Spanish language, highlighting the Spanish Adaptation of ANEW developed by [ 27 ]. With the help of 720 participants, they labeled the translation of 1,034 words of ANEW in the dimensions of polarity, activation and control. Moreover, the Spanish Emotion Lexicon (SEL) [ 28 ] consists on 2,036 words associated with the measure of "Probability Factor of A ective use" (PFA) related to one of the six basic emotions of Ekman[ 5 ]: joy, disgust, anger, fear, sadness, surprise. SEL de nes four possible degrees of relationship with each emotion (null, low, medium, high). 19 annotators indicated these values for each word and the PFA was calculated as an average of the percentages assigned to each degree.

Although it is not a resource, we must cite the investigation carried out in [ 20 ]. They studied the necessity, or not, of using a ective dictionaries in the emotion analysis, trying to answer questions as if they improve the identi cation or if they could be replaced by general purpose dictionaries. 2.2

A ective processing methods

Automatic processing of a ectivity has been focused mainly on sentiment analysis, where one of the dimensions of the emotions is investigated: evaluation (or polarity). However, there are a series of methods oriented to classify documents in the corresponding emotional category, usually based on the six basic emotions of Ekman.

We highlight three methods presented in SemEval 2007, where the task of identifying emotions was included for 1,000 news headlines. UPAR7 [ 2 ] used the Stanford syntactic parser for identifying what the main topic was speaking about, estimating each word polarity with the help of Senti Wordnet and Wordnet A ect and obtaining incrementally the global classi cation. UA [ 12 ] utilized three search engines for searching all the words in the headline combined with each emotion, and then calculating the Pointwise Mutual Information according to the number of returned documents. SWAT [ 10 ] was a supervised system based on unigrams and trained with another 1,000 news manually annotated by their authors and which used the Roget thesaurus to expand synonyms and build the features.

In [ 32 ] results are presented and compared with ve own proposals [ 31 ]. WNAFFECT PRESENCE identi ed emotions based on the presence of words from WordnetA ect. LSA SINGLE WORD calculated the LSA similitude between each text and each emotion, taking some words like joy as representatives of the emotional class. LSA EMOTION SYNSET added Wordnet synonyms and LSA ALL EMOTIONS also included all the annotated words from WordnetA ect, as an emotion containers. NB TRAINED ON BLOGS was based on a Naive Bayes classi er trained on a corpus of blogs. Results are shown in Figure 1.

The rst global performance, measured by F1, was obtained by LSA ALL EMOTION WORDS with 17.57%. However, methods presented in SemEval performed better r measure5.

Other investigations related to the identi cation of emotions are: [ 6 ] based on detecting keywords; [ 21 ] based on lexical a nity according to the probability of certain words to be related to certain emotions and [ 15 ] based on the OMCS2 knowledge base.

Previous methods obtain features and approaches by analyzing the semantic content of the texts. On the other hand, in [ 3 ] authors introduced style features as the identi cation of imperative sentences, exclamation signs, the use of capital letters or the use of present and future, in order to identify polarity and emotional category.

Trying to unlink the method from the language, English in all the cases seen so far, in [ 7 ] is described a modular architecture with semantic disambiguation per language or the use of a ective dictionaries as ANEW. This system has also been applied to Spanish.

Following the line of style features and for a language di erent than English, in [ 33 ] authors used substantives, adjectives and verbs with the identi cation of keywords and types of sentences in Japanese in order to identify emotions.

A step forward to the link between emotions analysis and personality traits was given in [ 17 ], sentiment analysis by gender was incorporated. They analyzed three kind of emails: love letters, hate emails and suicide notes.

In Spanish, in [ 4 ] a method based on the SEL dictionary is presented, together with annotated short stories. Di erent machine learning methods are compared, demonstrating an improvement over baseline. 5 Pearson's Kappa to measure the correlation between the obtained result and the random chance

Style-based Identi cation

The vast majority of investigations on emotions analysis are oriented to obtain representative characteristics of the semantics of the documents, that is, they are focused on the analysis of the content, what can imply over tting and dependency on the domain, context or thematics.

On the basis of what was already studied for English by authors such as Pennebaker [ 23 ], we carried out some experiments to investigate the use of the di erent morphosyntactic categories for Spanish. The aim was verifying whether their use was di erent or not, depending on the channel [ 24 ] and its related language register. The nal goal was using the morphosyntactic categories information for identifying emotions and subsequently age and gender [ 26 ].

With the aim of modeling the style of writing we considered readability features as well as the use of emoticons. We used also the Spanish Emotion Lexicon, an a ective dictionary specially built for Spanish [ 28 ]. All these features are topic-independent. The complete set is described below. Each item is a list of individual features represented by frequencies and combined into a vector space model. We obtained the readability features (frequencies and punctuation marks) and emoticons using regular expressions, whereas the morphosyntactic categories where obtained with the Freeling library6.

(F)requencies: Ratio between number of unique words and total number of words; words starting with capital letter; words completely in capital letters; length of the words; number of capital letters and number of words with ooded characters (e.g. Heeeelloooo). (P)unctuation marks: Frequency of use of dots; commas; colon; semicolon; exclamations; question marks and quotes.

Grammatical (C)ategories or Part-of-speech: Frequency of use of each grammatical category; number and person of verbs and pronouns; mode of verb; number of occurrences of proper nouns (NER) and non-dictionary words (words not found in dictionary). (E)moticons7: Ratio between the number of emoticons and the total number of words; number of the di erent types of emoticons representing emotions: joy, sadness, disgust, angry, surprise, derision and dumb. (SEL) Spanish Emotion Lexicon: We obtained the Probability Factor of Affective use value from the SEL dictionary for each lemma of each word. If the lemma does not have an entry in the dictionary, we look for its synonyms. We add all the values for each emotion, building one feature for each emotion.

We do not use any content/context dependent features in order to obtain total independence from the topics. 6 http://nlp.lsi.upc.edu/freeling/ 7 http://es.wikipedia.org/wiki/Anexo:Lista de Emoticonos

Methodology

4.1

Dataset

In the following sections, we describe the raw dataset, the labeling process and the machine learning approaches.

We focused on social media since we are interested in everyday language and how it re ects basic social and personality processes. Due to that, we chose Facebook comments in Spanish language as the source of data for our experiments. Facebook comments have the freedom of expression (and style) without editorial guidelines unlike traditional media like newsletters and the spontaneity in the use of language unlike blogs. Facebook is massively used by people and the expected a ectivity in such media is very high. Facebook also allows us to obtain demographics such as gender, unlike similar media like Twitter, so that we will be able to link this task with groundbreaking tasks such as Author Pro ling at PAN 2013 [ 25 ].

We also chose Spanish because although its high penetration in Internet8, the amount of available resources is still low especially if compared with English. We selected three thematics, with high volume of participation9, and susceptible of emotional comments: politics, football and public gures. We balanced the data by theme and gender.

Neither selection nor cleaning has been done except for language ltering and for ensuring that comments have some text (not only links). Information about the dataset is shown in Table 4.1.

Theme Gender Comments Politics Male/Female 200/200 Football Male/Female 200/200

Public People Male/Female 200/200 Three independent annotators labeled 1,200 documents with the six basic emotions of the Ekman's theory. Annotators were provided with the information of Figure 2 that was obtained by Greenberg [ 8 ] on the basis of psychological relationships of emotional states with the six basic emotions of Ekman. It is remarkable that some secondary emotions are shared by more than one primary 8 http://eldiae.es/wp- content/uploads/2012/07/2012 el espanol en el mundo.pdf 9 http://www.pewglobal.org/ les/2012/12/Pew-Global-Attitudes-Project

Technology-Report-FINAL-December-12-2012.pdf emotion; for example, indignation (indignacion) is shared by anger and disgust, and fascination (fascinacion) is shared by joy and surprise. This issue hinders the unique identi cation of such basic emotions, as it was evidenced in [ 19 ]. Besides, the identi cation of multiple emotions and the absence of any has been allowed.

We calculated the inter-annotator agreement with the Kappa DS method [ 4 ], which allows multiple annotators (three in our case: A1, A2 and A3) and multinomial variables (six not mutually exclusive, the six basic emotions). We show results in Table 4.2.

A1 A2 A3 REST A1 - 0.0587 0.2738 0.1662 A2 0.0587 - 0.1042 0.0814 A3 0.2738 0.1042 - 0.1890

TOT 0.1455

The average value for Kappa, equal to 0.1455, shows a low index of agreement according to the recommendations of [ 13 ]. But, as it is shown by [ 4 ], we have to bear in mind the amount of variables intervening in the evaluation for the right interpretation of such index, that makes it not comparable to their original recommendation. We also grouped the nearest emotions, those which share secondary emotions, as we highlighted in gure 2: joy / surprise and anger / disgust. Results are shown in table 4.2. In this case, Kappa shows a higher value for the agreement, what suggests us that we have to bear in mind such discordance among annotators when assessing the results, especially with respect to joy/surprise and anger/disgust.

A1 A2 A3 REST A1 - 0.6618 0.5656 0.6137 A2 0.6618 - 0.5773 0.6196 A3 0.5656 0.5773 - 0.5715

TOT 0.6016

The nal selection of emotional tags for each document has been based on the concordance of at least two of three annotators. Figures are shown in table 4.2. The low number of documents labeled with the fear category did not allow us to perform experiments with this emotion.

Joy Anger

Fear Disgust Surprise Sadness Neutral A binary classi er has been proposed for each emotion with the aim to determine whether a given text contains such emotion. Each classi er was trained with the labeled examples for its emotion as a positive samples, and with the rest as negative samples. The evaluation method was 10-fold cross validation.

We carried out two di erent evaluations, as in SemEval 2007, the rst one based on Pearson's Kappa to measure the correlation between the obtained result and the random chance, and the second one based on precision, recall and F1.

We tested four learning algorithms implemented in Weka10 with their default parameters: J48 trees, Naive Bayes, Bayes Net and Support Vector Machines. 5

Experimental Results

The objective was to obtain an automatic method for identifying emotions in Facebook comments in Spanish language, attempting the maximum independence from the thematics and trying to link the emotional information with other personal dimensions such as gender. Our starting hypothesis was the style features described in Section 3. 5.1

Emotions identi cation

Style features were enriched with the information of the SEL a ective dictionary, allowing the construction of an adequate and competitive model for identifying emotions. We retrieved the described features in Section 3 for training each classi er and results are shown in Table 5.1. The best results obtained according to each individual metric are marked in bold.

We can appreciate that di erent methods have di erent strengths. J48 has the highest precision in most of the cases at the cost of low recall. In similar way, BayesNet obtains better recall but reducing precision, although it is the best method in terms of F1. In terms of r, values seems to be less correlated with the method. However in most cases the best methods are the statistical ones (Naive Bayes and BayesNet).

With respect to emotions, results for joy and surprise are the highest, mainly for F1 measure, which correlates with the size of the training dataset. Results for sadness are lower than the rest, probably due to the fact that the total number of documents labeled for this emotion is much lower than for the rest (see Table 4.2). This fact implies some dependency of the machine learning approach with the number of samples used in the training and it must be studied further by the parametrization of the methods.

It is necessary to remark the lower results of the SVM method in some experiments, due to the imbalance of the class and the small amount of training data, being this method more sensible to both factors. This could be improved by tunning its con guration parameters.

The proposed features, all independent from thematics and mainly based on the style of writing, achieved competitive results compared to the state-of-the-art approaches in social media in the Spanish language. 5.2

Gender Identi cation

In order to link emotions with demographics, we carried out an experiment consisting in using the features we used for identifying emotions, to learn a new 10 http://www.cs.waikato.ac.nz/ml/weka/ model to identify gender of the authors of the Facebook comments. The hypothesis was that proposed features, which describe the authors' style of writing, could be useful for identifying personal dimensions such as gender.

We trained the Support Vector Machine method implemented in Weka. We experimented with di erent parameters and nally used a Gaussian kernel with g=0.01 and c=3,500. Results for gender identi cation are shown in Table 5.2.

Gender Acc r

Male / Female 59.0 18.0

An r value equal to 18.0 means that the classi er works over the random chance and suggests that style features provide some kind of information about the gender, as [ 11 ] showed for English. An accuracy value of 59.0 allows us to think that our method is competitive for such task in comparison with approaches presented in the Author Pro ling task at PAN 2013. We plan to perform further experiments with the dataset provided for this task.

The fact that features used for identifying emotions allowed us to identify gender with a good accuracy, suggests that there is a certain correlation between the use of emotions and the gender of the authors. 6

Conclusions and Future Work

We have built a dataset of Facebook comments for Spanish, manually labeled it with six basic emotions from Ekman's theory and carried out a Kappa-DS analysis of concordance.

We have proposed a method for automatically identifying emotions based on a combination of stylistic features with the use of the SEL a ective dictionary, obtaining competitive results. We have also veri ed the di culty to label, even for a human, among primary emotions which share secondary emotions, as is the case of joy and surprise, or anger and disgust.

Finally, we have employed the proposed approach for identifying authors' gender, showing that style features provide certain information valuable for such task.

As a future work we plan to investigate further what are the most relevant features for identifying emotions and gender, and their possible relationship. We will investigate the identi cation of combined emotions (joy and anger will be joined respectively with surprise and disgust ), in order to verify if the current results are due to the di culty of discriminating such emotions. We also plan to carry out with the PAN-AP13 dataset for the identi cation of gender and age. For that, we will include the detected emotions as new features in order to investigate the relationship between emotions and demographics. Finally, we aim at introducing some more features trying to obtain a better description of the way people use language (e.g. collocations) and, therefore, analyze discourse in depth).

ACKNOWLEDGEMENTS The work of the rst author was partially funded by Autoritas Consulting SA and by Ministerio de Econom a de Espan~a under grant ECOPORTUNITY IPT-2012-1220-430000. The work of the second author was carried out in the framework of the WIQ-EI IRSES project (Grant No. 269180) within the FP 7 Marie Curie, the DIANA APPLICATIONS Finding Hidden Knowledge in Texts: Applications (TIN2012-38603-C02-01) project and the VLC/CAMPUS Microcluster on Multimodal Interaction in Intelligent Systems.

References

1. Bradley , M. , Lang , P.: A ective norms for English words (ANEW): Instruction manual and a ective ratings . Gainesville: Center for Research in Psychophysiology, University of Florida ( 1999 )

2. Chaumartin , F. Upar7: A knowledge-based system for headline sentiment tagging . In Proceedings of SemEval2007 , Prague, Czech Republic ( 2007 )

3. Dhaliwal , K. , Gillies , M. , O 'connor , J., Oldroyd , A. , Robertson , D. , Zhang , L.: Facilitating online role-play using emotionally expressive characters . Arti cial and Ambient Intelligence, Proceedings of the AISB Annual Convention , 179 - 186 ( 2007 )

4. D az Rangel , I.: Deteccion de afectividad en texto en espan~ol basada en el contexto ling stico para s ntesis de voz . Tesis Doctoral. Instituto Politecnico Nacional. Mexico ( 2013 )

5. Ekman , P. : Universals and cultural di erences in facial expressions of emotion . Symposium on Motivation, Nebraska , 207 - 283 ( 1972 )

6. Elliot , C. : The a ective reasoner: A process model of emotions in a multi-agent system . Northwestern University: Tesis doctoral, The Institute for Learning Sciences , Northwestern University ( 1992 )

7. Garc a , D. , Al as, F.: Emotion identi cation from text using semantic disambiguation . Procesamiento del Lenguaje Natural. (50) , 75 - 82 ( 2008 )

8. Greenberg , L. Emociones: Una gu a interna . Bilbao: Desclee De Brouwer ( 2000 )

9. Huettner , A. , Subasic , P. : Fuzzy Typing for Document Management . ACL 2000 . Hong Kong ( 2000 )

10. Katz , P. , Singleton , M. , Wicentowski , R.: Swat-mp:the semeval-2007 systems for task 5 and task 14 . In Proceedings of SemEval-2007 , Prague, Czech Republic ( 2007 )

11. Koppel , M. , Argamon , S. , Shimoni , A. : Automatically categorizing written texts by author gender . Literay and Linguistic Computing 17 ( 4 ), 401 - 412 ( 2003 )

12. Kozareva , Z. , Navarro , B. , Vazquez , S. , Montoyo., A. : Ua-zbsa: A headline emotion classi cation through web information . In Proceedings of SemEval-2007 , Prague, Czech Republic ( 2007 )

13. Landis , R. , Koch , G. : The measurement of observer agreement for categorical data . Biometrics(35) , 159 - 174 ( 1977 )

14. Lasswell , H. , Namenwirth , J.: The Laswell Value Dictionary . Yale University Press. New Haven ( 1969 )

15. Liu , H. , Lieberman , H. , Selker , T. : Automatic a ective feedback in an email browser . MIT Media Lab Software Agents Group Technical Report ( 2002 )

16. Mohammad , S. , Turney , P. : Emotions evoked by common words and phrases: Using mechanical turk to create an emotion lexicon . In Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, Association for Computational Linguistics , Los Angeles, CA June ( 2010 )

17. Mohammad , S.M. , Yang , T. : Tracking sentiment in mail: how gender di er on emotional axes . Proceedings of the 2nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis. Portland, Oregon , 70 - 79 ( 2011 )

18. Oberlander , J. , Gill , A. J. : Language with character: A strati ed corpus comparison of individual di erences in e-mail communication . Discourse Processes (42) , 239 - 270 ( 2006 )

19. Ortony , A. , Turner , T. : What's basic about basic emotions? Psychological Review ( 97 ), 315 - 331 ( 1990 )

20. Osherenko , A. , Andr , E.: Lexical A ect Sensing: Are A ect Dictionaries Necessary to Analyze A ect? Proceedings of the 2nd international conference on A ective Computing and Intelligent Interaction , ACII ' 07 , Pages 230 - 241 ( 2007 )

21. Pang , B. , Lee , L. , Vaithyanathan , S. : Thumbs Up? Sentiment Classi cation Using Machine Learning Techniques . Conference on Emprirical Methods in Natural Language Processing ( 2002 )

22. Pennebaker , J. W. , Booth , R. E. , Francis , M. E. : Linguistic inquiry and word count: LIWC2007 - Operator's manual . Austin, TX: LIWC.net ( 2007 )

23. Pennebaker , J. W. , Mehl , M. R. , Niederho er, K.: Psychological aspects of natural language use: Our words, our selves . Annual Review of Psychology , ( 54 ), 547 - 577 ( 2003 )

24. Rangel , F. , Rosso , P. El Uso del Lenguaje en los Diferentes Canales de Internet. In: Proceedings Comunica 2.0 . Gandia , Spain, February 21 - 22 . ( 2013 )

25. Rangel , F. , Rosso , P. , Koppel , M. , Stamatatos , E. , Inches , G. : Overview of the Author Pro ling Task at PAN 2013 . In: Forner P., Navigli R. , Tu s D .(Eds.), Notebook Papers of CLEF 2013 LABs and Workshops , CLEF-2013, Valencia, Spain, September 23 - 26 . ( 2013 )

26. Rangel , F. , Rosso , F. : Use of Language and Author Pro ling: Identi cation of Gender and Age . In: 10th International Workshop on Natural Language Processing and Cognitive Sciences NLPCS 2013 CIRM , Marseille, France, October 13 - 17 ( 2013 )

27. Redondo , J. , Fraga , I. , Padron , I. , Comesan~a, M.: The Spanish adaptation of ANEW (A ective Norms for English Words) . Behavior Research Methods (39) , 600 - 605 ( 2007 )

28. Sidorov , G. , Miranda , S. , Viveros , F. , Gelbukh , A. , Castro , N. , Velasquez , F. , D az , I., Suarez , S. , Trevin~o, A. , Gordon , J.: Empirical Study of Opinion Mining in Spanish Tweets . LNAI 7629-7630 ( 2012 )

29. Stone , P. , Dunphy , D. , Smith , M.: General Inquirer: A Computer Approach to Content Analysis . M.I.T . Press. Oxford, England ( 1966 )

30. Strapparava , C. , Valitutti , A. : Wordneta ect: an a ective extension of wordnet . Proceedings of the 4th International Conference on Language Resources and Evaluation. Lisboa , 1083 - 1086 ( 2004 )

31. Strapparava , C. , Mihalcea , R. : Learning to identify emotions in text . In Proceedings of the 2008 ACM Symposium on Applied Computing (SAC 2008 ), 1556 - 1560 ( 2008 )

32. Strapparava , C. Mihalcea , R.: SemEval- 2007 Task 14: A ective Text . Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007) ( 2008 )

33. Sugimoto , F. , Yoneyama , M.: A method for classifying emotion of text based on emotional dictionaries for emotional reading . Proceedings of the 24th IASTE D International Multi-Conference Arti cial Intelligence and Applications . Innsbruck. 91 - 96 ( 2006 )

34. Whissell , C. The dictionary of a ect in language . Emotion: Theory, research and experience. The measurement emotions , 113 - 131 ( 1989 )