Workshop "From Objects to Agents" (WOA 2019) Application of Lovheim Model for Emotion Detection in English Tweets Paolo Fornacciari, Stefano Cagnoni, Monica Mordonini, Leonardo Tarollo, Michele Tomaiuolo Dipartimento di Ingegneria e Architettura Università degli Studi di Parma Parma, Italy paolo.fornacciari@unipr.it, stefano.cagnoni@unipr.it, monica.mordonini@unipr.it, leonardo.tarollo1@studenti.unipr.it, michele.tomaiuolo@unipr.it Abstract—Emotions are central for a wide range of everyday a fundamental role, such as medicine [3]. Emotions represent human experiences and understanding emotions is a key problem and identify the human being and therefore they influence both in the business world and in the fields of physiology and a person’s behavior and decisions and also the relationships neuroscience. The most well-known theory of emotions proposes a categorical with others [4]. In evolutionary terms, their main function is system of emotion classification, where emotions are classified to make the individual’s reaction more effective. in situations as discrete entities, while psychologists say that in general man where an immediate response is needed for survival. Such a will hardly express a single basic emotion. According to this reaction does not use cognitive processes and conscious pro- observation, alternative models have been developed, which define cessing. An interesting theme in physiology and neuroscience multiple dimensions corresponding to various parameters and specify emotions along those dimensions. is the question of how emotions interact with and influence Recently, one of the most used models in affective computing other domains of cognition, in particular, attention, memory, is the Lovheim’s cube of emotions, i.e., a theoretical model that and reasoning [5]. In fact, the question of how emotions are focuses on the interactions of monoamine neurotransmitters and represented in nervous system activity is still an unresolved emotions. problem in affective neuroscience [6]. This work presents a comparison between a single automatic classifier able to recognize the basic emotions proposed in the The most well-known theory of emotions proposes a cat- Lovheim’s cube and a set of independent binary classifiers, egorical system of emotion classification, where emotions each one able to recognize a single dimension of the Lovehim’s are classified as discrete entities, independent of each other cube. The application of this model has determined a notable and easily distinguishable [7]. Thus the main taxonomies of improvement of results: in fact, in the best case there is an emotions divide them into positive and negative and more into increment of the accuracy of 11,8%. The set of classifiers has been modeled and deployed on the dis- details in a few basic emotions: surprise, interest, joy, rage, tributed ActoDeS application architecture. This implementation fear, disgust, shame, and anguish [8]. In reality, psychologists improves the computational performance and it eases the system and social science experts say that in general man will hardly reconfiguration and its ability to recognize particular situations, express a single basic emotion [9]. Every facial expression, consisting of particular combinations of basic emotions. every text, every gesture is a composition of multiple emotions. Index Terms—Machine learning, sentiment analysis, emotion detection, natural language processing, software actors. Such expressions change together with the evolution of speech. Similarly to the different shades of color, emotions are not clearly distinguishable and it is difficult to sharply discern I. I NTRODUCTION one emotion from the others. In argumentations about the Sentiment Analysis (SA) applies Machine Learning (ML) human personality, even the objective and neutral information, techniques to textual content, for extracting the feelings and typically used to classify simple information about an object other information, useful for understanding a person’s opinion or a theme, cannot be expressed in their pure form. about a given entity (product, person, topic, etc.). Emotions Recently, one of the most used models in affective com- are central for a wide range of everyday human experiences puting is the Lovheim’s cube of emotions ( [10], [11]), and understanding emotions is a key problem in the business that is a theoretical model that focuses on the interactions world, especially in an era where online communities will of monoamine neurotransmitters and emotions. Although the define future products and services [1], [2]. validity and reliability of this model are still to be determined, Mainly in market analysis, being able to discriminate be- some researches demonstrate that a neurocognitive approach tween positive and negative comments is often sufficient. But, is important to determine emotional reactions to, for example, for other applications that refer to Affective Sciences (AS), visual stimuli [12]. a more detailed analysis of emotions is more appropriate. As This work presents a comparison between a single automatic refer to the scientific study of emotion (or affect) and cover a classifier able to recognize the basic emotions proposed in wide range of interdisciplinary fields in which emotions play the Lovheim’s cube and a set of independent binary classi- 149 Workshop "From Objects to Agents" (WOA 2019) fiers, each one able to recognize a single dimension of the In a previous work of ours [14] we used a hierarchical Lovehim’s cube. A set of classifiers has been modeled on the approach so that first we looked for the polarity of a text and actor-based architecture proposed in [13] for data analysis. then the primary emotion associated with it. This hierarchy This implementation improves the computational performance of classifiers has made the system more flexible and slightly and it eases the system reconfiguration, to recognize particu- improved the performance respect to a flat classifier trained lar situations, that is particular combinations of percentages on the same dataset based on six emotions. In the present of basic emotions. In fact in the Lovehim cube, the basic work, we have tried to exploit a more recent and more refined emotions are classified on a three-dimensional model that theory of emotions that is able to perform a multidimensional allows highlighting their reciprocal relations, which difficult analysis of them. Some work of this kind is present in [28], to distinguish in a hierarchical model, in which all emotions even if the authors used a two-dimensional model, while our are first of all divided into negative, neutral and positive (see reference model has three dimensions: the different emotions for example [14]. are inside a cube whose vertices are the eight "pure" emotions. A curiosity, and a difficulty in the implementation of this Also, in this case, the system is more flexible than a flat model for the recognition of emotions on Twitter, was the lack classifier able to classify the eight primary emotions as it is of available data for the “distress / anguish” emotion. That possible to define a set of points in our multidimensional class is commonly found in the datasets for facial emotion model that represent combinations of “pure” emotions to recognition but not in textual datasets, probably due to the associate "real" human moods. Furthermore, in this case, the approach of the human beings towards the written text [15]. comparison of the flat classifier trained on the same dataset The paper is structured as follows. Section 2 gives a brief gave significantly better results. It is not possible to make a overview of the background in emotion detection. Section 3 direct comparison between the two systems as they refer to describes the implemented framework and Section 4 describes different datasets and to different data collection approaches. the experimental results. Section 5 reports the concluding The models of Parrot, Plutchik, and Lovheim are shown remarks. below. Parrot’s model belongs to the set of categorical emotion models which define a list of discrete categories of emotions. II. R ELATED W ORK Parrott proposed an emotion classification model structured The techniques of Sentiment Analysis aim at the study and on three levels, each representing a list of emotions [29]. The analysis of textual information, with the purpose of detecting model collects more than 100 emotions and is shown in Figure evaluations, opinions and emotions related to a specific entity 1. (product, person, topic, etc.). This type of analysis has impor- tant applications in the political, economic and social fields, such as Web Reputation and Social Media Analytics. Social media and the rise of social networking platforms are one of the most important social phenomena in the last years [16] and the studies of SA on platforms like Twitter are very common in the scientific literature ( [17]–[19]. Datasets based on tweets are used in different tasks of Semantic Evaluation (SemEval), such as in [20] and [21]. SA, with thousands of articles written about its methods and applications (see for example [22]–[24], is a well-established field in Natural Language Processing. Emotion Detection can be viewed as a natural evolution of Sentiment Analysis and its more fine-grained model [25]. This paper summarizes the emo- tion models that are mostly used in emotion-based research. Emotion extraction from different types of social network components is a research topic which is being investigated for a long time now. An updated more comprehensive survey of Emotion Detection from a text can be found in [26]. Some works in the literature for emotion detection have an approach based on lexicon or keyword and in this case they refer to an annotated dictionary or a knowledge base; but many recent works adopt a machine learning approach [27], in this work we use a framework that is based on supervised machine learning algorithms. Most of the works in the literature refer to the theory of Fig. 1. Parrot’s model. the six primary emotions: in general, a text is recognized as belonging to one of these emotions by an automatic classifier. The other two models belong to the class of dimensional 150 Workshop "From Objects to Agents" (WOA 2019) emotion models, which define a few dimensions with some pa- rameters and specify emotions according to those dimensions. Plutchik argued that there are only eight basic emotions (joy, trust, fear, surprise, sadness, anticipation, anger, and disgust), but these emotions can be combined. He suggested the wheel model (2D) in 1980 to describe how emotions are related [30]. The model proposed by Plutchik is summarized in Figure 2. It is possible to observe that this model describes a wide spectrum of emotions, each of them representing a different combination of primary emotions. Fig. 3. Emotions’ Mapping on Lovheim’s Cube Fig. 2. Plutchik’s wheel. Finally, Lovheim proposed a three-dimensional model based Fig. 4. Role of neurotransmitters in the Lovheim model on the relationship between neurotransmitters and emotions [10]. In the model, the three neurotransmitters serotonin, dopamine and noradrenaline form the axes of a coordinate a phenomenon that occurs when the energy of a cell rapidly system, while the eight basic emotions (shame, anxiety, fear, grows/decreases , starts the release of these neurotransmitters anger, disgust, surprise, joy, and interest) are placed in the from the presynaptic terminal nerve through a process of eight vertices. Each vertex of the cube corresponds to one of exocytosis: a cellular process by which a cell expels molecules. the eight possible combinations of the three neurotransmitters, Neurotransmitters are packed into vesicles in the presynaptic as shown in Figure 3. The relationship between emotions and neuron. Once released, they enter the synapse by attaching to neurotransmitters is shown in Figure 4. receptors in the post-synaptic neuron. In biology, the neuro- Neurotransmitters represent a certain type of hormones transmitters considered by Lovehim in relation to emotions that affect the amino acids located in the brain and transmit have the following functions: information from one neuron to another. To gain a better 1) Serotonin, also known as the happiness hormone, deals understanding of how neurotransmitters have an effect on our with numerous functions such as regulation of circadian emotions, a brief explanation of their functioning is needed. rhythms (sleep), appetite control, blood pressure control, An electrical signal (or nerve impulse) travels along the neural control of sexual behavior. It has a positive effect on pathways until it reaches their end. Once the end of the memory, has inhibitory effects on the perception of path is reached, the electrical signal is transformed into a pain, and intervenes in social relations. Low levels of chemical signal (neurotransmitter) inside the synapses (space serotonin lead to a decline in mood, depression, states of between neurons). This signal, crossing the synapses, will be anxiety, and aggression. High levels of serotonin instead transformed back into an electrical signal. The action potential, determine a state of well-being, serenity, tranquillity, and 151 Workshop "From Objects to Agents" (WOA 2019) happiness. TABLE I 2) Dopamine deals with the following functions: move- M APPED E MOTIONS ment, memory, sleep, mood, learning, attention, and Lovheim Emotion Emotion reward. An excess or deficiency of dopamine is the Joy happiness;fun;relief cause of numerous diseases such as Parkinson’s, drug Excitement love;enthusiasm Surprise surprise addiction and schizophrenia. Anger hate 3) Noradrenaline is released by the brain in response to Disgust boredom strong physical or psychological stress. It accelerates Fear worry the heart rate, increases the release of glucose from Shame sadness Anguish - energy reserves, and increases blood flow. Noradrenaline also intervenes in the preparation of the body in the so-called attack or flight reaction. Noradrenaline is the B. Architectures of the classification systems neurotransmitter of excitement: low levels are related to depression, poor memory, lower than average alert One of the main targets of this study is to compare results levels. Too high levels are linked to increased anxiety obtained by a FLAT classifier, trained using the training set and fear. directly, with results obtained by a classifier trained using a Lovheim architecture and a corresponding training set. The III. C LASSIFICATION SYSTEM file modeled according to Lovheim’s theory is split into 3 binary training sets, according to high/low levels of the three This chapter describes the structure of the realized classi- characteristic neurotransmitters: serotonine, dopamine, nora- fication system, including the preparation of the dataset, the drenaline. Then, three binary classifiers are trained and their learning techniques, and the actor-based architecture. results are combined and compared with those obtained from the simple FLAT classifier. Moreover, since in all publicly available dataset there are only 7 out of the 8 emotions of A. Actor-based System Lovheim’s Cube, an assumption has been adopted to avoid For realizing this kind of multilevel classification system, the problem. we have used ActoDES, which is a software framework which The dataset contains a collection of tweets (22900), which adopts the actor model for simplifying the development of address different topics. Each tweet is labeled with one of the complex distributed systems [13], [31]. In fact, it eases the following emotions: sadness, worry, hate, boredom, surprise, creation of complex intelligent systems, supporting the exe- happiness, fun, relief, love, enthusiasm. cution of multiple autonomous classifiers which communicate Since the annotated classes do not correspond to Lovheim’s by means of asynchronous messages. This way, a composite basic emotions, another step is needed before training files classifier can be deployed as a set of loosely coupled cooper- can be created: mapping the dataset emotions onto Lovheim’s ating actors. Each simple classifier and each processing step cube (3). So the emotion classifications proposed by Parrott can instantiated as an actor, allowing the whole architecture and Plutchik have been used. The list of mapped emotions is to be defined at a high level of abstraction, where single shown in table I. actors can be replaced and reconfigured to evaluate alternative approaches. According to the received messages, an actor IV. T RAINING S ETS can update its state and change its behaviors, terminate its own execution, send messages to other actors, create new After the mapping has been completed, 4 training files have actors, etc. Particular interaction patterns allow communities of been created: actors to self organize in dynamic scenarios, involving online • 1 FLAT Training set: this file has 22900 tweets, where learning. A subscription service is available in ActoDES, to each one is labelled with one of the following emotions: facilitate the development of collaborative applications with Shame, Fear, Anger, Disgust, Surprise, Joy, Excitement. actors, as shown in [32]. It is very useful for the integration of • 3 BINARY Training sets: these files have 22900 tweets, multiple actors participating in a structured classification task. where each one is labeled with 1/0 which indicates the Other services, developed for this project, provide additional high/low level of the neurotransmitter for the correspond- functionalities to actors, for the continuous analysis of various ing Lovheim emotion. social streams. E.g., let us consider the following tweet: "wants to hang Leveraging ActoDES and the additional mentioned services, out with friends soon!" we have built a software system which can be used to track This tweet is labeled with the interest/excitement emotion and study a news feed from social media, with an architecture in the FLAT training set. According to Lovheim’s theory, that can be extended to different cases and also to more this emotion has the following coordinates in the cube complex problems. In particular, it can be configured for online (1,1,1), which correspond to high levels of serotonine, operation, handling streams of messages and continuously dopamine and noradrenaline. So, this tweet has been learning from a growing dataset. labeled in each binary training sets with a 1. 152 Workshop "From Objects to Agents" (WOA 2019) Fig. 5. Distributed ActoDeS application architecture. TABLE II TABLE III R ESULTS FOR FLAT TRAINING SET. C OMBINED RESULTS FOR N EUROTRANSMITTERS TRAINING SETS . FLAT Results (%) Neurotransm. Results (%) Lovheim Emotions P R F1 Lovheim Emotion P R F1 Joy 48.1 47.9 48.0 Joy 56.8 65.9 61.0 Excitement 53.0 22.9 32.0 Excitement 64.5 36.5 46.6 Surprise 42.1 5.5 9.7 Surprise 87.3 8.7 15.9 Anger 48.4 23.2 31.3 Anger 52.2 33.9 41.1 Disgust 29.2 4.5 7.7 Disgust 4.0 7.0 5.1 Fear 41.1 73.9 52.9 Fear 52.4 79.2 63.0 Shame 46.7 24.7 32.3 Shame 69.5 42.6 52.8 Weighted Average 45.6 44.1 40.6 Weighted Average 59.9 55.7 53.5 Accuracy 44.1 Accuracy 55.7 A. Pre-Processing C. Assumption For each training file, a quite classical set of preprocessing Since a dataset which contains at least an emotion that filters is applied. The configuration is obtained after an itera- could be labeled as distress/anguish has not been found, an tive optimization process. In particular, sentences are vector- escamotage has been adopted to remove predictions that could ized according to the bag of words approach, after applying the have been classified as this class, because it wasn’t present in Iterated Lovins stemmer, stopwords removal, and tokenization the training file. for unigrams and bigrams. Finally, the most relevant and useful This problem arises because the results are obtained from features are selected for the resulting data, according to the the 8 combinations of serotonine, dopamine and noradrenaline InfoGain algorithm, with a zero threshold. levels: it could happen that the following combination (0,0,1) is predicted, which corresponds to the distress/anguish emo- V. E XPERIMENTAL R ESULTS tion in Lovheim’s cube. This chapter presents the results retrieved applying the dif- The following assumption is adopted: considering the pre- ferent compared classification systems, supervised approach dictions and the confidence levels associated with each tweet, using the 10-folds-cross validation methodology on the differ- classify the text with the closest emotion. Table IV shows ent training sets. the neurotransmitters results, after applying the assumption. In table VII, the related confusion matrix is shown. A. FLAT training set The results for the FLAT training set are shown in table II, VI. C ONCLUSIONS while its confusion matrix is shown in table V. Recently, one of the most used models in affective comput- ing is the Lovheim’s cube of emotions: a theoretical model that B. Binary training sets focuses on the interactions of monoamine neurotransmitters The results obtained from each binary training file would and emotions. This research work has compared a single be irrelevant if taken individually: it is necessary to consider automatic classifier, able to recognize the basic emotions pro- the combination of the predictions on these 3 datasets. The posed in the Lovheim’s cube, with a set of independent binary combined results are shown in table III. classifiers, each one able to recognize a single dimension of The corresponding confusion matrix is shown in table VI. the Lovehim’s cube. 153 Workshop "From Objects to Agents" (WOA 2019) TABLE IV C OMBINED RESULTS FOR N EUROTRANSMITTERS TRAINING SETS USING [11] J. Vallverdú, M. Talanov, S. Distefano, M. Mazzara, A. Tchitchigin, ASSUMPTION . and I. Nurgaliev, “A cognitive architecture for the implementation of emotions in computing systems,” Biologically Inspired Cognitive Results assumpt. (%) Architectures, vol. 15, pp. 34–40, 2016. Lovheim Emotion P R F1 [12] B. D. Moyle, C.-l. Moyle, A. Bec, and N. Scott, “The next frontier in Joy 56.8 65.9 61.0 tourism emotion research,” Current Issues in Tourism, pp. 1–7, 2017. Excitement 64.5 36.4 46.6 [13] G. Lombardo, P. Fornacciari, M. Mordonini, M. Tomaiuolo, and Surprise 74.8 9.9 46.6 A. Poggi, “A multi-agent architecture for data analysis,” Future Internet, Anger 49.0 33.4 39.8 vol. 11, no. 2, p. 49, 2019. Disgust 4.0 7.0 5.1 [14] G. Angiani, S. Cagnoni, N. Chuzhikova, P. Fornacciari, M. Mordonini, Fear 52.4 79.1 63.0 and M. Tomaiuolo, “Flat and hierarchical classifiers for detecting emo- Shame 68.5 42.4 52.4 tion in tweets,” in Conference of the Italian Association for Artificial Weighted Average 59.2 55.9 53.7 Intelligence. Springer, 2016, pp. 51–64. Accuracy 55.9 [15] M. A. Riordan and L. A. Trichtinger, “Overconfidence at the keyboard: Confidence and accuracy in interpreting affect in e-mail exchanges,” Human Communication Research, vol. 43, no. 1, pp. 1–24, 2017. [16] G. Angiani, P. Fornacciari, M. Mordonini, M. Tomaiuolo, and E. Iotti, Table VIII shows a comparison of the results obtained “Models of participation in social networks,” in Social Media Perfor- through a flat classification and a classification based on mance Evaluation and Success Measurements. IGI Global, 2017, pp. 196–224. Lovheim theory, respectively. The application of this model [17] A. Z. Syed et al., “Applying sentiment and emotion analysis on brand has determined a notable increase in accuracy. In particular: tweets for digital marketing,” in 2015 IEEE Jordan Conference on there is an increment, of precision (14.3%), of recall (11.6%), Applied Electrical Engineering and Computing Technologies (AEECT). IEEE, 2015, pp. 1–6. of precision (12.9%). The applied assumption determined a [18] P. Fornacciari, M. Mordonini, and M. Tomaiuolo, “A case-study for little better results: there is an increment of precision (0.7%), sentiment analysis on twitter.” in WOA, 2015, pp. 53–58. of recall (2.4%), of precision (0.2%). Though the emotions of [19] N. Öztürk and S. Ayvaz, “Sentiment analysis on twitter: A text mining the dataset were only 7 of the 8 proposed by Lovheim, it can approach to the syrian refugee crisis,” Telematics and Informatics, vol. 35, no. 1, pp. 136–147, 2018. be asserted that Lovheim’s Cube Theory is more effective than [20] S. Rosenthal, P. Nakov, S. Kiritchenko, S. Mohammad, A. Ritter, and a simple flat classification regarding the sentiment analysis. V. Stoyanov, “Semeval-2015 task 10: Sentiment analysis in twitter,” in The realization of the system on an actor framework has Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015), 2015, pp. 451–463. allowed creating a flexible architecture of composable classi- [21] S. Mohammad, F. Bravo-Marquez, M. Salameh, and S. Kiritchenko, fiers. Moreover, the actor model allows using the realized sys- “Semeval-2018 task 1: Affect in tweets,” in Proceedings of The 12th tem for online classification and continuous learning, since it International Workshop on Semantic Evaluation, 2018, pp. 1–17. naturally leads to the management of streams of asynchronous [22] A. Yadollahi, A. G. Shahraki, and O. R. Zaiane, “Current state of text sentiment analysis from opinion to emotion mining,” ACM Computing messages. Surveys (CSUR), vol. 50, no. 2, p. 25, 2017. [23] L. Zhang, S. Wang, and B. Liu, “Deep learning for sentiment analysis: A R EFERENCES survey,” Wiley Interdisciplinary Reviews: Data Mining and Knowledge [1] E. Cambria, “Affective computing and sentiment analysis,” IEEE Intel- Discovery, vol. 8, no. 4, p. e1253, 2018. ligent Systems, vol. 31, no. 2, pp. 102–107, 2016. [24] G. Angiani, L. Ferrari, T. Fontanini, P. Fornacciari, E. Iotti, F. Magliani, [2] P. Fornacciari, M. Mordonini, and M. Tomaiuolo, “Social network and and S. Manicardi, “A comparison between preprocessing techniques for sentiment analysis on twitter: Towards a combined approach.” in KDWeb, sentiment analysis in twitter.” in KDWeb, 2016. 2015, pp. 53–64. [25] A. Seyeditabari, N. Tabari, and W. Zadrozny, “Emotion detection in text: [3] G. Lombardo, A. Ferrari, P. Fornacciari, M. Mordonini, L. Sani, and a review,” arXiv preprint arXiv:1806.00674, 2018. M. Tomaiuolo, “Dynamics of emotions and relations in a facebook group [26] K. Sailunaz, M. Dhaliwal, J. Rokne, and R. Alhajj, “Emotion detection of patients with hidradenitis suppurativa,” in International Conference from text and speech: a survey,” Social Network Analysis and Mining, on Smart Objects and Technologies for Social Good. Springer, 2017, vol. 8, no. 1, p. 28, 2018. pp. 269–278. [27] V. K. Jain, S. Kumar, and S. L. Fernandes, “Extraction of emotions from [4] P. Fornacciari, M. Mordonini, A. Poggi, L. Sani, and M. Tomaiuolo, multilingual text using intelligent text processing and computational “A holistic system for troll detection on twitter,” Computers in Human linguistics,” Journal of computational science, vol. 21, pp. 316–326, Behavior, vol. 89, pp. 258–268, 2018. 2017. [5] R. J. Dolan, “Emotion, cognition, and behavior,” science, vol. 298, no. [28] I. Perikos and I. Hatzilygeroudis, “Recognizing emotions in text using 5596, pp. 1191–1194, 2002. ensemble of classifiers,” Engineering Applications of Artificial Intelli- [6] P. A. Kragel and K. S. LaBar, “Decoding the nature of emotion in the gence, vol. 51, pp. 191–201, 2016. brain,” Trends in cognitive sciences, vol. 20, no. 6, pp. 444–455, 2016. [29] W. G. Parrott, Emotions in social psychology: Essential readings. [7] P. Ekman, “Facial expression and emotion.” American psychologist, Psychology Press, 2001. vol. 48, no. 4, p. 384, 1993. [8] S. S. Tomkins, “Affect theory,” Approaches to emotion, vol. 163, no. [30] R. Plutchik, “Emotion,” A psychoevolutionary synthesis, 1980. 163-195, 1984. [31] F. Bergenti, A. Poggi, and M. Tomaiuolo, “An actor based software [9] J. Posner, J. A. Russell, and B. S. Peterson, “The circumplex model framework for scalable applications,” Lecture Notes in Computer Science of affect: An integrative approach to affective neuroscience, cognitive (LNCS), vol. 8729, pp. 26–35, 2015, proc. 7th International Conference development, and psychopathology,” Development and psychopathology, on Internet and Distributed Computing Systems (IDCS 2014); Calabria; vol. 17, no. 3, pp. 715–734, 2005. Italy; 2014-09-22/24 [MT]. [10] H. Lövheim, “A new three-dimensional model for emotions and [32] S. Gallardo-Vera and E. Nava-Lara, “Developing collaborative applica- monoamine neurotransmitters,” Medical hypotheses, vol. 78, no. 2, pp. tions with actors,” in Proceedings of the World Congress on Engineering 341–348, 2012. and Computer Science, vol. 1, 2015, pp. 1–5. 154 Workshop "From Objects to Agents" (WOA 2019) TABLE V C ONFUSION M ATRIX FOR FLAT TRAINING SET. FLAT Confusion Matrix Joy Excitement Surprise Anger Disgust Fear Shame Joy 2441 230 34 17 5 2193 175 Excitement 857 593 13 12 4 966 145 Surprise 320 47 88 16 1 1043 98 Anger 80 13 8 275 2 678 131 Disgust 12 1 1 6 7 106 24 Fear 865 122 43 118 2 5492 791 Shame 502 112 22 124 3 2871 1194 TABLE VI C ONFUSION M ATRIX FOR N EUROTRANSMITTERS TRAINING SET. Neurotransmitters Confusion Matrix Joy Excitement Surprise Anger Disgust Fear Shame Joy 3358 199 0 4 14 1460 60 Excitement 887 944 3 158 4 553 37 Surprise 338 157 131 71 105 448 248 Anger 49 37 1 396 1 652 33 Disgust 23 0 1 0 11 78 44 Fear 913 83 0 66 9 5881 478 Shame 347 43 14 63 132 2160 2047 TABLE VII C ONFUSION M ATRIX FOR N EUROTRANSMITTERS ( ASSUMPTION ) TRAINING SET. Neurotransmitters Confusion Matrix (assumption) Joy Excitement Surprise Anger Disgust Fear Shame Joy 3358 199 0 4 14 1460 60 Excitement 887 944 6 158 4 553 38 Surprise 338 157 160 117 105 448 288 Anger 49 37 15 397 1 652 36 Disgust 23 0 1 0 11 78 44 fear 913 83 3 66 9 5881 478 Shame 347 43 29 68 132 2160 2049 TABLE VIII R ESULTS COMPARISON Neurotransmitter FLAT (%) Neurotransmitter (%) (assumption) (%) P R F1 P R F1 P R F1 Joy 48.1 47.9 48.0 56.8 65.9 61.0 56.8 65.9 61.0 Excitement 53.0 22.9 32.0 64.5 36.5 46.6 64.5 36.4 46.6 Surprise 42.1 5.5 9.7 87.3 8.7 15.9 74.8 9.9 46.6 Anger 48.4 23.2 31.3 52.2 33.9 41.1 49.0 33.4 39.8 Disgust 29.2 4.5 7.7 4.0 7.0 5.1 4.0 7.0 5.1 Fear/Terror 41.1 73.9 52.9 52.4 79.2 63.0 52.4 79.1 63.0 Shame 46.7 24.7 32.3 69.5 42.6 52.8 68.5 42.4 52.4 Weighted Average 45.6 44.1 40.6 59.9 55.7 53.5 59.2 55.9 53.7 Accuracy 44.1 55.7 55.9 155