In the last years, emotions recognition tools have become more and more popular, aiming at detecting the emotions of human actors while performing di erent intelligent tasks by means of headsets and facial emotions detection tools. In addition to this kind of technology, when participants interact with each others by means of textual exchanges, sentiment analysis techniques, from the natural language processing research area, are exploited to detect the polarity of the exchanged messages. Investigating how these two connected components interacts and can support each other towards a better emotions and sentiment detection is a relevant but unexplored research challenge. In this paper, we start from a dataset of debate interactions annotated with the emotions of the involved participants, captured by means of EEG headsets and a facial emotions recognition tool, and the argumentative structures of the debates, and we compare this information to the polarity of the proposed textual arguments, retrieved through a sentiment analysis algorithm. A pragma-semantic analysis of the obtained results is provided, along with a discussion of the potential future work.
Analyzing and detecting the emotions felt by people engaged in a debate is
becoming more and more important in Arti cial Intelligence. Reasoning techniques
such as argumentation theory [
Up to our knowledge, this is the rst time such a kind of analysis is proposed, while it is important to start considering the interplay of these three components, i.e., argumentation, emotions and sentiment, that are indispensable to model cognitive agents. 2
In this section, we rst describe the dataset of textual arguments we use to address our pragma-semantic analysis (Section 2.1), and second, we provide some insights about sentiment analysis techniques (Section 2.2). Finally, we discuss the results of our ongoing pragma-semantic analysis (Section 2.3). 2.1
In [
The dataset of debates consists of 10 debates carried out by 4 participants at a time (20 total participants), excluding the moderator. The dataset is composed of three main layers: (i) the basic annotation of the arguments proposed in each debate (i.e. the annotation in xml of the debate ow downloaded from the debate platform); (ii) the annotation of the relations of support and attack among the arguments; and (iii) starting from the basic annotation of the arguments, the annotation of each argument with the emotions felt by each participant involved in the debate. Table 1 shows some statistics on the dataset1.
An example, from the debate about the topic \Religion does more harm than good" where arguments are annotated with emotions (i.e., the third layer of the annotation of the textual arguments we retrieved), is as follows: <argument id="30" debate_id="4" participant="4" time-from="20:43" time-to="20:43" emotion_p1="neutral" emotion_p2="neutral" 1 The dataset of annotated arguments is available here: http://project.inria.fr/ seempad/datasets/ emotion_p3="neutral" emotion_p4="neutral">
Indeed but there exist some advocates of the devil like Bernard Levi who is decomposing arabic countries. </argument> <argument id="31" debate_id="4" participant="1" time-from="20:43" time-to="20:43" emotion_p1="angry" emotion_p2="neutral" emotion_p3="angry" emotion_p4="disgusted">
I don't totally agree with you Participant2: science and religion don't explain each other, they tend to explain the world but in two different ways. </argument> <argument id="32" debate_id="4" participant="3" time-from="20:44" time-to="20:44" emotion_p1="angry" emotion_p2="happy" emotion_p3="surprised" emotion_p4="angry">
Participant4: for recent wars ok but what about wars happened 3 or 4 centuries ago? </argument>
Figure 1 shows one of the visualizations of the debates that we used to explore the data set (number 9, \Fear government power over Internet"). The participants are color-coded (gray always indicates the moderator). The number inside the nodes represents the identi ers of the arguments, in chronological order. A single directed edge indicates a support relation, where the tail node supports the head node, while a double-lined edge indicates an attack. The shape of the argument nodes shows the associated sentiment (see Section 2.2). The goal of sentiment analysis (sometimes referred to as opinion mining ) is detecting the polarity, whether positive, neutral, or negative, of the attitude contained in a natural language utterance. A typical rst step is to determine whether a statement is objective or subjective, and then only in the latter case one can proceed to identify its polarity. However, often only the second task is performed, thus collapsing objective statements and a neutral attitude.
The last years have seen an enormous increase in research on developing
sentiment analysis systems of various sorts that employ several natural language
processing techniques. Solutions range from simple lookups in polarity or a
ection resources, i.e., databases where a polarity score is associated to terms, to
more sophisticated models built through supervised, unsupervised, and distant
learning involving various sets of features [
Several approaches are found in literature for polarity detection. The
simplest route is detecting the speci c words which are known to express a positive,
negative or neutral feeling. For example, [
In order to extract the sentiment from the text of the debates, we submit every argument separately to Alchemy API2, a free service provided by IBM via 2 http://www.alchemyapi.com/api/sentiment-analysis an online HTTP REST Web service. For any given text, Alchemy API returns a sentiment label, either neutral, negative or positive, and a sentiment score ranging from -1 (totally negative sentiment) to 1 (totally positive sentiment).
We correlate the sentiment of the arguments with the emotion of the participants who wrote them. Moreover, we only consider the primary emotion.
The confusion matrix in Table 2 shows how the sentiment compares to the emotions detected by the FaceReader. It is clear that, while the FaceReader is quite conservative in assigning non-neutral emotions, the output of Alchemy API reveals highly polarized sentiment, with 212 arguments categorized as negative, 140 as positive, and a relative minority of 104 as neutral. The emotion happy is observed in the dataset only twice as primary emotion, and in neither case is the emotion detected for the proponent of the argument.
In order to study the correlation of sentiment and emotions, we proceed to manually map the polarity of emotions and sentiment, following this simple scheme: { angry, disgusted, scared ! negative { happy, surprised ! positive { neutral ! neutral This mapping results in the confusion matrix shown in Table 3. The sentiment extracted by Alchemy API matches the polarity of the emotion in 116 cases (37 negative, 1 positive, 78 neutral, about 25% of all arguments). Out of all the cases of mismatch, in 30 cases the polarities are inverted: in 27 cases a positive sentiment is associated to a negative emotion, and in 3 cases the opposite happens.
We believe that the cases where a full mismatch is observed are the most interesting to explore the relation between the sentiment found in the text, and the emotions felt by the participants in a debate. Therefore, we inspected them one by one and report our ndings in the next section. 2.3
In the three cases where the sentiment is negative, the emotion is \surprised". The arguments seem to be genuinely of a negative nature (\Racial pro ling is a prototype which is unacceptable I think. "), therefore the mismatch could be the result of either a misclassi cation of the emotion by the FaceReader or the nave mapping between sentiment polarity and emotions shown in Section 2.2, where surprise is classi ed as a positive emotion.
In eight cases, the polarity of the argument has been wrongly predicted. Given the statistical nature of the majority of state-of-the-art language analysis tools, including the software for sentiment analysis, a certain rate of errors is always to be expected. Interestingly, here most of the errors are due to the word choice, in particular to the use of certain nouns that are typically associated to positive sentiment. Examples of this phenomenon include, for instance: \Do we know what would be a good way to make someone not a bully? i.e. to teach \respect"?". The word \respect" in particular is associated with a positive polarity by the system. While the sentiment score of the original message is 0:33, replacing the word \respect" with a neutral word like \math" gives a totally neutral message, according to Alchemy API. Another word that seem to confuse the automatic classi cation of sentiment is \thanks", in phrases such as \thanks to ...".
In nine cases, the argument is a reply (possibly an attack or a support) to an argument proposed by another participant. Since we fed isolated messages to the sentiment analysis component, it is natural that the analysis of such cases will not be accurate, since the system is missing important contextual information.
Two of the mismatching arguments are phrased in a quite convoluted way that contributes to confusing the classi er. One of such examples recites: \Of course, from university you can learn a lot of stu , have better degree, but don't think that such degree will be helpful to get a better job later." Note that the pattern \of course X, but Y" is di cult to interpret by automatic language analysis without resorting to some logical interpretation of the text structure.
Finally, in three cases, the sentiment seems to be genuinely positive. In two of them, the corresponding emotion is \scared", which is seldom observed across the entire dataset. Since there is no element of fear in the text of these arguments, we tend to attribute these mismatches to noise in the original data.
The six remaining examples include a mix of the aforementioned phenomena or their features are too sparse to draw proper conclusions. It is worth noticing that one argument is interestingly of ironic nature (\RFID ALL THE PEOPLE!"), a case where the positive polarity of the literal meaning of the text is correctly associated with a negative emotion.
In this paper, we have presented some preliminary results of a pragma-semantic analysis over a dataset of textual arguments from human debates annotated with their emotions, and on which we have then applied sentiment analysis techniques. More precisely, we have studied the cases where a mismatch holds between the sentiment captured from the textual arguments through sentiment analysis, and the emotion(s) detected from the participants proposing such arguments in the debate. Some patterns emerge from this analysis. However, our intuition is that adding also the argumentative information into the loop, i.e., considering not only the detected emotions but also the attack and support relations among the arguments, would return useful information to enrich such patterns. This is our main direction for future work, i.e., to study the interplay of argumentation, sentiment analysis and emotions in debates, in order to detect patterns of information from the argumentation and emotion components to improve the performance of sentiment analysis techniques, and enrich their results.