In this paper, we focus on a particular task which consists in explaining the source and the target of sentiments expressed in social networks. We propose a method for French, which overcomes a fine syntactic parsing and successfully integrate the Conditional Random Field (CRF) method and a smart exploration of a very large lexical network. Quantitative and qualitative experiments were performed on real dataset to validate this approach.
In this article, we focus on a particular task of
sentiment analysis which consists in explaining the
target and the source of sentiments. For example,
in the sentence ”We like the green initiative”, the
sentiment is expressed by the verb ”like”, the
target is ”green initiative” and the source is ”We”. In
In this context, we propose a new approach based on machine learning methods and a very large lexical network, in French, issued from a contributory game JeuxDeMots1. The challenge is twofold: 1) Instead of using fine syntactic parsing, we use a statistical modeling method called Conditional Random Field (CRF), to extract candidates for targets and sources in large volumes of texts 1http://www.jeuxdemots.org/ jdm-accueil.php issued from poorly written social web messages 2) we also exploit the huge French lexical network JeuxDeMots (more than 300, 000 nodes and 7, 000, 000 relations) to choose the best sources and targets among the candidates identified with CRF. This new method has been successfully exploited to analyse the sentiments expressed in the French tweets dealing with environment and climate change.
The paper is organized as follows. In section 2, we briefly present the state of the art. In section 3, we provide a description of our method. In section 4, we provide all the detail of the experiments carried out and the prime results. Finally in section 5, we conclude this work by providing the main perspectives associated with this work. 2
Since the early 2000s, sentiment analysis, also
called “opinion mining”, has experienced
growing interest. Many methods have been developed
to extract emotional states expressed or implied
in texts. To identify sentiments, many resources
exist (e.g. list of words, phrases, idioms), which
were built mostly for English and polarity (e.g.
Linguistic Inquiry and Word Count
Two categories of approaches are used to link
sentiments and potential target and source. 1)
Methods that essentially implement syntactical
aspects, represented by combinations of rules
To choose the best candidate, we use the Games
with a purpose JeuxDeMots, created in 2007
Step 1: Corpus. The corpus we used and annotations have been made in the Ucomp project4. These tweets deal with climate change. Table 1 and 2 present detailed statistics on the corpus.
CRF model was developed with domain indepen2https://FrameNet.icsi.berkeley.edu/
• The original token from the text (word form);
• Surface features: capitalization of the
token (all in upper/lower case, combination
of both), and punctuation mark in the token
(PUNCT, NO PUNCT);
• Lexical features: n-grams, number of
consecutive repeats. Token frequency was
computed based on the entire training corpus.
• Brown clustering: we used Percy Liang’s
implementation of Brown clustering
We experimented with standard tokenization
(provided by TreeTagger) and custom tokenization
Step 3: Lexical network construction of each sentence. The purpose of this step is to extract a part of the lexical network JeuxDeMots representing the relationship between the meaning of the word and the candidates identified in Step 2. The intuition of the algorithm is the following one. We cross the lexical network from node to node. We stop when we no longer encounter new words or if we reach a maximum depth. Two other constraints are used to limit the expansion of the graph.
Constraint 1. To consider only the parts of the network related to our topic (environment and sentiment), we expand a node to another if the new one belong to these two predefined lexical fields chosen via Larousse thesaurus6. If there is no node in the lexical field, we expand to all neighboring nodes.
Constraint 2. We use the association strength Cjdm weighting the edges and consider only the relations frequently instantiated by players. A threshold is set by default.
Step 4: Identification of shortest paths. The objective of this step is to identify in the graph generated in step 3, the paths that must correspond to a compromise between the shortest paths, with a little depth, most reliable according to strength of association Cjdm. We have therefore redefined weights wrt to foster some relationships like synonymy or significant semantic roles such as patient and agent. To identify the paths, we have adapted the shortest path algorithm and used the weights computed according to formulas 1 and 2. The weight wi foster relationships that interest us with wrt while taking into account the strength of association Cjdm. In equation 1, we verify a balance between wrt and Cjdm terms. In equation 2, the term (n 1)2 enables to penalize depth. We only consider paths which contain at least one agent or patient relationship. The path with the best score is proposed to the user to explain the link between the sentiment and the target or source.
w1 = 1 1
1 Cjdm + wrt) ⇤ (n (2) Our experiments were carried out using 10-fold cross-validation. To do this, the training corpus was divided into 10 folds. To build our model, we need a training, development and test corpus. Cross-validation has been distributed as follows: The model is built on 8 folds, the optimization of the construction is performed on the ninth part (development) and the model evaluation performed on the last fold (test).
To perform our experiments, we use
Wapiti7
Table 3 presents the results obtained by different CRF models on training set by cross validation. The features of four bests configurations are : • Configuration 1 : Part Of Speech tagging + lemmatization + lowercase • Configuration 2 : Part Of Speech tagging + lemmatization + lowercase + brown clustering • Configuration 3 : All (Part Of Speech tagging + lemmatization + lowercase + brown clustering + emotion lexicon • Configuration 4 : Part Of Speech tagging + lemmatization + lowercase + emotion lexicon
The results of the evaluation are reported in terms of precision (the number of source and target correctly extracted over the total number of source and target extracted), recall (the number of source and target correctly extracted over the total number of source and target marked the corpus used) and F-measure (the harmonic average of precision and recall). We show two types of results. the first is the results achieved by Exact match and the second by Partial match. We consider that there is an Exact match when the tokens obtained with our model match exactly those of the standard test annotation and we consider a Partial match when the obtained token are included. For example, governor partially matches The governor. Overall, with Exact match, configuration 1 is the best performing configuration. Results show that, we performed a good results with Partial match. Compared to other configurations, configuration 4 gives the best results on precision ( Precision 0.78), and configuration 1 and 2 give the same results and the best results on recall and F-measure (recall 0.52 and F-measure 0.62). Contrary, with the sentiment lexicon as feature, we increase precision, decrease recall and f-measure. Brown Clustering is good feature if we want to have a good precision.
If CRF is relevant for extracting target and source candidates, how can we link them to the sentiments also expressed in the sentences? In figure 1, a sentence is annotated after the exploration of the lexical network. Sentiment tokens are represented by red points. Target and Source obtained with CRF are colored (in blue and yellow). Arrows correspond to the paths identified in the network. The more the arrow is thick the more the path is valuated.
A combination of CRF and huge lexical network exploration seems promising for explaining sentiments in social networks. By experimenting with the CRF model, we found that the results varied depending on the features. The best results are obtained with the features: lemmatization, cluster ID and Part Of Speech tagging.
The first advantage of this method is that we can detect multiple tokens (e.g. parc eolien terrestre, La France). Another advantage of this method is that it is efficient even if the sentence contains misspelling. For example, the system identify mode`le de de´velopement durable (sustainable development model) even if the word de´velopement is misspelled. Finally, the main advantage of this contribution is not to restrict sentiment, source and target identification to the case in which sentiment word is present. Indeed, in most cases people express sentiments implicitly without using these sentiment words. An emotion cannot be limited to something a person feels about a fact and not the sentiment that a person expresses about this fact. Thus, it could be common to explicitly express sentiments about things, but it is more common to feel emotions without expressing them explicitly. Our method take into account this fact and try to identify source and target beyond the explicit cases.
The principal limitation of our method is the length of the sentences in the considered corpora (size of the tweets). In many sentences, there is no source or no target. Results are significantly reduced. Moreover, a quantitative study has to be performed on step 3 and 4 to evaluate the quality of the computed relations between sentiment and targets/sources.
Prospects associated with this work are numerous. First, in this work we focus only on the targets/sources expressed in sentences and we now have to focus on inter-sentence relationships at paragraph level. In future work, we are going to use the best model we obtained on health forum messages with longer sentences. We will also compare our method to identify relations between sentiments and source/target with the methods of the state of the art. We will also adapt CRF to extract directly relations. Finally, we will present to users the part of the network used to identify relations in order to help their interpretation