Topic Sentiment Joint Model with Word Embeddings

Abstract

Topic sentiment joint model is an extended model which aims to deal with the problem of detecting sentiments and topics simultaneously from online reviews. Most of existing topic sentiment joint modeling algorithms infer resulting distributions from the co-occurrence of words. But when the training corpus is short and small, the resulting distributions might be not very satisfying. In this paper, we propose a novel topic sentiment joint model with word embeddings (TSWE), which introduces word embeddings trained on external large corpus. Furthermore, we implement TSWE with Gibbs sampling algorithms. The experiment results on Chinese and English data sets show that TSWE achieves significant performance in the task of detecting sentiments and topics simultaneously.

-

With the rapid development of e-commerce and social media, it is extremely urgent and valuable to automatically analyze the reviews to detect sentiments and topics simultaneously. Great effort on new methodologies for detecting topics and sentiments simultaneously has flourished in the recent years [ 1-5 ].

Several works extending probabilistic topic models[ 6,7 ] have been designed to tackle the problem of the joint extraction of sentiments and latent topics from documents in the recent years [ 2, 3, 8 ]. The joint sentiment topic model (JST) [ 2 ] extends LDA to a four-layer model by adding an additional sentiment layer between the document and the topic layers. Topic sentiment mixture (TSM) [ 8 ] jointly models topics and sentiments in the corpus built on the basis of PLSI. These approaches infer sentiment and topic distributions from the co-occurrence of words within documents. However, when the training corpus is small or when the documents are short, the sentiment and topic distributions might be not very satisfactory. Additionally, most of recent works [ 2, 3, 9 ] try to incorporate some polarity lexicons into their models as the prior knowledge. However, these approaches still have their limitations, for example if the polarity lexicons are not rich, the improvement of the prior is very limited. As a result, we have to seek for other approaches.

Most recently, word embeddings are gaining more and more attention, since they show very good performance in a broad range of natural language processing (NLP) tasks [ 10-12 ]. For example, [ 10 ] incorporates latent feature vector representations of words to LDA model, and [ 11 ] employs latent topic models to assign topics for each word in the text corpus, and learns topical word embeddings (TWE). But these models only complete the task of mining topics. Little attention has been devoted to topic sentiment model with word embeddings so far. In this paper, we propose a new topic sentiment model which incorporates word embeddings. To the best of our knowledge, it is the first work to formulate topic sentiment model with word embeddings.

In contrast with other topic sentiment modeling frameworks, our model is distinguished from them as follows: (1) we incorporate word embeddings trained on very large corpora. It significantly improves the sentiment-topic-word mapping and extends semantic and syntactic information of words. (2) experiments are performed on four real online review data sets for two kinds of language (English and Chinese), which show that our model is used more extensive. (3) we also compare the performance on incorporating the sentiment polarity and without introducing sentiment polarity respectively to demonstrate that our new model is fully unsupervised. We find that our unsupervised model is highly portable to other domains for the sentiment classification task and achieves significant performance in the task of sentiment analysis, and extracting sentiment-specific topics. 2

Topic and Sentiment Model with Word Embeddings

In this section, we propose a novel topic sentiment model with word embeddings called TSWE, as shown in Fig. 1. TSWE is formed by taking the original topic sentiment model JST [ 2, 3 ] and replacing their Dirichlet multinomial component with a two components mixture of a sentiment-topic-to-word Dirichlet multinomial component and a word embeddings component. Our model defines the probability that it generates a word from embeddings component as the multinomial distribution with:

The negative log likelihood according to our model factorizes topic-wise into factors for each topic associated with sentiment. we derive:

Then we apply L-BFGS implementation [ 13 ] from the Mallet toolkit [ 14 ] to derive the topic vector that minimizes . 2.2

Generative process for the TSWE model

The formal definition of the generative process of TSWE model is as follows: For each of sentiment-topic pair ( , )

generate the word distribution of the sentiment-topic pair ~ For each document

draw a multinomial distribution ~ For each sentiment label under document

draw a multinomial distribution ~ For each word in document -draw a sentiment label ~ -draw a topic ~ -draw a binary indicator variable ~ -draw a word ~ 2.3

Gibbs sampling for TSWE model

In this section, we introduce the Gibbs sampling algorithm [ 15 ] for the TSWE model.The detailed derivation process on Gibbs Sampling for topic models can refer the literature [ 16 ].

The Posterior probability can be obtained from the joint probability as follows: Samples derived from the Markov chain are then used to estimate , and picted in equation (4), (5), (6). (2) (3) as de(4) (5)

Experiment

In this section, we explore the performance of TSWE model on document-level sentiment classification and topic extraction evaluations on different kinds of datasets for English and Chinese. 3.1

Experimental setup

3.1.1 Training word embeddings

We train 300 dimensional word embeddings on two corpus by using the Google word2vec toolkit [ 17 ]: Chinese Wikipedia1 and English Wikipedia2.

3.1.2 Experimental datasets

We perform experiments on two kinds of sentiment mining datasets, Chinese and English. Chinese datasets consists of three categories of product reviews datasets3 including book, hotel, and computer, with 1000 positive and 1000 negative examples for each domain. English corpora is the polarity dataset version 2.04 which is introduced by Pang and Lee in 2004, consisting of 1000 positive and 1000 negative movie reviews, which we call MR04 dataset.

Preprocessing: We remove the repetitive comments and stop words, the words that word frequencies are less than 2 or larger than 15 and the words that are not found in Google embeddings representations trained from Chinese Wikipedia corpus and English Wikipedia corpus. In addition, we perform word segment for Chinese datasets 3.2

Parameter Setting

We set the symmetric prior hyper-parameter =0.01 in our TSWE model. The symmetric hyper-parameter is set to , where is the average document length and is total number of sentiment labels, as noted by [ 3 ]. The is set to the standard setting . 3.3

Experimental Results and Analysis

In this section, we present and discuss the experimental results of both document-level sentiment classification and topic extraction.

3.3.1 Sentiment classification evaluation

We use the common metrics to evaluate classification performance: Accuracy. Table 1 presents classification accuracy results obtained by TSWE on the computer data set with the number of topics set to either 1 or 20. By varying , as shown in Table 1, the TSWE model obtains its best result at =0.1, where the is set 0.1 to 0.5 is better than =0.0 on computer data sets. That shows the word embeddings is effective in capturing positive and negative sentiments. So we fix at 0.1, and report experimental results based on this value for the rest of this section. 5 http://www.cs.pitt.edu/mpqa/ 6 http://www.datatang.com/datares/go.aspx?dataid=603399

3.3.2 Topic extraction evaluation.

The other goal of evaluation task is to extract topics and evaluate the effectiveness of sentiment topic. First we need to evaluate the topic clustering performance under the corresponding sentiment polarity. We use two common metrics to evaluate the performance: perplexity and normalized mutual information(NMI)[ 18 ]. More formally, for a test set of documents, the perplexity is: 2400 1600 y t i x e l rep 800 p 0 computer hotel

MR04

book 1 5 10 20 40 60 80

100 K topics extracted from computer data set with JST and TSWE. Each row shows the top 15 words for corresponding topics. We can see that some words of TSWE such as “cooling,f an,r adiator,v oice , temperature, workmanship, operation”a rea boutt hec om-­ puter Heat-dissipationp roblem,a nds omew ordss ucha s“ good, quietness, perfect, like, nice, suitable” are the emotional tendencies of the computer Heat -dissipation problem. It shows that TSWE can extract topic and sentiment simultaneously. Overall, the above analysis illustrates the effectiveness of TSWE in extracting opinionated topics under sentiment from a corpus. In this paper, we propose a novel unsupervised generative model (TSWE) for jointly mining sentiments, sentiment-specific topics from online reviews. To the best of our knowledge, this is the first work to model topic sentiment joint model with word embeddings. Most importantly, the experiments on real review data sets for English and Chinese show that TSWE is effective in discovering sentiments and topics simultaneously. In the future work, we will explore how to properly introduce the lexicon with HowNet lexicon to improve the performance of detecting sentiments and sentimentspecific topics.

Acknowledgements.

This research is supported by the National Nature Science Foundation of China under Grants 61472258, 61402294, National Key Technology Research and Development Program of the Ministry of Science and Technology of China (2014BAH28F05), Science and Technology Foundation of Shenzhen City under Grants JCYJ20140509172609162 and JCYJ20130329102032059.

References

1. Dermouche , M. , Kouas , L. , Velcin , J. , Loudcher , S.: A Joint Model for Topic-Senti ment Modeling from Text . In: ACM/SIGAPP Symposium On Applied Computing (SAC) . ( 2015 )

2. Lin , C. , He , Y. : Joint sentiment/topic model for sentiment analysis . In: Proceedings of the 18th ACM conference on Information and knowledge management , pp. 375 - 384 . ACM, ( 2009 )

3. Lin , C. , He , Y. , Everson , R. , Rüger , S. : Weakly supervised joint sentiment-topic detec tion from text. Knowledge and Data Engineering , IEEE Transactions on 24 , 1134 - 1145 ( 2012 )

4. Pavitra , R. , Kalaivaani , P. : Weakly supervised sentiment analysis using joint sentiment topic detection with bigrams . In: Electronics and Communication Systems (ICECS) , 2015 2nd International Conference on, pp. 889 - 893 . IEEE, ( 2015 )

5. Brody , S. , Elhadad , N.: An unsupervised aspect-sentiment model for online reviews . In: Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics , pp. 804 - 812 . Association for Computational Linguistics, ( 2010 )

6. Blei , D.M. , Ng , A.Y. , Jordan , M.I. : Latent dirichlet allocation . the Journal of machine Learning research 3 , 993 - 1022 ( 2003 )

7. Hofmann , T. : Probabilistic latent semantic indexing . In: Proceedings of the 22nd an nual international ACM SIGIR conference on Research and development in information retrieval , pp. 50 - 57 . ACM, ( 1999 )

8. Mei , Q. , Ling , X. , Wondra , M. , Su , H. , Zhai , C. : Topic sentiment mixture: modeling facets and opinions in weblogs . In: Proceedings of the 16th international conference on World Wide Web , pp. 171 - 180 . ACM, ( 2007 )

9. Chen , Z. , Li , C. , Sun , J.-T., Zhang , J. , Li , C. , Zhang , J., Sun , J.-T., Chen , Z. : Sentiment Topic Model with Decomposed Prior . In: SDM, pp. 767 - 775 . ( 2013 )

10. Nguyen , D.Q. , Billingsley , R. , Du , L. , Johnson , M.: Improving Topic Models with La tent Feature Word Representations . Transactions of the Association for Computational Linguistics 3 , 299 - 313 ( 2015 )

11. Liu , Y. , Liu , Z. , Chua , T.-S., Sun , M. : Topical Word Embeddings . In: AAAI, pp. 2418 - 2424 . ( 2015 )

12. Das , R. , Zaheer , M. , Dyer , C. : Gaussian LDA for topic models with word embeddings . In: Proceedings of the 53nd Annual Meeting of the Association for Computational Linguistics . ( 2015 )

13. Liu , D.C. , Nocedal , J.: On the limited memory BFGS method for large scale optimiza tion . Mathematical programming 45 , 503 - 528 ( 1989 )

14. McCallum , A.K. : {MALLET: A Machine Learning for Language Toolkit} . ( 2002 )

15. Walsh , B. : Markov chain monte carlo and gibbs sampling . ( 2004 )

16. Heinrich , G.: Parameter estimation for text analysis . Technical report ( 2005 )

17. Mikolov , T. , Sutskever , I. , Chen , K. , Corrado , G.S. , Dean , J. : Distributed representa tions of words and phrases and their compositionality . In: Advances in neural information processing systems , pp. 3111 - 3119 . ( 2013 )

18. Manning , C.D. , Raghavan , P. , Schütze , H.: Introduction to information retrieval . Cambridge university press Cambridge ( 2008 )