The quality of topic models highly depends on quality of used documents. Insufficient information may result in topics that are difficult to interpret or evaluate. Including external data to can help to increase the quality of topic models. We propose sparsity and grouped sparsity inducing priors on the meta parameters of word topic probabilities in fully Bayesian Latent Dirichlet Allocation (LDA). This enables controlled integration of information about words.
Topic models have been used for text analysis in the last decade very successfully. Topic models assign a number of latent topics to documents and words from a given corpus. These topics can be interpreted as different meanings of words or semantic clusters of the documents in a text corpus. In text analysis the topics can be used in many ways.
The estimation of the topics highly depends on the amount of text data used. Considering the case when we have only very limited amounts of texts to estimate a topic model, the quality of the found topics can be quite poor. In such situation external information about the words can be quite beneficial. For instance prior word probabilities can help sampling word topic distributions from a Dirichlet distribution by adding prior weights on more likely words. In this sense, we try to align the topics with an external probability model like a language model p(w) over some of the words. Structural external information like similarities of words can provide further help to align the topics. Hence, prior weights of whole groups of similar words can be used to estimate the topics.
To measure the quality of the found topics, intrinsic measures like the perplexity
have been used in the past. Recently, coherence measures have been introduced as an
evaluation measure for topics that agree well with human judgements, see [
To increase the coherence, we propose to integrate external information like word
probabilities or word similarities from external data sources. To control the influence
from the external information we weight these information additionally. We integrate
external word probability information by appropriate prior distributions. We add a
sparsity prior and a group sparsity prior on the log-likelihood of the topic model, see [
There are many previous approaches integrating external information into the
generation of a topic model. [
In the last years many approaches have been proposed to evaluate topic models.
[
We integrate external information into LDA via priors on the word-topic distributions.
Similar to the approach by [
The optimal parameters must be found by optimizing the likelihood of the topic model. We perform alternating optimization of the parameters with quasi Newton methods and Gibbs sampling of topics to find the optimal topic model.
For the optimization of the parameters we minimize the part of the negative log likelihood from standard LDA that depends on :
X
X t w:nw,t>0
L =
X log ( ˜t + nk) t
log ( ˜t)+ log ( w,t) log ( w,t + nw,t) with ˜t = Pw
w,t. 3.1
We propose to use a sparsity inducting priors on the parameter w,t weights to influence the prior information about word w for topic t. We expect that some parts of the prior information play a bigger role than other parts in the estimated topic model. To find out which parts are important we impose sparsity to identify them.
We add a Laplace prior on the parameters to gain sparsity. This means, we aim at reducing the amount of adaptation of the external information. This has three advantages. First, we can easily read from the parameters which parts of the prior information influences the topics. Second, we get a simpler model that adapts the external prior information only for some words. Third, we gain control on the amount of external information to be integrated into the topic model.
The difference to standard LDA is that we have now an asymmetric prior that is derived from the external information (the word probabilities) and the weight of this information has a Laplace prior. Adding the Laplace prior of the parameters of the DMR and optimizing for the negative log-likelihood is the same as putting a sparsity inducing penalty on them. Now, the negative log likelihood is simply extended by k tk1: L1 = L +
Hence, the Laplace prior is integrated into the optimization via a sparse lasso penalty
k k1. We solve the optimization problem via Orthantwise Quasi Newton Optimization
[
We propose to add a group lasso penalty to the negative log likelihood to gain group sparsity:
L2 = L + 1 k k1 +
X g 1 k gk2 for the group lasso penalty Pg 1 · k gk2 for the groups g and the variance . Conceptionally, this is the same as having a prior on the parameters that induces group sparsity.
Similar to above we solve the group lasso via Blockwise Coordinate Descent with
Proximal Operators for the group penalty, see [
To find the groups for the grouped sparsity priors on the weight parameters we use
external information about similarities of words. From such similarities we can easily
generate clusters that are used as groups. We divide the weight parameter = ( 1, · · · , G)
with G partial weights g = ( w1,g, · · · wk,g). The partial weights build a group g if
the words w1, · · · wk build a cluster based on the similarities from the external
information. The similarities we use are based on WordNet (see [
In this section, we investigate the topics extracted by our proposed methods (SparsePrior)
for LDA with sparsity prior, (GroupPrior) for LDA with group sparsity prior) and
compare them with two standard state-of-the-art implementations of topic models that
integrate external information about words: (RegLDA) by [
We use two standard text data sets used in previous approaches of topic modelling. First, we use the 20 Newsgroups1 data set. The data set contains about 20.000 text documents from 20 different newsgroups. Overall we have 1000 documents per newsgroup. We additional remove stop words and prune very infrequent and very frequent words. Second, we use the Senseval-32 data set of English lexical samples. The data set contains 1 http://qwone.com/ jason/20Newsgroups/ 2 http://www.senseval.org/senseval3
Data Method
20 newsgroups NPMI UCI UMASS nLL
Wikipedia NPMI UCI UMASS nLL texts from Penn Treebank II Wall Street Journal article. The sizes of the data sets range from 20 to 200 documents per word. Further, we use the wikipedia talk pages to apply the method to a more recent data source of internet based communication. As example, we extract 10.000 postings of discussions on wikipedia from 2002 to 2014 that contain the term ”cloud”. 4.2
In the first experiments, we compare to the state-of-the-art LDA implementations with
external information about words and standard LDA in terms of quality. We want to
show that our model produces more coherent topics. To evaluate the coherence of
the found topics, we use Pointwise Mutual Information (UCI), normalized Pointwise
Mutual Information (NPMI) and arithmetic mean of conditional probability (UMass),
see [
The results on the 20 newsgroups data set on the left in Table 1 show that our proposed group sparsity prior results in topic with better coherence measures than the state-of-the art methods and the standard LDA. From the state-of-the-art competitors only WordFeatures performs comparably good. In terms of loglikelihood, WordFeatures performs best. For the wikipedia talk pages we get similar results as shown on the middle in Table 1.
Finally, we compare the different topic model methods on collection of very small data sets. In Table 2 shows the resulting coherence values on the SensEval data set. LDA with our proposed grouped sparsity prior performs better on all data samples compared to the competitor.
We are especially interested in how the different methods perform on very small data sets. To investigate this, we evaluate the NPMI for the different methods on different sample sizes and different document lengths of the samples. For the 20 news groups date, we sample 100, 100, 5000 and 10000 documents to extract topics. From the wikipedia talk pages we extract postings of different context sizes from 100 to 1000 characters. In Figure 1, we see that our propose sparsity and group sparsity priors results for small samples and small context sizes in the highest NMPI. In these situations our proposed methods of using the group sparsity pays of the most. 5
In this paper we propose to integrate external information about words into topic models to increase topic coherence. We use different priors on the metaparameters for LDA. To control the amount of the integration of the external information we weight them individually. Adding sparsity inducing priors on these weights enables active control on the how much we adapt the external information. By this we trade off topic coherences and likelihood of the topics. Our proposed group sparsity prior further enables integration of external similarity information about words. Now, we can influence the external information of whole groups of words that are similar. The results on large data collections showed the benefit of our proposed method in terms of topic coherence. Finally, we showed that on very small data sets, the group sparsity inducing prior results in better performance.