We propose a new supervised topic model that uses a nonparametric density estimator to model the distribution of real-valued metadata given a topic. The model is similar to Topics Over Time, but replaces the beta distributions used in that model with a Dirichlet process mixture of normals. The use of a nonparametric density estimator allows for the fitting of a greater class of metadata densities. We compare our model with existing supervised topic models in terms of prediction and show that it is capable of discovering complex metadata distributions in both synthetic and real data.
Supervised topic models are a class of topic models that, in addition to modeling documents as mixtures of topics, each with a distribution over words, also model metadata associated with each document. Document collections often include such metadata. For example, timestamps are commonly associated with documents that represent the time of the document’s creation. In the case of online product reviews, “star” ratings frequently accompany written reviews to quantify the sentiment of the review’s author.
There are three basic reasons that make supervised topic models attractive tools for use with document collections that include metadata. Better Topics: one assumption that is often true for document collections is that the topics being discussed are correlated with information that is not necessarily directly encoded in the text. Using the metadata in the inference of topics provides an extra source of information, which could lead to an improvement in modeling the topics that are found. Prediction: given a trained supervised topic model and a new document with missing metadata, one can predict the value of the metadata variable for that document. Even though timestamps are typically included in modern, natively digital, documents they may be unavailable or wrong for historical documents that have dbeen igitized using OCR. Also, even relatively modern documents can have missing or incorrect timestamps due to user error or system mis-configuration. For example, in the full Enron email corpus1, there are 793 email messages with a timestamp before 1985, the year Enron was founded. Of these messages 271 have a timestamp before the year 100. Analysis: in order to understand a document collection better, it is often helpful to understand how the metadata and topics are related. For example, one might want to analyze the development of a topic over time, or investigate what the presence of a particular topic means in terms of the sentiment being expressed by the author. One may, for example, plot the distribution of the metadata given a topic from a trained model. Several supervised topic models can be found in the literature and will be discussed in more detail in Section 3. These models make assumptions about the way in which the metadata are distributed given the topic or require the user to specify their own assumptions. Usually, this approach involves using a unimodal distribution, and the same distribution family is used to model the metadata across all topics. 1http://www.cs.cmu.edu/˜enron These modeling assumptions are problematic. First, it is easy to imagine metadata and topics that have complex, multi-modal relationships. For example, the U.S. has been involved in two large conflicts with Iraq over the last 20 years. A good topic model trained on news text for that period should ideally discover an Iraq topic and successfully capture the bimodal distribution of that topic in time. Existing supervised topic models, however, will either group both modes into a single mode, or split the two modes into two separate topics. Second, it seems incorrect to assume that the metadata will be distributed similarly across all topics. Some topics may remain fairly uniform over a long period of time, others appear quickly and then fade out over long periods of time (e.g., terrorism after 9/11), others enter the discourse gradually over time (e.g., healthcare reform), still others appear and disappear in a relatively short period of time (e.g., many political scandals).
To address these issues, we introduce a new
supervised topic model, Topics Over Nonparametric Time
(TONPT), based on the Topics Over Time (TOT)
model [
Significant work has been done in the document
modeling community to make use of Dirichlet process
mixtures with the goal of eliminating the need to
specify the number of components in a mixture model. For
example, it is possible to cluster documents without
specifying a-priori the number of clusters by
replacing the Dirichlet-multinomial mixing distribution in
the Mixture of Multinomials document model with a
Chinese Restaurant Process. The CRP is the
distribution over partitions created by the clustering effect of
the Dirichlet process [
The DP is a distribution over probability measures G
with two parameters: a base measure G0 and a
total mass parameter m. Random probability measures
drawn from a DP are generally not suitable as
likelihoods for continuous random variates because they
are discrete. This complication can be overcome by
convolving the G with a continuous kernel density f
[
G xijG
DP(m; G0) Z
f (xij )dG( ) This model is equivalent to an infinite mixture of f distributions with hierarchical formulation:
G ijG xij i
DP(m; G0) G f (xij ) In our work we use the normal distribution for f . The normal distribution has many advantages that make it a useful choice here. First, the parameters map intuitively to the idea that the parameters in the DPM are the “locations” of the point masses of G and so are a natural fit for the mean parameter of the normal distribution. Second, because the normal is conjugate to the mean of a normal with known variance, we can also choose a conjugate G0 that has intuitive parameters and simple posterior and marginal forms. Third, the normal is almost trivially extensible to multivariate cases. Fourth, the normal can be centered anywhere on the positive or negative side of the origin which is not true, for example, of the gamma and beta distributions. Finally, just as any 1D signal can be approximated with a sum of sine waves, almost any probability distribution can be approximated with a weighted sum of normal densities. j zdi wdi
Nd td
In this section we will describe the three models which
are most closely related to our work. In particular,
we focus on the issues of prediction and the posterior
analysis of metadata distributions in order to highlight
the strengths and weaknesses of each model.
The most closely related models to TONPT are
Supervised LDA (sLDA) [
Prediction in sLDA is very straightforward, as the latent metadata variable for a document can be marginalized out to produce a vanilla LDA complete conditional distribution for the topic assignments. The procedure for prediction can thus be as simple as first sampling the topic assignments for each word in an unseen document given the assignments in the training set, and then taking the dot product between the estimated topic proportions for the document and the GLM coefficients. In terms of the representation of the distribution of metadata given topics, however, the model is somewhat lacking. The coefficients learned during inference convey only one-dimensional information about the correlation between topics and the metadata. A large positive coefficient for a given topic indicates that documents with a higher proportion of that topic tend to have higher metadata values, and a large negative coefficient means that documents with a higher proportion of that topic tend to have lower metadata values. Coefficients close to zero indicate low correlation between the topic and the metadata. In TOT, metadata are treated as per-word observations, instead of as a single per-document observation. The model, shown in Figure 2, assumes that each per-word metadatum tdi is drawn from a pertopic beta distribution: tdi Beta( zdi1; zdi2). The inference procedure for TOT is a stochastic EM algorithm, where the topic assignments for each word are first sampled with a collapsed Gibbs sampler and then the shape parameters for the per-topic beta distributions are point estimated using the Method of Moments based on the mean and variance of the metadata values for the words assigned to each topic. Prediction in TOT is not as straightforward as for sLDA. Like sLDA, it is possible to integrate out the random variables directly related to the metadata and estimate a topic distribution for a held-out document using vanilla LDA inference. However, because the model does not include a document-level metadata variable, there is no obvious way to predict a single metadata value for held-out documents. We describe a prediction procedure in Section 5, based on work one by Wang and McCallum, that yields acceptable results in practice.
Despite having a more complicated prediction procedure, TOT yields a much richer picture of the trends present in the data. It is possible with TOT, for example, to get an idea of not only whether the metadata are correlated with a topic, but also to see the mean and variance of the per-topic metadata distributions and even to show whether the distribution is skewed or symmetric.
Another related model is the Dirichlet Multinomial
Regression (DMR) model [
T
d zdi wdi m sdi tdi
Nd
D jk
Kt
T 2 j
T discussion of supervised topic models. 4
TONPT models metadata variables associated with each word in the corpus as being drawn from a topicspecific Dirichlet process mixture of normals. In addition, TONPT employs a common base measure G0 for all of the per-topic DPMs, for which we use a normal with mean 0 and variance 02.
The random variables are distributed as follows: dj tj zdij wdijzdi;
j2j ;
Gj jG0; m tdijGzdi ; z2di
) DP(G0; m) Z
f (tdi; ; z2di )dGzdi ( ) where f ( ; ; 2) is the normal p.d.f. with mean and variance 2. Also, j 2 f1; : : : ; T g, d 2 f1; : : : ; Dg, and, given a value for d, i 2 f1; : : : ; Ndg. We note that, as in TOT, the fact that the metadata variable is repeated per-word leads to a deficient generative model, because the metadata are typically observed at a document level and the assumed constraint that all of the metadata values for the words in a document be equivalent is not modeled. The advantage of Symbol Meaning
Common Supervised Topic Modeling Variables
Prior parameter for document-topic distributions d Parameter for topic mixing distribution for document d Prior parameter for the topic-word distributions j Parameter for the jth topic-word distribution zdi Topic label for word i in document d z di All topic assignments except that for zdi w Vector of all word token types wdi Type of word token i in document d tdi Timestamp for word i in document d td Timestamp for document d t Vector of all metadata variable values t^ A predicted value for the metadata variable D The number of documents T The number of topics V The number of word types Nd The number of tokens in document d
TONPT Specific Variables m Total mass parameter for DP mixtures sdi DP component membership for word i in document d s di All DP component assignments except that for sdi G0 The base measure of the DP mixtures 0 The mean of the base measure 02 The variance of the base measure jk The mean of the kth mixture component for topic j
A vector of all the values j2 jk Thweivthaoriuatncejkof the components of the jth DP mixture 2 A vector of all the DPM 2s , Shape and scale parameters for prior on topic s Kj The number of unique observed s for topic j nj The number of tokens assigned to topic j njk The number of tokens assigned to the kth component of topic j ndj The number of tokens in document d assigned to topic j njv The number of times a token of type v was assigned to topic j Kz<ddii The number of unique s observed for topic zdi before the ith token of document d (jk) The set of all tdi s.t. zdi = j and sdi = k f (y; ; 2) The normal p.d.f. at y with mean and variance 2 this approach is that this configuration simplifies inference, and also naturally balances the plurality of the word variables with the singularity of the metadata variable, allowing the metadata to exert a similarly scaled influence on the topic assignments during inference. In addition, this modeling choice allows for a more fine-grained labeling of documents (e.g., at the word, phrase, or paragraph level) and for finer grained prediction. For example, while timestamps should probably be the same for all words in a document, sentiment does not need to meet this constraint– there are often positive comments even in very negative reviews.
This model does not lend itself well to inference and sampling because of the integral in the distribution over tdi. A typical modification that is made to facilitate sampling in mixture models is to use an equivalent hierarchical model. Another modification that is typically made when sampling in mixture models is to separate the “clustering,” or mixing, portion of the distribution from the prior over mixture component parameters. The mixing distribution in a DPM is the distribution known as the Chinese Restaurant Process. The Chinese Restaurant Process is used to select an assignment to one of the points that makes up the DP point process for each data observation drawn from G. The locations of these points are independently drawn from G0.
Figure 3 shows the model that results from decomposing the Dirichlet process into these two component pieces. The Kj unique values that have been sampled so far for each topic j are drawn from G0. The sdi variables are indicator variables that take on values in 1; : : : ; Kj and represent which of the DPM components each tdi was drawn from. This model has the following changes to the variable distributions: sdijzdi; s<di; m 8= k with prob / n<di >< zdi;k
for k = 1; : : : ; Kz<ddii >:= Kz<ddii + 1 with prob / m jkjG0
G0 tdijzdi; szdi ; ; 2
f tdi; zdisdi ; z2di Where s<di refers to all the sd0i0 that came “before” sdi and before is defined to mean all (d0; i0) such that (d0 < d) or (d0 = d and i0 < i). Likewise, nz<ddi;ik is the count of the number of times that sd0i0 = k for all d0; i0 before sdi and Kz<ddii is the highest value of any sd0i0 (number of unique observed s) before sdi. So, conditioned on zdi the sdi are distributed according to a Chinese Restaurant Process with mass parameter m. The and variables in the model are nuisance variables: they are not necessary for the assignment of tokens to topics or for the estimation of the distributions of the response variables so, as is typical when conducting Gibbs sampling on these models, we integrate them out before sampling. 4.1
Gibbs Sampler Conditionals Now we derive the complete conditionals for the collapsed Gibbs sampler used for inference in the model. There are four groups of variables that must be sampled during inference: the per-word topic labels z, the per word DPM component assignment variables s, the DPM component means , and the per-topic DPM component variances 2. Note that, because of normal-normal conjugacy, it would be possible to collapse the variables from the model. We choose to sample values for anyway because the parameters of the DPM are useful artifacts in their own right, as they enable rich posterior analyses of the per-topic metadata distributions. 4.1.1
Complete Conditional for z and s We choose to sample zdi and sdi in a block, since the calculations necessary to sample zdi include those sufficient to sample both variables jointly.
/ ?dj PV v=1 ?jv ?jwdi t; m; ;
; ; G0; ; ) 8>> nnj+jkm f (tdi; jk; j2) > > <
if k j j j, >> njm+m f (tdi; 0; 02 + j2) > > : if k = j j j + 1 (2) where ?dj = j + ndj , ?jv = v + njv. 4.1.2
Complete Conditional for When sampling a zdi; sdi pair if sdi = Kzdi + 1 (i.e., we are creating a new component for the DPM for that topic), then we need to draw a new for the zdith [zdi; sdi] =p(zdi = j; sdi = kjz di; s di; 2; w; (1) 5
DPM. Also, each jk needs to be resampled each iteration of the Gibbs sampler.
Let (jk) = ftdi : zdi = j and sdi = kg ordered arbitrarily, which groups the tdi by the topic and DPM component that they are associated with. The complete conditional for each is: [ jk] =p( jkjs; t; w; z; jk; 2; ; ; m; ; ; 0; 02) =f jk; jk?; j2k? (3) (4) where ?2 = ? = 12 + j (j2k)j 0 j
j (jk) ! 02 + Pji=(1jk)2 i 0 j 1 and 2 ? The complete conditional is a common result for gamma-normal conjugacy. In this case, the likelihood is restricted to those tdi for which zdi = j: j2 = InverseGamma ( ? ; ? ) (5) where ? =
+ n2j , ? = +
PdD=1 PiN=d1[1j (zdi)( zdi;sdi 2 tdi)2] ; and 1j (x) is the Kronecker delta.
We inferred topic assignments and metadata distributions for several real-world datasets using sLDA, TOT, TONPT, and a baseline method that we will refer to as PostHoc in which a vanilla LDA model is inferred over the dataset and then a linear model is fit to the metadata using the document topic proportions as predictors.
Because it is difficult to know a-priori what form the distributions over metadata given topics will take in real-world data, we also ran one experiment with synthetic data, where the metadata distributions were prespecified. Synthetic data was used in order to determine whether TONPT can accurately recover complex metadata distributions in conjunction with topic distributions.
The focus of our experiments was to measure
quantitatively how well each model can predict metadata
values on unseen data and to assess qualitatively (e.g.,
via inspection) whether the trained models capture
human intuition and domain knowledge with respect to
the correlations between topics and metadata values.
These words are especially common in this corpus
(e.g., in datelines) and provide a strong signal that is
not based on the topical content of the articles (i.e.,
they allowed the models to “cheat”). After feature
selection there were 10,230 non-empty documents in the
final dataset.
We ran our experiments on three real-world datasets.
For each dataset the timestamps of the documents
were extracted and used as the metadata. For all
realworld datasets, stopwords were removed using the
stopwords file included in the MALLET topic
modeling toolkit [
The first dataset consists of the State of the Union
Addresses delivered by Presidents of the United States
from the first address by George Washington in 1790
to the second address by Barack Obama in 2010. The
data was prepared in the manner similar to that of
Wang and McCallum [
The final dataset was the Reuters 21578 corpus[
R2(t; ^t) = 1
Pd(td
Pd(td
t^d)2
t)2
;
where td is the actual metadatum for document d, t^d
is the prediction and t is the mean of the observed tds.
For linear models this metric measures the proportion
of the variability in the data that is accounted for by
the model. More generally, it is one minus the
relative efficiency of the supervised topic model predictor
to a predictor that always predicts the mean of the
observed data points. This value can be negative in cases
where the model being evaluated performs worse than
the mean predictor. The means and standard errors
of the R2 values across all ten folds were recorded.
In order to assess the statistical significance of the
results, a one-sided permutation test was used to
calculate p-values for the hypothesis that the mean R2 for
the model with the highest mean R2 was greater than
the mean R2 for each of the other models being tested.
P-values less than 0.05 were considered significant.
As discussed above, prediction in the case of sLDA is
quite simple. In the case of TOT and TONPT,
prediction is complicated by the fact that these models have
per-word metadata variables, and not per-document
variables. In addition, they do not produce a
prediction using a simple dot product, but instead they
provide a distribution over predicted values given a topic
assignment. In order to perform prediction in TOT
one finds the metadata value with maximal posterior
probability given the topic assignments for all of the
words in the test document[
t i In order to approximate this value, we first infer topic assignments for each word in the document using a version of the model in which the metadata and related variables have been integrated out (i.e., vanilla LDA). Next, because the posterior is a fairly complicated product, and difficult to maximize directly, we approximate by choosing several discrete points and check the value of the posterior at each test point. In the original TOT paper, the candidate points were chosen to represent decades. In an attempt to be more general and to choose candidates that are likely to be of high posterior probability we generate candidates by sampling a metadata value for each word from the beta distribution for the topic assigned to that word. The mean of the sampled points is also added as a candidate. Finally, to generate a prediction the posterior density is calculated at each of the candidates and the one producing the greatest value is chosen. We found that in the case of TONPT the multimodality of the p(tjzdi) distribution caused this prediction algorithm to perform poorly. For TONPT, predictions are determined by first estimating the d parameter for the test document using samples obtained from the model with the metadata marginalized out, and then using d to estimate the mean of p(tjzdi) as the d weighted average of the means of the topic DPMs. For the TONPT runs, G0 was chosen to be a normal with mean and variance equal to the sample mean and variance for the observed metadata, was 2.0, was 1.0, and m was 1. For all runs, the documenttopic parameter = 0:1, and the topic-word parameter = 0:01. 5.3
Synthetic Data Results The synthetic dataset was created such that there are 2 topics and a vocabulary of 5 words: “common”, “semicommon1”, “semicommon2”, “rare1” and “rare2”. The “common” word occurs with 0.6 probability in both topics, “semicommon1” is slightly more likely than “semicommon2” in the first topic, and slightly less likely in the second topic. The “rare1” word is much more likely in the first topic than the second and “rare2” is much more likely in the second topic than the first, but both are much less common in general than the “semicommon”s. Each topic was given a fixed metadata distribution: t0 t1 0:3 f (50; 7) + 0:7 f (80; 7) f (20; 40) Figure 4 shows how, for one run of the inference procedure, the model was able to separate the two topics and recreate the original metadata distributions with high degree of fidelity. Some runs result in slightly better approximations, while others do worse, but these plots seem to be representative of TONPT’s performance on this task. 5.4
Prediction Results Table 2 shows the performance of the various models for the prediction task with 40 topics, which we found to be a number of topics at which peak performance was observed for most of the models. It can be seen that TONPT is significantly superior on the State of the Union and Reuters data, though TOT does come out ahead on the Enron dataset (but not significantly so). 5.5
Posterior Analysis Figure 5 shows the distribution over time for two topics found during runs of TONPT on the State of the Union dataset. The topic shown in 5a is typical of the majority of the distributions we find (a DPM with only one observed component and thus very close to a simple symmetric distribution). It shows the relatively recent rise in prevalence of the topic of health care in U.S. politics.
The topic shown in 5b is an example of a more complex distribution. This particular example appears to capture several conflicts the United States was involved in during the early 1800s, including The War of 1812 and several conflicts related to the Seminole Wars in Florida (which was a Spanish territory until it was ceded to the U.S. in 1821). 6
We have presented TONPT, a supervised topic model that models metadata using a nonparametric density estimator. The model accomplishes the goal of accommodating a wider range of metadata distributions and, in the case of the datasets that we evaluated against, prediction performance remains competitive with previous models. Future work could extend the model to multivariate metadata, such as temporalspatial data including both timestamps and geolocation information. For example, a multidimensional version of TONPT could be used to capture the development of trends in Twitter data, identifying areas where topics originate and how they spread across the country over time. A multivariate normal component distribution would also capture correlations between metadata elements through a topic covariance matrix.