Many posterior distributions take intractable forms and thus require variational inference where analytical solutions cannot be found. Variational Inference and Monte Carlo Markov Chains (MCMC) are established mechanism to approximate these intractable values. An alternative approach to sampling and optimisation for approximation is a direct mapping between the data and posterior distribution. This is made possible by recent advances in deep learning methods. Latent Dirichlet Allocation (LDA) is a model which o ers an intractable posterior of this nature. In LDA latent topics are learnt over unlabelled documents to soft cluster the documents. This paper assesses the viability of learning latent topics leveraging an autoencoder (in the form of Autoencoding variational Bayes) and compares the mimicked posterior distributions to that achieved by VI. After conducting various experiments the proposed AEVB delivers inadequate performance. Under Utopian conditions comparable conclusion are achieved which are generally unattainable. Further, model speci cation becomes increasingly complex and deeply circumstantially dependant - which is in itself not a deterrent but does warrant consideration. In a recent study, these concerns were highlighted and discussed theoretically. We con rm the argument empirically by dissecting the autoencoder's iterative process. In investigating the autoencoder, we see performance degrade as models grow in dimensionality. Visualization of the autoencoder reveals a bias towards the initial randomised topics.
High dimensional data such as text, speech, images and spatiotemporal data are
typically labelled as big data, not only because of high volumes, but also because
of veracity and velocity. It is for these reasons that unsupervised representations
are becoming more in demand in order to project the data onto a lower
dimensional space that is more manageable. Most often, this involves the computation
of a posterior distribution which comes at a high computational expense. One
such method is topic modelling which infers latent semantic representations of
text. The high dimensional integrals of the posterior predictive posterior
distribution of a topic model is intractable and approximation techniques such as
sampling (Markov Chain Monte Carlo) or optimization (variational inference)
are standard approaches to approximate these integrals. MCMC samples from
the proportionate posterior and is guaranteed to converge to the true posterior
given enough data and computational time [
An alternative approach to sampling and optimization, is to directly map
input data to an approximate posterior distribution. This is called an inference
network and was introduced by Dayan et al.[
In this paper we describe and investigate the implementation of an
autencoder variational Bayes (AEVB) for LDA. We are speci cally interested in the
quality of posterior distributions it produces. Related work [
The structure of the paper is as follows: In Section 2 we provide background theory on LDA and in section 3 we introduce AEVB in LDA before de ning the experiments in Section 4. Section 5 is a dedicated discussion into the the AEVB's performance which is followed by conclusions in section 6.
LDA is probably the most popular topic model. In LDA, each document is
probabilistically assigned to each topic based on the correlation between the words
in each document. The generative process of the LDA is as follows[
L( ; j ; ) = DKL[q( ; zj ; )jjp( ; zjw; ; )]
log p(wj ; ):
(2)
In the above equation, DKL is the Kullback-Leibler divergence and is utilized
to minimize the distance between the variational and posterior distribution[
Before we introduce AEVB, we rst need to de ne autoencoding in general. We
use Figure 2 as a simple illustration. An autoencoder is a particular variant of
neural network; di erent in that the input matrix X is mapped to itself X^ as
apposed to a response variable Y [
From a coding perspective, the latent variables z can be interpreted as code.
The variational posterior can be interpreted as a probabilistic encoder and the
original posterior (p( ; zjw; ; )) as a probabilistic decoder [
DKL[q( ; zj ; )jjp( ; zjw; ; )] + Eq( ;zj ; )[log p(wjz; ; ; )]:
The rst term attempts to match the variational posterior over latent variables to
the prior on the latent variables [
Stochastic gradient descent (SGD) - a scalable variation of regular gradient descent - is the optimization algorithm used to minimize the KL divergence (maximize the ELBO) - stochastic in that it computes an approximate gradient as apposed to a true gradient (from a random sample) to speed computation. After initializing the parameters of interest, gradient descent optimizes a speci ed loss function by iteratively computing the gradient w.r.t each parameter; multiplying the gradient with the learning rate and subtracting the computed quantity from the gradient, formally:
end until convergence; Algorithm 1: Stochastic Gradient Descent (SGD) Input: Training data S, learning rate , initialization Output: Model parameters = ( ; )
0; 0; repeat for (x; y) 2 S do
( @@ L( ; j ; );
The learning rate is normally dynamically adjusted to improve e ciency further. The true gradient can be smoothed by adding regularization term to improve ease of computation. 3.3
Autoencoder Variational Bayes (AEVB) is based on ideas from Variational Inference (VI) to o er a potentially scalable alternative. VI works by maximizing the ELBO (Eq. 2) where q( ; zj ; ) can be thought of as the latent `code' that describes a xed x - thus should map the input x the lower-dimensional latent space. q( ; zj ; ) - the encoder.
Optimizing the objective function tries to map the input variable x to a speci ed latent space and then back to replicated the input. This structure is indicative of an autoencoder - where the name AEVB comes from. ADVI is used to e cient maximize the ELBO - equivalent to minimizing the original KL divergence. ADVI utilizes Stochastic Gradient Descent, but derivatives are not computed numerically (in the traditional sense), nor symbolically, but rather relies on a representation of variables known as dual numbers to e ciently compute gradients (the details of which are super uous to this discussion).
But how do we parameterize q( ; zj ; ) and p( ; zjw; ; )? We ought to choose q( ; zj ; ) such that it can approximate the true posterior p( ; zj ; ), and p( ; zjw; ; ) such that it is exible enough to represent a vast variety of distributions? Parameterizing these functions with neural networks allows for great exibility and are e ciently optimized over large datasets.
q( ; zj ; ) - the encoder - is parameterized such that the code's dimensionality corresponds to a mean and variance of each topic. The parameter space of the decoder is speci ed as the reciprocal of the encoder. In the case of LDA, the weights and biases of the encoder are speci ed as:
W0 w h b0 h 1
W1 h 2 (k 1)
b1 2 (k 1) 1 where w is the number of words, h the number of hidden layers and k the number of topics. A uniform Dirichlet prior with = k (number of topics) is speci ed. The objective (ELBO) is then maximized by ADVI and model parameters are learnt. 4
A number of experiments were conducted in aid of answering the follow research questions: 1. Does the AEVB LDA provide comparable results - and predictive capability - to the well establish - VI learnt - LDA? 2. Does the autoencoder o er signi cant processing-e ciency advantages as datasets scale? 3. What drawbacks does the autoencoder su er? 4.1
The experiments were conducted on the 20 Newsgroups [
An LDA model with w tokens, D documents and K topics requires learning two matrices. describing the topic distribution over documents and
D K K V portraying the word distribution over topics. To learn the topic distribution over documents with an autoencoder we specify the lowest dimensional hidden layers to correspond with the dimensionality of a K dimensional simplex to soft cluster the D documents over the K topics, that is of dimensions K 1. However since we want to learn distributions over the topics - leveraging a standard Gaussian so we specify dimensionality 2 (K 1) to represent a mean and variance for each topic. The model is constructed with 100 hidden layers of these dimensions. A uniform Dirichlet prior is speci ed.
All experiments were conducted on a 2012 Macbook Air with 1,8 GHz Intel Core i5 processor and 4GB of memory. All relevant code is well documented and available here: https://www.zachwolpe.com/research. 4.3
Perplexity is an intrinsic evaluation metric for topic models and an indication of
`how surprised' the model is to see the new document. Recent studies have shown
that perplexity scores and human judgement of topics are often not correlated,
i.e. high perplexity scores might not yield human interpretable topics [
X score(wi; wj ) i<j
Where words W = w1; w2; :::; wn are ordered from most to least frequently
appearing in the topic. The two leading coherence algorithms (UMass and UCI)
essentially measure the same thing [
D(wi; wj ) +
D(wi)
Where Ki is ith topic returned by the model and wi is more common than
wj (i < j). D(wi) is the probability of a word wi is in a document (the number
of times wi appears in a document divided by total documents). D(wi; wj ) is
the conditional probability that wj will occur in a document, given wi is in the
document - which eludes to some sort of dependency between key words within
a topic [
Topic model coherence was used as the primary metric in assessing model performance. To better generalize the ndings we compute coherence for a variety of topics K - measuring the performance as models grow in complexity. Further, to account for the sampling distribution coherence was repeatedly computed (10 times) for the same model with di erence random samples. It is apparent from Figure 3 that although the autoencoder matches the VI's performance under simple textbook conditions (K = 5 topics); as models grow in complexity and dimensionality the autoencoder's coherence scores steadily decline - note that the labels reading pymc3 and sklearn correspond to the AEVB and VI implementations respectively. Although LDA is an unsupervised model, the 20Newsgroups dataset is labelled. So we have the advantage of knowing the true structure of the dataset to be K = 20 - which coincides with the best performance using VI. Tables 1 and 2 provides topic examples of both algorithms respectively. The repetive top words in topics 1 and 2 in Table 2 con rms the AEVB's inability to produce meaningful topics.
A callback function was written to assess the autoencoder's iterative process. The callback samples the distribution per epoch for the purpose of under
D K standing how the autoencoder allocates documents to topics. Since is learnt as a posterior distribution, we need to sample from to assess its current state. This was performed under two variations: 1. 1 sample per epoch: to assess randomness. 2. 100000 samples per epoch: to assess the current state of .
This was conducted for the rst 100 epochs to assess the initial conditions of the autoencoder (with a constant K = 10 latent topics). Figure 4 depicts the topic allocation for a random sample of documents - coloured by the probability density. One would expect the distribution over topics to begin uniformly spread and thereafter narrow to a single topic. However we observe the vast majority of documents are even initially biasedly assigned to the rst few topics, with an increasing density over the rst 100 epochs cementing the allocation, despite a uniform Dirichlet prior.
Inspection of the single sample depicted in Figure 5 would ideally be completely randomized under initial conditions, however it is apparent it yields some bias. Numerically we can assess the distribution of topic allocations in Figure 4. One would expect a far more uniformly spread distribution over the K topics, with the autoencoder exhibiting bias as early as 5 epochs into running the model; which is then exaggerated.
Fig. 5. A single sample per epoch. AEVB for LDA boasts the theoretical advantage of brisk computation, however this allure evades comparisons of quality. In conducting this analysis our contribution is two fold: { Perform a thorough comparison between AEVB and VI posterior inference for LDA, uncovering the bene ts and drawback of the AEVB as an alternative, speedy, approach. { Uncover AEVB's shortcomings in an attempt to deduce bias that limits its performance - in addressing why and where it fails.
After detailed experimentation it is readily apparent that the autoencoder falls short when juxtaposed against established, well engineered techniques. When analyzing the topic coherence performance the autoencoder o ers analogous results to the VI LDA for simple models, however, as models grow in complexity it is unambiguously inferior to established methods - when accounting for sampling variability. Results are only comparable under textbook conditions. We show that the encoder fails to adequately explore the domain spaces and heavily biases initial random conditions. Lower predictive accuracy was also found in the PyMC3 tutorial mentioned earlier (https://docs.pymc.io/notebooks/ldaadvi-aevb.html). More speci cally, in this study the log-likelihood function for topics on held-out words was use as a goodness-of- t test.
Future work includes an implementation of the solutions o ered in [