With the increasing availability of large-scale textual data in various domains, understanding the evolution of topics over time has become crucial for extracting meaningful insights and capturing the dynamic nature of information flow. In this work, we propose a model based on Mixtures of Normalizing Flows, able to extract topics from a collection of time-varying documents in a dynamic way. Our model takes as input embeddings generated by a pre-trained transformer-based language model for each timestamp, and learns the parameters of a complex high dimensional mixture density distribution, grouping documents into clusters, each representing a diferent topic. The parameters of the mixture distribution are updated at each timestamp and topic representations are generated using the TF-IDF procedure. A preliminary analysis shows that the topics generated using this procedure are competitive with those generated by state-of-the-art dynamic topic modelling.
Topic models are useful tools for uncovering hidden thematic patterns and structures within a
collection of documents. Among the many models proposed, Latent Dirichlet Allocation (LDA)
[
from very diferent time instants would be generated by the exact same distribution. Thus, in this paper we aim to design an architecture that would enable the modeling of topics that rise and fall in popularity in time. Alternatively, topics that emerge and disappear, enabling the exploration of the relationship between topics and external temporal factors. In fact, unlike traditional static TOPIC 1 Find N most Relevant Words
-0.14 -0.52 -0.41 -0.4 -0.54 0.32 -0.50 0.50 0.50 0.5 BERT -0.45 0.13 -0.05 0.32 0.49 -0.24 0.16 -0.05 -0.10 -0.54 2.68 0.85 1.34 1.05 0.04 -0.37 1.03 -0.90 -1.12 0.45
TOPIC 2 Find N most Relevant Words
CLUSTERING MODEL TOPIC 3 Find N most Relevant Words
Timestep 1
Timestep 2
Timestep 3
topic models, Dynamic Topic Models (DTMs) are designed to capture the temporal dynamics and
changing patterns of topics within a document collection [
word and document embeddings clustering [
In this work, a BERTopic-inspired approach is discussed and the pipeline of work is summarised in Fig. 1 and in depth explained in Section 2. Unlike BERT in which clustering of documents is carried out independently of their temporal nature, in the approach we propose the idea is to study how the mixture distributions evolves over time (i.e. how components of the mixture, which represent topics, change progressively) using time-varying collection of documents.
lfexible and tractable approaches for modeling complex data distributions. In this work, we
present a novel approach based on Mixture of Normalizing Flows [
transformed distributions, where each component in the mixture is associated with a particular latent variable. The latent variable determines the choice of the component in the mixture, and the transformed distribution captures the data distribution after applying a series of bijective transformations.
Consider a random variable ∈ R representing the observed data, and a latent variable that follows a categorical distribution with categories. Mixture of Normalising Flow is defined as:
() = ∑︁ ( = ) (| = )
=1 where ( = ) represents the prior probability of latent variable taking the value , and (| = ) represents the likelihood of observed data conditioned on the value of = .
Each (| = ) is modeled as a transformed distribution, typically using a normalizing
lfow. A normalizing flow is a series of invertible transformations applied to a base distribution,
such that the resulting transformed distribution captures the complex data structure. For our
implementation we chose a Masked Autoregressive Flow (MAF) [
(; ) = 1 1(; 1) + . . . + (; ) where the components are fixed a priori. In practice, Mixture of Normalizing Flows are trained by maximizing the log-likelihood of the observed data using techniques such as maximum likelihood estimation or variational inference.
incrementally as new time steps as it is processed. At each time step, the documents in the collection are fed into the existing model to learn the distribution. The initial model is trained using the first time step, and subsequent time steps update the model incrementally. The scheme used to train our model and update the parameters of the distribution over time is described in
Algorithm 1 Training Process
layer consists of a mixture of normalizing flows capturing specific topics.
the parameters of the mixture of normalizing flows using gradient-based optimization to maximize the likelihood of the observed data.
highest likelihood.
into a single large document. Once the large documents have been created, a TF-IDF [
knowledge. Add a new layer to the model to capture the topics in the next time step.
each time step, identifying dominant topics through clustering, extracting representative words, and incrementally updating the model by freezing previous layers and adding new ones. This iterative process allows to capture topics evolution and to analyse temporal dynamics in the document collections.
obtained with those in BERTopic’s Table 3 paper with respect to the Tweet’s Trump dataset (a
collection of 44’253 tweets, excluding retweets, from 2009 to 2021) in which BERTopic and LDA
Sequence (inspired by [
• Topic Coherence [
I-MoNF LDA Sequence BERTopic Topic Coherence 0.072 0.009 0.079
Topic Diversity 0.818 0.715 0.863
lfows, whose parameters dynamically evolve over time. One significant advantage of this clustering model is its probabilistic nature, allowing us to infer the characteristics of topics not only from the generated qualitative representations but also from the parameters of the learned distribution over time. For instance, a change in the estimated mean could signify topic evolution, while a small variance might represent a cohesive cluster, that is a coherent topic.
intermediate dimensionality reduction step.
Even if preliminary results from our analysis demonstrate that the implemented model performs to the level of the state-of-the-art dynamic topic models, future developments are to be considered to be able to scale the model to larger collections of data and to improve the topic representation. More specifically: • It is necessary to evaluate the model on datasets traditionally used to assess the quality of dynamic topic modeling models, to properly compare the model performance with recognised standards. • In order to obtain more consistent topic representations, a sliding window approach would be useful. • Considering the rise and fall of a new topic, think of adjusting the number of mixture components based on the parameters provided by the learned distribution at each iteration. • Consider exploring alternative deep probabilistic models instead of normalizing flows, i.e., mixture of density networks, which are faster learning approaches.