English. OCTIS is an open-source framework for training, evaluating and comparing Topic Models. This tool uses singleobjective Bayesian Optimization (BO) to optimize the hyper-parameters of the models and thus guarantee a fairer comparison. Yet, a single-objective approach disregards that a user may want to simultaneously optimize multiple objectives. We therefore propose OCTIS 2.0: the extension of OCTIS that addresses the problem of estimating the optimal hyper-parameter configurations for a topic model using multi-objective BO. Moreover, we also release and integrate two pre-processed Italian datasets, which can be easily used as benchmarks for the Italian language.
Italiano. OCTIS e` un framework opensource per il training, la valutazione e la comparazione di Topic Models. Questo strumento utilizza l’ottimizzazione Bayesiana (BO) a singolo obiettivo per ottimizzare gli iperparametri dei modelli e quindi garantire una comparazione piu` equa. Tuttavia, questo approccio ignora che un utente potrebbe voler ottimizzare pi‘u di un obiettivo. Proponiamo percio` OCTIS 2.0: l’estensione di OCTIS che affronta il problema della stima delle configurazioni ottimali degli iperparametri di un topic model usando la BO multi-obiettivo. In aggiunta, rilasciamo e integriamo anche due nuovi dataset in italiano preprocessati, che possono essere facilmente utilizzati come benchmark per la lingua italiana.
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Topic models are statistical methods that aim to
extract the hidden topics underlying a collection
of documents
Recently, OCTIS
Contributions. In this paper, we propose OCTIS 2.0, an extension of the existing framework that integrates both a single-objective and multi-objective hyper-parameter optimization strategy, using Bayesian optimization. Moreover, we also pre-process and include two novel datasets in Italian. We will then briefly show the potentiality of the extended framework by comparing different topic models on the new released Italian datasets. We believe these resources can be useful for the topic modeling and NLP communities, since they can be used as benchmarks for the Italian language. 2
2.1
OCTIS
We will now briefly describe the two components that we will extend in this work: the preprocessed datasets and the hyper-parameter optimization module.
Pre-processing and Datasets. OCTIS currently
provides functionalities for pre-processing the
texts, which include the lemmatization of the text,
the removal of punctuation, numbers and
stopwords, and the removal of words based on their
frequency. Moreover, the framework already
provides 4 pre-processed datasets, that are ready to
use for topic modeling. These datasets are 20
NewsGroups,1 M10
All the currently provided datasets are in English. OCTIS already provides language-specific pre-processing tools (e.g. lemmatizers for multiple languages), but it does not present datasets in other languages. Creating benchmark datasets for 1http://people.csail.mit.edu/jrennie/2 0Newsgroups/
2https://github.com/shiruipan/TriDNR/
tree/master/data
other languages is useful for investigating the
peculiarities of different topic modeling methods.
Single-Objective Hyper-parameter
Optimization. OCTIS uses single-objective Bayesian
Optimization
The use of a single-objective approach is however limited. In fact, this strategy disregards other objectives. For example, a user may require to optimize the coherence of the topics and their diversity at the same time. 2.2
Since OCTIS provides only English datasets, we extend the set of datasets by including two new datasets in Italian. We build the two datasets from the Italian version of the Europarl dataset3 and from the Italian abstracts of DBPedia.4 In particular, we randomly sample 5000 documents from Europarl and we randomly sample 1000 Italian abstracts for 5 DBpedia types (event, organization, place, person, work), for a total of 5000 abstracts.
We preprocess the datasets using the following strategy: we lemmatize the text, we remove the punctuation, numbers and Italian stop-words, we iflter out the words with a document frequency higher than the 50% and less than the 0.1% for Europarl and 0.2% for DBPedia and we also remove the documents with less than 5 words. These values have been chosen by manually inspecting the resulting pre-processed datasets.
We report the most relevant statistics of the 3https://www.statmt.org/europarl/ 4https://www.dbpedia.org/resources/on tology/ novel Italian datasets in Table 1. Following the original paper, we split the datasets in three partitions: training (75%), validation (15%), and testing (15%).
Num. of
documents
From Single-objective to Multi-objective
Hyper-parameter Bayesian Optimization.
Given the limitations of the single-objective
hyperparameter optimization approach, we
extend OCTIS by including a multi-objective
approach
Since the original Scikit-Optimize library does
not provide multi-objective optimization tools, we
use the dragonfly library5
The snippet will run a multi-objective optimization experiment that will return the Pareto front of the diversity and coherence metrics on the Italian dataset DBPedia by optimizing the hyperparameters (defined in a configuration file) of LDA with 25 topics.
In keeping with the spirit of the first version of
OCTIS, the framework extension is open-source
and easily accessible, in order to guarantee
researchers and practitioners a fairer, accessible
and reproducible comparison between the
models
In the following, we will show the capabilities of the extended framework on the new datasets by carrying out a simple experimental campaign.
We assume an experimental setting in which a topic modeling practitioner is interested in discovering the main thematic information of the two novel datasets in Italian. However, the user does not have prior knowledge on the datasets, therefore does not know which topic model is the most appropriate. Moreover, the user aims to get topics which are coherent and make sense together but which are also diverse and separated from the others. Let us notice that a user could consider a different set of metrics to optimize, by selecting one of the already defined metrics available in OCTIS or by defining novel metrics. 3.1
We briefly describe the two evaluation metrics (one of topic coherence and one of topic diversity) that we will target as the two objectives of the multi-objective Bayesian optimization. Both metrics need to be maximized.
IRBO
We focus our experiments on four well-known
topic models that OCTIS already provides, two
of them are considered classical topic models
and the others are neural models. In
particular, we trained Latent Dirichlet Allocation
α prior β prior
L1-L2 ratio [0,1] Initialization method nnnnddssvvdd,ar, rnannddsovmda, Regularization bVomthatrix, H matrix,
We summarize the models’ hyper-parameters
and their corresponding ranges in Table 2. For
each model, we optimize the number of topics,
ranging from 5 to 100 topics. We select the
ranges of the hyper-parameters similarly to
previous work
Regarding LDA, we also optimize the hyperparameters α and β priors that the sparsity of the topics in the documents and sparsity of the words in the topic distributions respectively. These hyper-parameters are set to range between 10− 3 and 10− 1 on a logarithmic scale.
The hyper-parameters of NMF are mainly related to the regularization applied to the factorized matrices. The regularization hyper-parameter controls if the regularization is applied only to the matrix V , or to the matrix H, or both. The regularization factor denotes the constant that multiplies the regularization terms. It ranges between 0 and 0.5 (0 means no regularization). L1-L2 ratio controls the ratio between L1 and L2-regularization. It ranges between 0 and 1, where 0 corresponds to L2 regularization only, 1 corresponds to L1 regularization only, otherwise it is a combination of the two types. We also optimize the initialization method for the two matrices W and H.
Since ETM and CTM are neural models, their hyper-parameters are mainly related to the network architecture. We optimize the number of neurons (ranging from 100 to 1000, with a step of 100). For simplicity, each layer has the same number of neurons. We also consider different variants of activation functions and optimizers. We set the dropout to range between 0 and 0.9 and the learning rate, that to range between 10− 3 and 10− 1, on a logarithm scale. We fix the batch size to 200 and we adopted an early stopping criterion for determining the convergence of each model.
Moreover, only for CTM we also optimized the
momentum, ranging between 0 and 0.9, and the
number of layers (ranging from 1 to 5).
Following
For all the models, we set the remaining parameters to their default values. Finally, we train each model 30 times and consider the median of the 30 evaluations as the evaluation of the function to 6Let us notice that there is not a Sentence BERT-like model for Italian. Therefore we used a multilingual one: distiluse-base-multilingual-cased-v1. be optimized. We sample the n initial configurations using the Latin Hypercube Sampling, with n equal to the number of hyperparameters to optimize plus 2 to provide enough configurations for the initial surrogate model to fit. The total number of BO iterations for each model is 125. We use Gaussian Process as the probabilistic surrogate model and the Upper Confidence Bound (UCB) as the acquisition function. 4
In the following, we report the results of the comparative analysis between the considered models on the Italian datasets.
We jointly consider the results of both objectives by plotting the Pareto frontier of the results of topic diversity and topic coherence. Figure 1 shows the frontier of each model for the pair of metrics (NPMI, IRBO). We can notice that the topic models have similar frontiers in each dataset. The most competitive models are NMF and CTM. In particular, NMF outperforms the others for the topic coherence but gets a lower coherence as the diversity increases. Therefore, CTM is the model to prefer if a user wants to get totally separated topics but good coherence. Instead, LDA and ETM have lower performance than the others. We also noticed from our experiments that the performance of ETM is affected when the documents are shorter (on the Europarl dataset), often originating the phenomenon of mode collapsing, i.e. obtaining all the topics equal to the others. 4.2
In Table 3 we report an example of topics discovered by the models. We selected the best hyperparameter configuration discovered by the models with 5 topics and randomly sampled a model run among the 30 runs. Let us notice that, for the sake of simplicity, we have to fix the number of topics here and select a run among the total of 30 runs. Therefore, the qualitative results reported in Table 3 may not reflect the overall results.
We can notice that NMF obtains more coherent
and stable topics. CTM and LDA obtain topics
that have a higher variance: in particular, CTM
discovers a topic (the fourth one, NPMI=-0.51)
that lowers the average coherence, while LDA
discovers a topic (the second one, NPMI=0.48) that
effectively increases the average coherence. On
the other hand, the topics discovered by ETM are
more stable but have a lower coherence on
average. As already observed in previous work
In this paper, we presented OCTIS 2.0, the extension of the evaluation framework OCTIS for topic modeling. This tool can now address the problem of estimating the optimal hyper-parameter configurations of different topic models using a multiobjective Bayesian optimization approach. Moreover, we also released two novel datasets in Italian which can be used as benchmark datasets for the Italian topic modeling and NLP communities.
We conducted a simple experimental campaign to show to potentiality of the extended framework.
We have seen that using a multi-objective hyperparameter optimization approach allows us not only to identify the best performing model over the othde album pubblicare italiano the uniti situare fondare universita` noto torneo giocare tennis edizione tour atp ambito open categoria cemento iflm pubblicare the album serie musicale venire statunitense rock band guerra battaglia venire situare statunitense spagnolo partito esercito distretto mondiale comune campionato squadra abitante calcio regione situare societa` francese vincere comune abitante dipartimento regione situare francese alta distretto est grand torneo giocare tennis tour atp open edizione ambito categoria cemento album pubblicare studio the musicale statunitense records singolo cantante rock calciatore ruolo allenatore calcio centrocampista difensore attaccante portiere settembre aprile contea america uniti situare comune censimento designated census place capoluogo album the pubblicare band statunitense singolo brano of musicale rock superare argentino calciatore el buenos maria en svezia situare chiesa partito battaglia guerra venire politico de linea isola stazione regno st stella vendetta dollaro robert company ritorno west superiore soggetto edizione tennis giocare torneo vincere tour campionato maschile disputare squadra sede de italiano fondare nome azienda noto francese compagnia parigi guerra partito battaglia venire nord politico tedesco esercito regno militare torneo situare comune giocare abitante edizione tennis tour regione uniti iflm serie the dirigere gioco pubblicare statunitense televisivo venire romanzo album pubblicare campionato squadra musicale the calcio statunitense singolo vincere -0.05 0.48 0.11 -0.14 -0.03 0.29 0.48 0.29 0.24 0.39 ers, thus guaranteeing a fairer comparison among different models, but also to empirically discover the relationships between different objectives.
As future work, we aim to extend the framework by considering additional datasets in different and possibly low-resource languages, which require different pre-processing strategies and would allow researchers to investigate the peculiarities of different topic modeling methods.