Topic Modeling algorithms help unveil the latent thematic structure from large document collections. Previous works showed that traditional approaches could be less efective when applied to short texts, e.g., tweets; however, that can be mitigated by assuming that each document is about a single topic, as done in Twitter-LDA. In this work, we relax this assumption and propose a new model where a document can be about single or multiple topics. Our model allows the generation of diverse types of descriptors from latent topics, e.g., words and hashtags, similarly to Hashtag-LDA. Moreover, words/hashtags can be generated from topics or a background/global distribution. The proposed model is modular, and our goal is to tailor it to collections that can be heterogeneous both in the presence of single or multiple-topic documents and in the adoption of diverse topic representations.
Topic Modeling algorithms are Machine Learning approaches introduced to unveil the latent
thematic structure from unstructured document corpora. In Probabilistic Topic Models [
Topic Modeling has been adopted in many tasks and settings [
Besides text length, another issue is the heterogeneity of the descriptors. For instance,
Twitter allows the use of hashtags: an hashtag is a sequence of characters – not including
punctuation or spaces – starting with “#” which “is used to index keywords or topics on
Twitter”.1 Hashtag-LDA [
The overall model in plate notation is depicted in Fig. 1. The model can be considered an extension of LDA, Twitter-LDA, and Hashtag-LDA: the latent structure of LDA is used to model multi-topic documents, while the latent structure of the other two is used to model single-topic documents. We will describe the model in the context of Microblog, e.g., Twitter; however, the model is modular and we plan to apply it to heterogeneous document collections constituted by diverse types of documents, e.g., news, forum posts, blog posts, and tweets.
Our model can be decomposed in four conceptual blocks depicted in diferent colors in Fig. 1.
The first block, highlighted in red, models the key idea underlying our approach: two types of documents can be distinguished, those about a single topic and those about multiple topics. In the model, each user has her own inclination to write document on a single topic or multiple topics; this inclination is encoded in the probability , which afects the type of document written by . This is a simplifying assumption since aspects other than the user might afect the choice of writing on single or multiple topics. Our approach allows diverse types of users – in terms of their inclination on single or multiple topics – to be modeled. For instance, influencers or politicians, through their oficial accounts, usually write long and elaborated posts to express their point of view; other users publish very concise messages, e.g., for answers to other tweets.
The second and third blocks are highlighted in blue and green; they are responsible for the topic assignment to documents, words and hashtags. The assignment depends on the document type identified in the first block: if the document is about multiple topics – blue block –, assignment is very close to that proposed in LDA, where a single topic is associated to each textual element, e.g., a word or an hashtag; if the document is about a single topic – green block –, topic assignment follows Twitter-LDA and Hashtag-LDA, where a single topic is assigned to each document — we will refer to such topic as the main topic. 1https://help.twitter.com/en/using-twitter/how-to-use-hashtags
When a document is about multiple topics – blue block in Fig. 1 –, • a topic proportion, , is assigned to each document , where the th element denotes the importance of topic for document ; • a vector of active topics is assigned to each document : a non-active topic will have a very low weight, thus making very unlikely the observation of words or hashtags associated with that topic; denotes the probability that a topic is active; • a topic is assigned to each word and topics with a larger weight in the document will generate words more frequently; similarly a topic is assigned to each hashtag and topics with a larger weight will generate hashtags more frequently.
In the event of tweets, longer documents will be focused on a limited number of topics and the vector will introduce sparsity in the representation of documents as mixture of topics.
When a document is about a single topic – green block in Fig. 1 –, • a topic proportion, ∗, is assigned to each user and its th element denotes the preference of the user to select the th topic as the main topic; • a main topic, ∗ , is assigned to each document and topics with larger weight in ∗ are assigned more frequently.
In this case, no topic is associated to words and hashtags since they are generated from the main topic. The idea underlying this block is that more simple and concise documents are focused on a single topic and the selection of the topic depends on the personal preference of the user.
The last block is highlighted in orange and is the one responsible for the generation of words and hashtags from the topics. The model considers: • a double representation of topics: there is a fixed number of topics and each topic is represented both as a distribution over words and as a distribution over hashtags; • background words common to all the topics: this group of words is modeled as a “dedicated” topic and therefore is represented as a distribution over the word vocabulary; similarly, global hashtags are used independently from the topic and are modeled as a dedicated topic and represented as a distribution over the hashtag vocabulary.
The generative processes of words and hashtags are basically identical, and the diference lies in the latent variables and the parameters. In the case of words (hashtags), a source, ( ), is assigned to each word
(hashtag ) and indicates if it was generated from a topic or it is a background word (global hashtag). The observed word (hashtag) depends on the source, the type of document, and the topic: if it is a background word (global hashtag), the background distribution ( ) is considered; otherwise, the main topic distribution, ∗ ( ∗ ), or that of — respectively single or multiple-topic document. The two generative processes are identical and independent of each other – the presence of certain words does not afect the presence of hashtags in the same document and vice-versa –; therefore, our approach can be extended with topic representations based on additional vocabularies, e.g., emojis. the topic associated to the word, ( ), is considered, depending on the document type
We are currently focusing on the experimental evaluation of the proposed approach using
Twitter datasets and “generic” short texts [
Our approach achieved results comparable with Twitter-LDA, which was the most efective
baseline in terms of Topic Coherence – TC-PMI and TC-NZ [
The subsequent steps will be: (i) investigate in detail the efect of the number of topics on the
proposed approach; (ii) investigate how to tailor the model to heterogeneous test collections [