In this era, the explosive growth of electronic file archives has attracted a lot of attention. The report predicts that data storage capacity will increase to 40 trillion gigabytes by 2022, 50 times more than in early 2010 [2]. The most important concern now is to determine effective tools or methods that automatically organize, index, search, and browse this unstructured electronic text data. TM is one of the most widely used technology to organize these types of data. TM is a well-known advanced Machine learning technology. Using this technology, we can discover patterns that usually reflect basic themes. Given D is a set of documents composed of a set of terms W and T is a set of latent topics, TM will find T based on statistical inference on the term W. Thus, the document is a mixture of topics where topics represent a statistical distribution of words. A graphical representation of topic modeling is shown in Figure 1. In machine learning and NLP, the topic model is a statistical model used to discover the theme "topic" that occurs in the documents collection. TM is a widely used text mining tool for finding hidden semantic structures in text documents. A popular algorithm for modeling the text data is LDA, and different extensions have been proposed So far: Online variational inference for LDA in [5], Correlated topics Model (CorrLDA) [4], Hierarchical Topic Model (hLDA) [3], etc. Recently, Word2Vec [10] words Embedding has been used for theme extraction and achieved hopeful results [8] [9]. However, the most commonly used topic model is LDA, which provides a powerful framework to extract hidden topics from text documents. But, the researchers found that extracted topics of unsupervised models are unexplainable or meaningless [12]. This is not a problem with LDA only: it is potentially a problem with any extension. Several knowledge-based models have been proposed to address this problem, such as seed-LDA, in which seed words are used as input to guide the model [6]. This model can produce coherent topics of particular interest to users.
Most topic models are designed, developed, and implemented for English text corpus. Therefore, these techniques are very effective for the English corpus. But Urdu has distinct nature from famous languages (such as Chinese, English, Arabic, etc.), and has distinct grammatical forms, Synonyms, antonyms of various words, morphological structure, Semantics, and syntax. Therefore, TM becomes a challenging task in Urdu and the limited contribution is committed to Urdu in NLP. There are many research communities for English and most software, application programming interface (API), and tools are specially developed for English which do not work effectively for Urdu language. To use these tools and software in Urdu language, further work will be required for better performance.
According to the literature review, there is little work on topic modeling in Urdu Language [15][16] [7]. However, there is no work to extract coherent topics in Urdu language and it is first work for the extraction of coherent topics from Urdu documents. In this paper, first, we apply our proposed semi-supervised topic model Seeded-ULDA. Second, we introduce word embedding in Urdu language to discover topics by clustering of semantic space; generated through word2vec word embedding. After intensive examination, the results show that word embedding is unable to extract coherent topics in Urdu language and the semi-supervised model Seeded-ULDA outperforms ULDA based on coherence measure.
In this section, we present the techniques used in this study for topic modeling. We will first focus on using the Word2Vec word embedding model and then discuss the Seeded-ULDA approach. All these techniques are implemented and made the performance comparison in our experiment.
We start with a short explanation of the effectiveness of the Seeded-ULDA. It is considered a challenging task to develop an efficient semi-supervised Urdu topic model "Seeded-ULDA" that combines preprocessing, seeded-LDA and Gibbs sampling [21]. Figure 2 gives a complete overview of the proposed model Seeded-ULDA. We introduce the technologies involved in Seeded-ULDA in the following subsections.
Text preprocessing aims to standardize the representation of texts to improve the accuracy of the topic detecting models. In this step, encoding UTF-8, diacritics removal, tokenization, and stop words removal will be performed to standardize the input dataset.
Enconding UTF-8 Computer programs face the problem of character recognition in Urdu text. We are using Unicode Transformation Format 8 (UTF-8) encoding for Urdu character recognition. Unicode is one of the most widely used encoding in the computer industry. UTF-8 means that every character is mapped by unique variable-width numeric code.
Removal of Diacritics A diacritic is a sign which is added with a letter to change the pronunciation. Urdu diacritics are a subset of Arabic diacritics. The most widely used diacritics in Urdu are Zabar, Pesh, and Zer which are called Aerab [20], and other diacritics are seldom used. When it is attached to a word, the sound and meaning of the word change [19]. Urdu is usually written with words only and diacritics are left to personal choice. But we discard diacritics to form the dataset standardise. Finger 3 shows some Urdu diacritics examples. Stopwords Removal In NLP, stopwords removal is the part of pre-processing to remove unworthy data and unworthy words (data) are regarded as stopwords. They do not make the addition of meaning or information and are found frequently in a sentence. We can safely ignore them without losing the information of the sentence. In order to get meaningful data, we exclude stopwords from our corpus. Few mostly used stopwords of Urdu are shown in figure 4. This approach allows a user to guide the topic discovery process by letting him provide seed words that are representative of the corpus [6]. This model can use the seed words in two ways: to improve both topic-word and document-topic probability distributions. To improve topic-word distributions, the model is set up in which each topic prefers to generate terms that are same to the terms in a seed set. To improve document-topic distributions, the model is encouraged to select document-level topics based on the existence of input seed words in that document. Our work aim is to produce coherent topic. So, we use the first way to improve topic-word probability distribution. In traditional topic models, Multinomial distribution φ k expresses each topic k over words. This notion is extended and the topic is defined as an intermixture of two different distributions: a regular topic distribution and a seed topic distribution. In seed topic distribution, the words are generated from the given seed set. In regular topic distribution, any words can be generated including seed words. It is emphasized that, like ordinary topics, all words of seed topics have non-uniform probability distribution. The model takes a set of seed words as input, and then outputs the probability distribution of these words. For simplicity, the model is explained by considering one-to-one compatibility between reg-ular and seed topics. when regular topics are more, then this consideration can be simply relaxed by making copies corresponding to the seed topics. The figure 5 shows that documents are a mixture of topics T where these topics are a mixture of seed topics φ s and regular topics φ r . The probability of picking a term from the regular topic distribution versus the seed topic distribution is controlled by parameter π k . The graphical notation is shown in Figure 6 and the generative process of seeded-LDA is as follow:
-Draw seed topic φ s k ∼Dir(β s ).
-Draw π k ∼ β(1, 1).
• For each document d, Choose θ d ∼Dir(α)
-Select a indicator x i ∼Burn(π zi ).
if x i = 0, Select a word w i ∼Mult(φ r zi ). // choose from regular topic. if x i = 1, Select a word w i ∼Mult(φ s zi ). // choose from seed topic.
When direct sampling is hard, then Gibbs sampling is used to get a sequence of observations by Markov Chain Monte Carlo (MCMC) which is a methodology to sample from statistical distribution [17][18]. It is an algorithm that uses the logic of randomness means it sample randomly, and is widely employed as a statistical inference. In this paper, we apply Gibbs sampling with the probabilistic generative model to discover coherent topics.
The Word2vec model is a two-layer neural network that can be trained to recreate the language context of words [10]. It captures the context, semantic and syntactic similarity of words in a document. Word2Vec is one of the most widely used method of learning word embedding using shallow neural networks. It takes a large number of text corpora as input, and generates a vector space that can have hundreds of dimensions, and assigns a corresponding vector in the space to each unique word in the lexicon. This technique is different from other topic models, which use documents as context. Word2Vec learns the distributed representation of each target word by identifying the context as surrounding terms.
We presented some experiments to demonstrate the effectiveness of the above defined topic modeling techniques. These experiments were performed using two corpuses discussed in the following section.
Urdu language is not owning any benchmark dataset for NLP tasks. Therefore, we create our own corpus, which contains text articles in Urdu of a widely known news website https://www.express.pk. It is now publicly available at https://github.com/Mubashar331/Urducorpus for research purposes. The collected dataset has five categories of documents named Health, Sports, Science, Entertainment, and Business. We also collected a dataset of English having four categories. After the completion of preprocessing steps that are discussed in the above subsections, we applied above defined topic modeling techniques on these corpora and evaluated performance. Table 1 is briefly describing the corpus.
We presented three experiments to evaluate the performance of topic modeling techniques. These experiments are performed on dataset or corpus discussed in above subsection.
The purpose of this experiment is to evaluate the accuracy of word2vec on Urdu text documents. Word2vec has two types of architecture, Skip Gram Model and Continuous Bag of Words (CBOW), to gain vectors of features. In this study, we use the CBOW model to build vectors of a given pre-processed dataset. Then, we cluster the gained vectors of features using K-means method. The process of topic modeling by word2vec is shown in figure 7. In this experiment, we compare the accuracy of our proposed model with the ULDA model proposed in an article [15]. This model was proposed to discover topics from a corpus of Urdu news articles. We employ the ULDA model on our own corpus. We ran this model several times to evaluate results on topics discovered from Urdu text documents.
In the NLP research community, evaluation of discovered topics from the topic model is considered an open challenge [13]. Some researchers use internal evaluation methods such as perplexity or likelihood for evaluating topic models. But these evaluation methods cannot measure the consistency of discovered topics. Through large-scale user studies, an author [13] argued that the topic model which performed well on perplexity or likelihood failed to produce coherent topics. Therefore, we do not use these evaluation methods to evaluate above defined topics models.
We evaluate topic models by using a manual evaluation technique named Coherence Measure (CM), it is the ratio of relevant words to total candidate words of a topic [14]. For this manual evaluation technique, we take 20 words with the highest values of every topic and request 5 students to examine and label them. First, they need to examine the words to label a topic as interpretable or irrelevant. When the topic is interpretable, then they need to inquire about the words of a topic that are relevant. CM is calculated using equation 1 where x is relevant words and n is total candidate words.
We presented the experiments to demonstrate the effectiveness of topic modeling techniques based on CM. First, we find topics by word2vec and mostly extracted topics are labeled irrelevant. Second, we extract five topics by LDA and ULDA from Urdu corpus and find out that all topics extracted by LDA are labeled irrelevant. We show seven words of one topic in figure 9 which is extracted by LDA and ULDA. Topic extracted by LDA is irrelevant and does not belong to any class of our given dataset. But Topic extracted by ULDA is relevant and belongs to the Health class of our given dataset. Finally, we apply LDA and sedded-LDA topic modeling techniques to the English corpus and find out that all extracted topics are labeled irrelevant. Then, we combine both models with Gibbs Sampling(GS) and extract topics from the English corpus. The results demonstrate that topics extracted by seeded-LDA(GS) are more coherent as compare to LDA(GS). As shown in table 2, the topic extracted by LDA and seeded-LDA does not belong to any class of given corpus. But the topic extracted by LDA(GS) and seeded-LDA(GS) belong to the health class of the English corpus.
Regular unsupervised topic models might not produce coherent topics due to their pure unsupervised nature. Several knowledge-based topic models have been proposed to address this problem, but most of them are for English. Therefore, NLP research involving the Urdu language is comparatively hard, compared to other popular languages, due to the speciality of Urdu such as syntax, semantics, and morphological structure. The main motivation behind this research is the lack of NLP resources and tools for the Urdu language. There are no important studies for extracting coherent topics from Urdu texts in literature. To meet the challenges of Urdu, we have proposed a topic model Seeded-ULDA for Urdu language to produce coherent topics. In order to evaluate the performance and effectiveness of our proposed Seeded-ULDA model, we conducted three experiments using the Urdu dataset generated by ourselves. The results demonstrate that unsupervised models produce less coherent or meaningless topics compared to semi-supervised framework. First, we apply word2vec word embedding and result shows that it is unable to extract coherent topics. In the second and third experiments, we apply Seeded-ULDA and ULDA respectively and results show that semi-supervised model Seeded-ULDA produces more than 39% coherent topics compared to unsupervised model ULDA.