Text clustering and topic learning are two closely related tasks. In this paper, we show that the topics can be learnt without the absolute need of an exact categorization. In particular, the experiments performed on two real case studies with a vocabulary based on bigram features lead to extracting readable topics that cover most of the documents. Precision at 10 is up to 74% for a dataset of scientific abstracts with 10,000 features, which is 4% less than when using unigrams only but provides more interpretable topics.
Text clustering is a huge research area with many applications, such as corpus
visualization [
Despite all these numerous works, it turns out that some confusion often subsists
between the task of text clustering (grouping similar texts, i.e. working on category’s
extension) and the task of topic identification (extracting within-category
commonalities, their intension), as highlighted by [
First, we show that a minimum number of features is necessary but sufficient to
achieve a good accuracy, both in term of clustering purity and topic description. It is
not as obvious as it seems since too many features might add noise and reduce the
generalization ability of the model, which actually happens in supervised settings [
Second, we show that the bigram-based vocabulary provide really useful topic
descriptions at the cost of a reasonable decrease in accuracy. The cost is not that important
with a drop of about 10%. Our results highlight that a careful choice for the features
allows a much better interpretation of the topics given by topic learning techniques (here,
LDA). This work is closely related to the task of topic labeling but, here, the
descriptive features are defined before the topic learning step. Therefore the extracted topics
are characterized by the very terms that constitute their backbone, and not labeled by
using one among the many heuristics proposed in the literature [
The paper is organized as follows. Section 2 defines the two complementary tasks of text clustering and topic identification, highlighting their close connection but also their difference. Section 3 shows the impact of making the vocabulary change in term of both size and nature (unigrams versus bigrams). Finally, we conclude and suggest future work in Section 4. 2 2.1
The first step consists in showing the clear distinction between the two tasks. As
illustrated in Fig. 1 (left), text clustering mainly aims at categorizing objects into clearly
separated clusters. Even though the membership can be gradual, like in fuzzy clustering
[
By adopting a different viewpoint, topic identification is more dedicated to extracting a set of topics that structure the dataset as shown in Fig. 1 (right). Topics can be viewed as weighted lists of keywords (e.g., with LSA) or distributions over words (e.g., with LDA). In order to give an overview of the whole corpus to the final users, the usual solution is to keep the top words (option 1 in Fig. 1) or the top n-grams (option 2 in Fig. 1, here with n=2).
Obviously, the two tasks are related but not fully aligned. Hence, the documents da, db and de can be misclassified as long as we find the expected topics, more identifiable on the colored groups of documents in Fig. 1. Let us note that we might easily get the top frequent terms for each cluster as a post-processing stage. However, most of the current state-of-the-art methods such as LSA, NMF and LDA address both tasks at the same time, which explains the confusion that may arise.
cluster 1 da dc cluster 2 db dd dc da de db dd topic 3 topic 2
1 data mining algorithm clustering learning paper approach classifica6on results method (…)
2 data mining 6me series experimental results knowledge discovery machine learning nearest neighbor support vector feature selec6on decision tree associa6on rule (…) de cluster 3
topic 1 --( text clustering )-
--( topic iden6fica6on )-
Several previous works have used n-grams either during the topic learning process
[
In the following sections, we experiment LDA on two datasets in order to address the two tasks simultaneously. For evaluation’s sake, we compare the output given by LDA to a gold standard that provides us the real class label of each object. In order to get a partition, we associate each text d to the most likely topic zˆ = arg max p(z/d). This way, we reduce the expressive power of topic models but we can leverage the usual Adjusted Rand Index (ARI) for assessing the clustering quality. We will see in the next section that the two datasets have been precisely chosen because they fit this crisp clustering assumption. The maximum of 1 with ARI is achieved with a perfect match between the partition and the gold standard.
Establishing a ground truth for the topic identification task is much more challenging. To begin with, we choose to restrict the evaluation to the quality of the top-10 terms associated to each cluster for 10 is the number usually shown to end users. Although we can imagine various ways to extract those lists from the gold standard partition (e.g., selecting the most discriminant terms, etc.), we have chosen to restrict to the most frequent terms for this study. In addition to the simplicity of this solution, we will see that the probability p(t/z) of the term t given the topic z output by LDA clearly favor frequent terms. We then propose to calculate the usual precision for this top-10 terms, noted pre@10. Let us remark that this manner to challenge the list of top K terms is especially uncommon in the literature, in which the list is always manually evaluated. The mapping between the true category and the topic z is simply derived by taking the category with the higher number of texts related to z. Obviously, pre@10 ranges from 0 (no common term) to 1 (perfect match between the two lists). 2.3
dataset #c #docs #unigrams #bigrams ART 5 18,465 20NG 20 18,828
The two datasets are the set of scientific abstracts gathered by Tang. et al. [
In our context, we extract bigrams based on classical patterns in terminology extraction domain (i.e. noun-noun, adjective-noun, and so forth)5. Terms extracted from our corpora are then ranked depending on their relative frequency. Other weightings have been experimented (e.g., TF-IDF, Okapi, C-value) but it turns out that the frequency is the more adapted ranking function for both tasks addressed in this study6. 3
We here focus our attention on the importance of vocabulary size and term nature (unigrams versus bigrams). We used the parallel LDA implemented in the MALLET package7. The priors ↵ and are not automatically estimated (default configuration) but we set them both to 0.1 after a preliminary grid search8. We set the maximum number of iterations for the Gibb’s sampling to 2000, as suggested with this implementation, and run the algorithm ten times. The final mean is only given since the observed standard deviation does not exceed 0.01, so we decided not to overload the figures.
We keep the most frequent K words, K ranging from 500 to 30,000. We then
compute the quality of LDA topics both for clustering (Fig. 3) and topic identification
4 http://arnetminer.org/collaboration and http://qwone.com/ jason/20Newsgroups/
5 To this end, we used the biotex tool [
In addition, we observe that with three times the number of features, bigram-based vocabulary is able to achieve a really good ARI score for ART, not very far from the maximum with unigrams (0.3865 against 0.4346). This is clearly not the same situation for 20NG with a difference of about 0.26 for the ARI.
We now take a closer look at the top terms returned by LDA, in comparison to the reference terms extracted from the true classes. The precision achieved by keeping the top-10 terms is shown in Fig. 4. We have been really surprised to notice that the 9 We stopped the size for ART-1g at the number of words occurring at least twice in the whole corpus (13,778 words). datamining (ART) sci.space (20NG) rec.sport.baseball (20NG) 1-grams 2-grams 1-grams 2-grams 1-grams 2-grams data data mining* space solar system* writes red sox* mining data sets* earth henry spencer* game san francisco (15) algorithm association rules* launch physical universe (30) article los angeles (18) clustering time series* writes night sky* year st louis* paper data streams (13) shuttle space shuttle* team world series* approach experimental results* nasa toronto zoology* games major league* learning knowledge discovery* mission oort cloud (12) good blue jays* classification data set* orbit jet propulsion* players power play algorithms machine learning (12) system dick dunn (25) baseball mark singer* results support vector* solar high-speed collision (26) time san diego (12) top-10 bigrams are really competitive in comparison to the top-10 words. For ART, the precision achieved is 4% to 7% below only with a score of about 70% (up to 74% for 10,000 terms). We believe that this is a crucial observation: even though we get one term less that with single words in average, the topics are much more readable by using seven bigrams than height unigrams. The bigram “data mining” is more informative than the two words “data” and “mining”, even if they are given in the same list.
This advantage is obvious if we take a look at the top terms given in the table of Fig. 5 (top bigrams next to top unigrams). For 20NG, it is even more interesting: despite the ARI collapse, four of the top bigrams are still accurate (six for the unigrams). However, it does not mean that the other terms are unrelated. We have highlighted the terms related to the ground truth with a * in Fig. 5, and noted their rank in the true list otherwise. Let us note that a vocabulary of 500 terms is sufficient to achieve such performances in term of precision. It seems that LDA easily finds the core of topics, without caring much for the result of the text clustering task.
Finally, we have run a last series of experiments in order to see the impact of a mixed unigram-bigram vocabulary. To this end, we fixed the number of features to 10,000 and changed the proportion in steps of 5% (e.g., 80% unigrams with 20% bigrams). The results confirm that bigrams might help increasing the overall clustering accuracy, but the bonus is limited and not significant. The best proportion seems to be highly dependent of the dataset (e.g., we got +0.01 ARI for ART with 5% of bigrams and +0.024 for 20NG with 30%). However, we observed no constant improvement for the precision. When we take a closer look to the top terms, bigrams are overwhelmed by unigrams, which explains the unchanged score. 4
Despite all the work done so far for integrating phrases into topic learning, we believe that this study is the first to highlight the potentiality of bigrams, not only for improving topic homogeneity (in addition to unigrams) or topic labeling, but for the whole task of topic identification. Even though we have observed a clear gap between unigram and bigram frequencies, the bigram frequency seems to be sufficient to cover most of topic’s aspects, getting rid of the ambiguity carried by unigrams. Hence, it is easy to provide readable topics to end users with a limited energy in the creation of terms (actually, any bigram library is expected to provide interesting features). Our preliminary experiments have shown that this reasoning can be transposed to trigrams as soon as their cumulated frequency is sufficient. We observed a decrease of about 10% for the pre@10 with trigrams (60% for ART and 30% for 20NG).
Interesting work lies ahead. One immediate follow-up is to design a new method that directly focuses on topic identification. By even more weakening our expectations on text clustering, we can find a way to improve the top K terms by favoring the topical core of each category (colored areas in Fig. 1). Improving the input representation, for instance by adding pseudo-counts for complex terms, can be a way to explore this idea. Another exciting, more theoretical question is to question the tradeoff between term frequency, term co-occurrences and performances. During our experiments, we observed a clear logarithmic correlation between the total number of tokens and the performances we can achieve (from R2 = 0.94 for 20NG until R2 = 0.98 for ART described with bigrams). This tells us that we cannot expect much by using too rare terms since they lead to really sparse matrices. However, it seems that the combination of complementary rare terms can compete with more frequent words. Information theory might be used for studying this kind of issues further.