Topic modeling techniques has been applied in many scenarios in recent years, spanning textual content, as well as many different data sources. The existing researches in this field continuously try to improve the accuracy and coherence of the results. Some recent works propose new methods that capture the semantic relations between words into the topic modeling process, by employing vector embeddings over knowledge bases. In this paper we study various dimensions of how knowledge graph embeddings affect topic modeling performance on textual content. In particular, the objective of the work is to determine which aspects of knowledge graph embedding have a significant and positive impact on the accuracy of the extracted topics. In order to obtain a good understanding of the impact, all steps of the process are examined and various parameterization of the techniques are explored. Based on the findings, we improve the state of the art with the use of more advanced embedding approaches and parameterizations that produce higher quality topics. The work also include a set of experiments with 2 variations of the knowledge base, 7 embedding methods, and 2 methods for incorporation of the embeddings into the topic modeling framework, also considering a set of variations of topic number and embedding dimensionality.
In the current age of information, larger and larger amounts of data are generated and collected every second around the world. A significant portion of this data is in the form of textual content.The need for understanding this vast amount of textual content keeps increasing as everything in the world becomes more data-driven but mostly because of the fact that it’s impossible for us to do it manually.
The fields of Natural Language Processing and Machine
Learning offer automated to understand large amounts of
textual data. Vector representations of words
Their approach improves the results with one specific method to obtain the word representations, but it’s not clear whether vectors obtained through other methods that can capture better semantics of networks are able to boost the accuracy of topic modeling. The vector embedding methods that have proven to be more successful in other fields such as Link Prediction can possibly capture the semantics of the external knowledge base more accurately.
Another question that remains to be answered in this context is whether a larger knowledge base in terms of entities or a denser knowledge base in terms of relations between entities can also contribute to better representations of words. The primary motive to this question lies in the fact that the knowledge graphs do not have the complete semantic representation of the real world, and can be improved with different relations between entities.
This paper presents two approaches to improve Topic
Modeling. The first approach applies various
Multirelational Network Embedding Methods by computing
the vectors on the same network, and incorporating the
results into the topic modeling framework that has been taken
as the base method of this work. The mentioned
embedding methods all follow a translation-based approach to
vectors with incremental improvements over the original work
which is TransE
The second approach modifies the network of the knowledge graph itself, and manages to significantly increase the density of the network by adding syntactic dependency relations between words in a sentence that are computed from the same text corpus used for the topic modeling. This combination is performed by computing the dependency trees of the sentences in the text corpus, and adding each relation to the knowledge graph between the corresponding entities, thus updating and enlarging the network. It studies the knowledge encoded by this denser network in terms of relations between entities, and how it affects the overall performance of embeddings, and consecutively topic modeling.
The paper is organized as follows: Section 2 presents the related works. Section 3 contains the details of the employed methods that are used in this paper. Section 4 describes the source codes and implementations of the used methods. Section 5 presents the results of the experiments and discusses these outcomes. Section 6 concludes and draws some possible future work.
In this section, the existing works in the literature that are
discussed constitute the basis for the main focus and
direction of this paper. KGE-LDA
KGE-LDA
KGE-LDA extends two entity topic models, namely
CILDA
LF-LDA, which stands for Latent Feature LDA, aims to
improve topic modeling by incorporating latent feature
vectors with a similar point of view as KGE-LDA. The
difference is that, apart from being published before KGE-LDA,
this paper obtains the latent feature representations directly
from the text corpus itself. It uses the famous word2vec
The following models are chosen for running the
experiments. TransE
The mentioned papers improve the state-of-the-art in knowledge graph embedding in their respective papers. The presumption is that the models which improve upon the results of TransE on other grounds such as Link Prediction, should also deliver similar improvements in Topic Modeling results. To create a comparison of equal grounds, all of these models should be trained with the same dataset, the same parameters, and produce an output of the same embedding dimension. By keeping all other variables same, it’s possible to directly observe the quality of the embeddings for the purpose of topic modeling. The result of this approach helps determine the methods and the configurations which moves the state of the art further by producing the highest accuracy in topic modeling.
The following subsections explain the main
characteristics and differences of the compared embedding methods:
TransE TransE model represents the relations in the graph
as translations in the embedding space
As stated before, TransE is part of the baseline method KGE-LDA that the following methods are compared to in the experiments.
TransH TransH model, models relations as hyperplanes in
addition to the translation operations as TransE does
DistMult This model also directly aims to improve on
TransE model, and the main difference is the composition of
vectors. Different from TransE where vectors are composed
by addition as explained in previous subsections (head +
relation = tail); DistMult composes vectors by weighted
element-wise dot product, in other words the following
multiplicative operation: head x relation = tail
TransR TransR model attempts to tackle the problem that
the same semantic space to model embeddings for all entities
and relations is insufficient
PTransE PTransE builds upon the previous methods by
utilizing multiple-step relation paths in the knowledge
graph. Their approach is similar to TransE, with the
addition of relation path-based learning
HolE Short for “Holographic Embeddings”, the
difference that this model adopts is the learning of the
compositional vector space representation of entire knowledge
graphs
One interesting fact is that, HolE was proved to be
equivalent to another method called ComplEx
Analogy Analogy proposes the optimization of latent
feature representations with respect to the analogical properties
of the embeddings of both entities and relations
In Table 1, the time and space complexities along with the scoring functions of the described methods are compared.
While the previous two sections observe the effects of the embedding models and process; this section focuses on the density and quality of the knowledge graphs, which the embedding models are trained with.
Therefore, as a source of new information for the knowledge graph, the text corpus itself is a great answer. The dependency relations in sentences constitute meaningful semantics, and a quite massive source of information. The question that remains to be answered is the fact that are semantic relations in a knowledge graph and a dependency graph are compatible with each other? Are they able to create a richer knowledge base? Are the current embedding methods able to capture the information encoded in the resulting massive graph?
To answer these questions, the Knowledge Graph used in this paper (WN18) was merged with the Dependency Graph obtained by the 20NG text corpus which is also used in this paper for topic modeling. As the details can be seen in the Datasets subsection of the Experiments section; the density of the graph increased about 5 times, which surely created a more complex semantic structure.
The general structure of the merging phase is illustrated in Figure 3. The process finds the dependency trees of each sentences. Then, the corresponding entity of each word in the knowledge graph is found. If the words and computed dependencies pass the filtering stage, a new link is added between the corresponding entities in the knowledge graph with the name of the dependency relation.
Dependency Tree of a Sentence
word1 word3 dependency_relation word2 word4
Knowledge Graph Synset("word1")
Synset("word2") relation1
relation2 Synset("word3")
Extended Knowledge Graph dependency_relation Synset("word1")
Synset("word2") relation1
relation2 Synset("word3") This section aims to increase the primary parameters to measure their effects on the final outcome. The motive is that as long as computational limits and feasibility allow, better parameters and settings should be used if it provides considerable improvements in the performance. In the light of this motive, the following aspects are considered.
The first aspect to be investigated is the effects of the embedding dimension on the Topic modeling performance. The motive for this aspect is the fact that the larger and denser the knowledge graph or the dataset gets, it creates more information to be stored in the embeddings. Larger vector dimensions offer more space to encode the semantics, but naturally it comes with performance costs.
Furthermore, the effects of topic number chosen for the
topic model also has a direct effect on the performance.
Considering the results in KGE-LDA
Lastly, the extended knowledge graph method that was described in the previous section should also be examined with the increased parameters as the information encoded from a larger graph might even provide greater performance with higher dimensional embeddings and higher topic numbers.
To merge learned embeddings with the process of the topic modeling, the original implementation of KGE-LDA by its authors was used1. The original implementation was chosen, because KGE-LDA is the baseline work that this paper follows; thus it’s the best choice for running the experimentations.
The source code is structured as a Java project, and has a
dependency for the Stanford CoreNLP library. Along with
KGE-LDA, the project contains the implementations for
LDA
Parsing 20NG dataset with the CoreNLP DependencyParser and to obtain dependency trees.
Updating the WN18 graph with the obtained dependencies.
Various minor alterations throughout the source code.
For the purpose of the experimentations for Embedding Method Comparison, the implementations of the chosen embedding methods were needed. Therefore, implementations of TransE, TransH, TransT and PTransE were taken from the open-source project KB2E2. The implementations of Dist
Mult, HolE and Analogy were taken from the open-source project OpenKE3.
For the purpose of the Knowledge Graph Extension part of this paper, Stanford CoreNLP DependencyParser Annotator was used. Using DependencyParser, the code for the Knowledge Graph Extension part was implemented in Java. The process and the implementation follows this algorithm: Algorithm 1: Knowledge Graph Extension with Dependency Trees 1 KnowledgeGraph WN18; 2 DependencyNetwork Empty Graph; 3 for Document d in 20NG do 4 for Sentence s in d do 5 t DepencencyParser(s); 6 DependencyNetwork append t; 7 end 8 end 9 KnowledgeGraph merge DependencyNetwork; 10 return KnowledgeGraph;
Knowledge Graph
The example in the Figure 4 shows how a dependency relation extracted from a sentence updates the knowledge graph. In this specific example, there is a “Hyponym” relation from “Equipment” entity to “Satellite” entity in the knowledge graph. The dependency parser finds out that these two words are used in a compound in the corresponding sentence, and updates the knowledge graph with the “Compound” relation.
In this section, a series of experiments that involve different methods and variations of parameters are presented. The
To visualize how the dependency relations are merged with the knowledge graph, please refer to the Figure 4. Two topic models are chosen to compare the results of experiments with: used datasets along with the chosen parameters are stated for each of the different experiment sets.
The experiments are conducted to find answers to following questions: 1. Are newer and improved embedding models able to capture better semantics for the purpose of topic modeling? 2. How does the number of topics affect the performance of these sets of methods? 3. Does a denser and more complex knowledge base create a better or worse encoding of entities? 4. What is the importance of the vector dimensions in capturing and encoding information? Do we need larger vectors for more accurate representations for the used datasets?
The experiments are grouped into three categories that each try to answer the corresponding questions stated above. We proceed with three sets of experiments: (1) Embedding Method Application and Comparison; (2) Knowledge Graph Extension; (3) Further Exploration of Parameters.
KGE-LDA (Yao et al. 2017)
LDA was chosen as the primary indicator of performance, because it’s the most widely used topic model which is considered as the baseline method for many other works in the field. KGE-LDA was chosen as the main indicator of performance since it is the baseline method and starting point of this paper work.
Text Corpus The datasets in the context of this work refer to the text corpus that is used to run the topic models. For this purpose, 20-Newsgroups (20NG) dataset was used. The dataset includes 18,846 documents, split into 20 categories, with a vocabulary of 20,881 distinct words. In the text preprocessing phase, the following steps are applied to the data: Tokenization (with Stanford CoreNLP), stopwords removal, and rare words removal (for words that appear less than 10 times throughout the dataset).
External Knowledge The external knowledge refers to the knowledge graph that was used to train the representation learning methods to obtain the word embeddings. WN18, which is a subset of a widely used lexical knowledge graph WordNet, was used for this purpose. WN18 has the following characteristics in the training set: 141,442 triplets (the missing 10,000 triplets of WN18 are in the test and validation sets); 40,943 entities; 18 types of relations; 8,819 common entities with the 20NG vocabulary.
Table 2 shows the top 10 occurring relation types in the knowledge graph, their occurring counts, and percentages in size over the whole graph.
Knowledge Graph in the previous subsection was merged with the dependency graph obtained from the 20NG text corpus. The resulting graph has the following characteristics that have increased relative to the original knowledge graph(WN18): 817,568 triplets, with respect to the original 141,442; 55 types of relations, increased from the original 18.
There were new relations introduced to the knowledge graph, but no new entities. To demonstrate how the knowledge graph changed, here are the top 10 occurring relation types, their occurring counts, and percentages in size over the whole graph:
It can be seen that the structure of the knowledge graph has changed substantially, with the high number of additions. With the extension, the size of the graph grew by 578% compared to the original knowledge graph, and 37 new relation types were added.
A set of settings of the different parameters have been defined for the execution and validation of the approach. Some parameters have been adopted with a constant value across the experiments, while others have been varying across experiments. The settings considered include: 1. Settings for Embedding Methods Comparison: all parameters have been fixed, except for the number of topics (and the respective parameter ), as reported in Table 4; 2. Settings for Knowledge Graph Extension: The settings are the same as the settings of Embedding Methods Comparison group. 3. Settings for Further Exploration of Parameters: with the aim of delving into detailed investigation of the parameter values, a further set of experiment with new variations of the settings have been launched, with values as reported in Table 5. With respect to the initial experiments (parametrized as in point 1 of this list), the embedding dimension is increased to 100 and the number of topics is increased to 100.
The results are obtained through two different evaluation
mechanisms, namely Topic Coherence and Document
Classification. UCI method which uses Pointwise mutual
information (Newman et al. 2010) was used for Topic
Modeling, and LIBLINEAR linear classification library
Topic Coherence Results As stated before PMI based topic coherence was used to obtain these results. To compute PMI, a dataset of 4,776,093 Wikipedia articles were used. For each method and topic, the results were run 5 times, after which the average and the standard deviation was calculated. The results can be found in Table 6. Overall Topic Coherence Results The best and second coherence scores for each topic number are different, and it should be noted that the performance of the original LDA is consistently good. TransR leads to more coherent topics with lower topic numbers, and Analogy performs best with higher topic numbers. The general trend shows improvement with higher topic numbers.
the topic coherence results are close and in the same range with each other. The only significant visual difference in coherence can be observed with topic number 20 where we see TransR performing better than other methods. It is also worth mentioning that TransR performs best with the topic number of 20 than higher numbers, and performs worst on 30 topics. With 20 topics, the standard deviation seems to be higher than higher topic numbers with the best(TransR) and worst(TransH) scores of all the combinations.
Model B With Model B, there is also a general trend of improvement with topic number. The standard deviation in the general trend also gets smaller with increasing topic number. TransR scores the lowest on 20 topics, even though it scored the highest on 20 topics with Model A. The highest score combination is Analogy method with 50 topics.
been classified using LIBLINEAR
show that in overall results with topic numbers 20,30,40 and 50; HolE and Analogy perform the best. Also on average, Model A results in slightly better scores than Model B; even though Analogy performs better with Model B. Another observation is that, performance almost always increases with topic number; noting that with 40 and 50 topics, the results are closer to each other than with other topic number increments. Model A The results of Model A show that in 20 topics, HolE and DistMult perform the best. Their approach apparently is better for small number of topics. Analogy also performs close to them. In 30 topics, the results show that HolE again scores best. However, this time DistMult scores low, and TransE, TransH and TransR which employ an addition based translation score better. In 40 topics, the performance of all methods converge, with all of them scoring more similarly than they do in other topic numbers. 50 topic results are also relatively similar, with TransR, Analogy and TransH scoring best.
The outcomes show that HolE is the best performer overall with Model A. Looking at the standard deviations, it seems that with Model A; the methods have similar consistency in their results. Model B The main difference of Model B generates the entity embeddings by topics in the same document, so it’s important to state that the embeddings of the best methods are a better fit for this approach.
The results of Model B reveal that in 20 topics, DistMult is the best performer along with PTransE RNN. In 30 topics, Analogy outperforms others, as all the other methods score similar to each other. In 40 topics, TransH and TransR score better than others by a landslide. In 50 topics, Analogy seems to outperform others with TransE and HolE scoring close.
The outcomes show that Analogy and DistMult are the best performers overall with Model B. It’s also important to note that Analogy gives more consistent results with multiple runs, which can be seen in lower standard deviation than other methods.
Topic Coherence Results The Topic Coherence experiments were run according to the parameters specified before. Each experiment was run 5 times, with averages and standard deviations reported in the Table 8.
Overview The results in Table 8 show that the Extended Knowledge Graph led to similar results with the Original Knowledge Graph. With an overall inspection of the table, it can be seen that the best performance are distributed to different models and graphs. The version with the Extended Knowledge Graph provided better average scores for 20 topics and 40 topics. Also, the overall trend is similar to the topic coherence results of the previous section, as 30, 40 and 50 topics resulted in the same range of performance with each other. 77
section were also run 5 times as the ones before. The averages with the standard deviations are reported in Table 9.
Document Classification with Knowledge Graph Extension Overview Results in Table 9 show that the knowledge graph extension created better semantics in the graph which in turn reflected to the classification results. We see an overall improvement with both Model A and Model B, whereas improvements with Model A are larger. Extended Graph with Model A performs better with smaller topic numbers, where as the extended graph with Model B is more accurate on larger topic numbers.
The experiment in this section corresponds to the previous subsections about further exploration of parameters. For this purpose, an increased topic number of 100 and an increased embedding dimension of 100 was used with TransE and Analogy on the original knowledge graph, and furthermore TransE on Extended Knowledge Graph.
The average and standard deviations obtained from 5 runs of each combinations are reported in Tables 10 and 11.
According to the Topic Coherence scores, the extended knowledge graph provides a better performance on 50 topics than both TransE and Analogy on the original graph. Even though it scores the equal as the same configuration with Original Knowledge Graph, its standard deviation is 90% lower. On 100 topics, Analogy with Model B stands out with the highest coherence score that was obtained throughout the experiments of this paper work by scoring 2.18% higher than the closest coherence score. Figure 11 offers a clear comparison of these results in a visual way. 77 75 73 71 69 67 65 63
The extended knowledge graph scores the highest Document Classification accuracy for both 50 topics and 100 topics with Model A. In fact, the Extended Graph with Model A on 100 topics scored the highest accuracy for Document Classification throughout the experiments of this paper by scoring 1.24% higher than the same configuration with the Original Knowledge Graph. On 50 topics, it scored the same average with the Original Knowledge Graph but with a smaller standard deviation. According to these results, the Extended Knowledge Graph leads to better accuracy than the Original Knowledge Graph with the exception of 50 topics with Model B. It also performs better than Analogy with Model A. These results can also be clearly seen in Figure 12.
The experiments were conducted on a computer with the following relevant technical specifications: 8 GB of DDR4 RAM @ 1866 MHz
Throughout the experiments, the elapsed execution time was measured. Embedding methods were run only once to obtain the representations from the knowledge graph. The fastest embedding happened to be TransE with approximately 1 hour of computation, and the slowest was HolE with approximately 17 hours of computation. All of other methods ran for a duration between 1 hour and 2 hours. It is safe to say that HolE was exceptionally slow during the training phase compared to other methods.
The more crucial and overall time consuming part was running the topic models with the obtained representations. The duration of topic modeling phase was not affected by the representations obtained by different methods, as they all provide an output of the same size. However, the topic number and embedding size had a significant effect on the execution time. The average durations are reported in two separate tables. For embedding size of 50 the results can be seen in Table 12 and for embedding size of 100 the results can be seen in Table 13.
To more clearly interpret the execution times, Figure 13 provides a visual representation. It can be seen in the figure on K = 50 that the runtime duration decreases by 3.3% with the embedding size with Model B, and increases 14.5% with Model A from 50 dimensional embeddings to 100 dimensional embeddings.
However an increase from 50 topics to 100 topics increases runtime duration by 79.9% with Model A and 45.4% with Model B. Considering these facts with the general trend of growth in the figure; it is safe to say that topic number has a larger impact on runtime duration than the embedding size during topic modeling.
400 350 300 250 200 150 100 The results on topic coherence throughout the three experiments share a similar pattern. From 30 topics to upwards, the scores are really similar with the consideration of standard deviation, with a few results having significant difference. These scores also do not vary much between different methods both for the incorporation models A and B. For the results in 20 topics, we have larger difference between methods. With the increased parameters of 100 topics and 100 dimensional embeddings, the highest score achieved is 75.1 2.21 by Analogy, which scored 72.6 1.41 with 50 topics and 50 dimensional embeddings. Topic Coherence with 100 topics shows that Analogy with Model B configuration proves to be successful also on higher dimensional embedding and higher topic numbers.
Therefore, some inferences can be made for the effects of embedding methods on topic coherence: The coherence increases with topic number on average, but inconsistently. This means that a general trend of increase is seen, except on 40 topics which resulted in lower coherence scores in general than 30 topics. The usage of different embedding methods create topic coherence results that are in 1.2% range of each other on average. Analogy with Model B leads to the highest coherence scores with high topic numbers. The extended knowledge graph clearly improved the Document Classification accuracy with the exception of 40 topics. The improvements on Topic Coherence is on 20 and 40 topics.
For a general purpose use, Analogy is a clear choice over DistMult and HolE. The first reason is the fact that Analogy is a generalized method which can reproduce DistMult and HolE with a selection of parameters; it allows a higher range of performance and parameters. This should allow a grid search to find a configuration which is better than DistMult and HolE. The second reason is the fact that even though HolE and Analogy with the same parameters perform quite similar to each other, it takes much longer to train HolE ( 17 hours) compared to Analogy( 1-2 hours). A much faster training with theoretically being able to produce the same results as HolE, makes Analogy more feasible.
The Document classification Evaluation produced results that are clearer and easier to interpret in general. With small exceptions, increased topic number produced better results. In the embedding method comparison section, it can be seen that some of the newer and more complex embedding methods like DistMult, HolE and Analogy led to higher classification accuracy. Model A seems to be on average 1% better than Model B, but they produce equally consistent results with the same standard deviation at 1.9% on average.
On the other hand, there are clear improvements in the accuracy of the document classification when the Extended Knowledge Graph was used to train the embedding methods. This means that the semantic structure of the knowledge graph was enhanced, which reflected into better vector representations of entities and relations.
In the last group of experiments, the Extended Knowledge Graph provides better results than TransE and Analogy on the Original Knowledge Graph with an accuracy of 0:734 0:002 which is the highest accuracy recorded throughout the experiments in this paper work.
In light of these outcomes, the following inferences are made for the effects of embedding methods on document classification. The accuracy consistently increases with topic number. Changes on the embedding method performance reflects on the document classification accuracy. Analogy with Model B leads to the highest accuracy scores on high topic numbers. The extended knowledge graph led to increased accuracy, and showed that dependency trees enhanced the semantics of the knowledge graph.
This paper explored the incorporation of knowledge graph embeddings into topic modeling, by experimenting on various aspects and identifying the ways for improvements. These aspects were the semantic information in the source knowledge graph, different embedding methods, performance effects of topic numbers and embedding dimensions. performance of 7 embedding methods, 2 topic models, 2 variations of the knowledge base and various parameters have been explored in the context of Topic Modeling. 2 evaluation methods, namely Topic Coherence and Document Classification, have been used to measure the success of the experimentations. In the light of these results, this paper work has made several contributions.
On Embedding Methods Comparison, Topic Coherence and Document Classification yields different performance by each method, but the results have similarities. The most obvious pattern is the performance of Analogy. It outperforms all other methods on higher topic numbers with Model B. For lower topic numbers, simpler methods like TransE and TransR produce the best results. Overall, the best average scores come from HolE.
The Knowledge Graph Extension scores similar results to the original graph on Topic Coherence, but on Document Classification it clearly improves the accuracy. With increased parameters and embedding dimension, the improvements of the Knowledge Graph Extension are clearer, especially in Document Classification.
The best performing embedding method Analogy with Model B achieves an average improvement of 0.50% over the baseline method (KGE-LDA using TransE) in Topic Coherence, and an average improvement of 1.01% over the baseline method (KGE-LDA using TransE) in Document Classification. The Knowledge Graph Extension achieves an average improvement of 0.52% over the Original Knowledge Graph in Topic Coherence, and an average improvement of 0.77% over the Original Knowledge Graph in Document Classification.
As the closing remark, the best embedding method, incorporation model and parameter combination is Analogy with Model B on high topic numbers, with high embedding dimension. The extension of the knowledge base along with high embedding dimension enables more information to be encoded into the vectors, which in turn creates a more accurate representation of the entities compared to the Original Knowledge Graph. This performance improvement of the Extended Knowledge Graph comes with a 578% growth in the size of the graph.
It has been shown that Analogy is the most optimal embedding method Secondly, the results clearly show that the Extended Knowledge Graph has improved both Topic Coherence score and Document Classification accuracy.
Deeper investigations on a few points can provide further improvements on the solution. For the embedding method comparison part, the different methods have been tested with the same parameters. This provided an equal ground for the methods to compete with each other. However, a comprehensive parameter grid search for each embedding method can increase their performance and reveal more realistic values. Finally, as the specific knowledge graph extension in the experiments yielded better results, there can be further exploration on the knowledge graph capabilities.
[Blei and Jordan 2003] Blei, D. M., and Jordan, M. I. 2003. Modeling annotated data. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, 127–134. ACM. [Blei and Lafferty 2006] Blei, D., and Lafferty, J. 2006. Correlated topic models. Advances in neural information processing systems 18:147.