Topic models are widely used to thematically describe a collection of text documents and have become an important technique for systems that measure document similarity for classi cation, clustering, segmentation, entity linking and more. While they have been applied to some non-text domains, their use for semi-structured graph data, such as RDF, has been less explored. We present a framework for applying topic modeling to RDF graph data and describe how it can be used in a number of linked data tasks. Since topic modeling builds abstract topics using the co-occurrence of document terms, sparse documents can be problematic, presenting challenges for RDF data. We outline techniques to overcome this problem and the results of experiments in using them. Finally, we show preliminary results of using Latent Dirichlet Allocation generative topic modeling for several linked data use cases.
Data presented as RDF triples can be problematic for tasks that involve identifying entities, nding entities that are the same, nding communities of entities and aligning ontological information. Data describing a resource can be sparse, making it harder to distinguish one resource from other similar resources. Data describing a resource can be noisy, having excessive data that is not relevant to the resource. There can be large volumes of data which may contribute to an increase in noise, errors, and ambiguities.
When data originating from multiple sources is used, combining and resolving
resource information can be challenging. For example, when aligning attributes
from one ontology to another, often there are attributes that simply are not
alignable [
Topic modeling has quickly become a popular method for modeling large
document collections for a variety of natural language processing tasks. Topic
modeling is based on statistics of the co-occurrence of terms (typically words)
and establishes topics that are groupings of terms to describe documents. It has
been used to describe [
Topic modeling is a statistical method
that results in abstract categories or
topics from the processing of a set of
documents. Several methods have been
developed for generating topics. Early work by
Deerwester et al. [
With LDA, the terms in the collection of documents produce a vocabulary
that is then used to generate the latent topics. Documents are treated as a
mixture of topics, where a topic is a probability distribution over this set of
terms. Each document is then seen as a probability distribution over the set of
topics. We can think of the data as coming from a generative process that is
de ned by the join probability distribution over what is observed and what is
hidden [
For each document: (1) Choose a distribution over topics; (2) For each word in the document: (a) select a topic from the document's distribution over topics and (b) select a term from the associated distribution over terms.
The computational portion of LDA involves learning the topic distributions by
means of inference. Though there are a number of variational and sampling based
methods for performing the inference, Gibbs sampling [
We describe how one might use topic modeling for RDF data and explore
its application to several problems faced by the Semantic Web community. RDF
data is less typical in terms of the documents that are used to create a topic
model but since a bag of words is typically used with this model, we will show
how RDF data can be used. Topic modeling was originally used to characterize
relatively long documents, such as newswire articles or scienti c papers. More
recently, researchers have outlined successful strategies [
There are several issues in applying topic models to short texts [
To give light to these problems we show how topic modeling for text documents di er from RDF documents by describing how topic models are used with text documents and how we apply them to RDF graphs.
Although there are a number of algorithms for de ning and using topic models, they share several common aspects. A topic model uses a xed set of K topics to describe documents in a corpus. K varies with the application and is usually between 100 and 1000. A text documents could be anything from a tweet to a 30-page scienti c article, but they typically contain at least several paragraphs of text. The mixture of topics in a document is represented as a vector of real numbers between 0 and 1, where the kth number speci es the amount of topic k that the document exhibits. Using topic vectors makes it easy to de ne the \semantic" distance between two documents (often using the cosine similarity).
The K topics making up a topic model are not speci ed in advance, but
learned by a statistical process that discovers the `hidden thematic structure
in a document collection' [
One common problem is that many of the automatically induced topics in a model may not correspond to concepts that are easy for people to identify. For topic models over text documents, the best that can be done is to list the most frequent words associated with each topic. This is often su cient to recognize that topic number 32 has something to do with politics and elections where as topic number 126 seems to be about software and computer applications. However, there are typically topics that are di cult or impossible to associate with familiar concepts.
Once a topic model has been learned or trained from a document collection, it can be used to infer a document's topic vector from its bag of words. These vectors can then be used for a number of di erent tasks, such as classifying, clustering or recommending documents. While topic models were originally de ned for text documents, they have been applied to other kinds of data (e.g., images and generic sequences) and can be used with RDF graphs. To do this, we must de ne what we will mean by a \document" and the word-like elements within them and also how to compile large collections of those \documents" to train our topic modeling system. For natural language, topic models sometimes exploit linguistic concepts like part-of-speech tags, stop words, and word lemmas and also apply normalization operations (e.g., downcasing, punctuation removal, abbreviation expansion, etc.) to improve performance or accuracy, so we might consider analogs to these notions for RDF data.
What's a document? We assume that a knowledge base is represented by triples, where a triple has a subject s, predicate p, and an object o, forming a t(s; p; o) with the following de nitions.
s 2 (U RI [ Blank),p 2 (U RI) and o 2 (U RI [ Blank [ Literal) We de ne an 'entity' by t1:::tn 2 T associated with a common s URI. In our current model, we treat a document as the set of triples that describe a single 'entity'. We experiment with this de nition of a document by working with di erent parts of the triple, supplementing the triples with additional data, and including 1-hop in-bound and out-bound links.
Alternatively, we could de ne it as the set of triples in which a given node is either the subject or the object. If we consider a large dataset like DBpedia to be a document collection, we probably want to further restrict the nodes in the graph that we will consider to be documents. A node like dbp:Alan Turing makes a good subject but T-box nodes like owl:sameAs or dbpo:birthDate probably do not. Similarly, structural nodes such as Freebase's compound value type nodes or nodes that link a measurement with a units and a numeric value may not be suitable subjects for documents.
What's a word? The \words" in a document are extracted from the subjects, predicates and objects of each of its triples and the extractions are treated as bags of words. The words related to an entity (given by a URI) are tokenized by extracting all the triples related to a particular URI from a triple store. Then by removing paths from subjects, predicates and objects. Speci cally, literals, i.e., strings, are used in which they are rst sanitized with stop words removed. Figure 2 shows an example of a set of triples forming a simple document and its associated word-like tokens. 2.3
Short text su ers from two distinct problems: sparseness a ects how well the
model can discriminate and the lack of context a ects word senses [
Sparseness. RDF data can su er from sparseness. If we choose to think of a document as a set of triples associated with a resource de ned by an URI, the set can be large, resulting in a larger, more context enriched bag of words or small, o ering very little information at all as shown in Figure 3. Even of the large set of triples, the data that could actually be used in the bag of words, after pre-processing could result in a smaller set of words.
Lack of Context. Context can be particularly problematic for RDF data, as often words are used that can have multiple meanings and due to the potential sparseness of RDF data in addition to the unnatural language characteristic, it could be hard to distinguish that meaning. For example, the description Alternative rock contains the word rock, without additional context, this word could be interpreted in multiple ways. Unnatural Language. RDF data su ers from unnatural language issues. Since RDF data is graph-based the natural structure of a sentence does not exist. Often the components of a sentence provide additional context for understanding words which may be polysemous or homonymous. In addition, the text is more prone to error during pre-processing. For example, it is not uncommon to nd parts of a triple that have unexpected characters, unusual letter casing, pointers to another resource and data that is simply hard to parse. We show some of these examples in Table 1.
The short text problems in RDF. Researchers tend to take two approaches to overcoming short text related problems. They either supplement the text or they create modi ed versions of LDA to support their speci c problem. We currently use the approach of supplementing text using a set of baseline techniques. Our future work will include additional techniques for supplementing text and a modi ed LDA algorithm for RDF graphs. We show ways to supplement RDF data in Figure 4. We could simply use the object literals of the triple, for example \University of North Carolina" is an object literal for the resource \University of North Carolina at Greensboro". We could also use the predicates, in addition to the object literal. For example, the predicate \http://dbpedia.org/property/name\, may be the predicate for the triple with the object literal \University of North Carolina". We may also choose to use Wordnet to supplement the RDF data. For example, for the word \Boston" if we take a subset of synsets and the de nition, we enrich the word \Boston" with the following data: [capital of Massachusetts, state capital and largest city of Massachusetts; a major center for banking and nancial services, Beantown, Bean Town, Boston, Hub of the Universe]. We also looked at using 1-hop in-links and 1-hop out-links. For example, \Boston" may refer to a mayor, which with 1-hop we could consume the triples related to the mayor of Boston. We could do this similarly with in-links. 3
Work by Hong et al. [
Since topic modeling works on the co-occurrences of terms, sparse documents
can be problematic. Work by Yan et al. [
Fig. 4: Bag of Words Variations co-occurrence'. Again, they speci cally address short text by modifying LDA. Though we think this work has merit in this paper we speci cally look at how to modify the data itself.
Work by Phan et al. [
Work by Dietz et al. [
Given our description of topic modeling and how it could be used with RDF data, we have outlined a number of ways topic modeling could be applied to research tasks within the Semantic Web community. 4.1
Often there is a need to associate type information with entities that are
dened within RDF data [
We use topic modeling to support entity type recognition by creating a topic model from a sample of data which contains known type information. We use the model to associate topics to the types. Given new data with missing type information, we then infer topics for new entities. Using KL divergence, for each entity with an unknown type, we measure the divergence between its topic vector and the topic vectors of each known type. Based on this measure we assign known types to new entities. 4.2
The need to match instances across di erent data sets or to link new instance
information with existing knowledge base instances is common [
Ontology alignment [
Community detection approaches [
This topical approach can be performed by using topic modeling. For example, authors of the Fantasy Literature and Penguin Publishing community might have more topics in common than authors of other genres associated with di erent publishing companies.
In our work we build a topic model from a data set that we identify as having communities of interest. From this model we then associate topic vectors to each entity. We look for entities which have a number n of topics in common. In order to nd entities which have n topics in common, we create a histogram from the topic probabilities for each entity. Assuming a topic de nes a sub-community, we use the histogram to tell us where the most density is among topics for the entity and set a threshold so that we only consider the topics which are most relevant to the entity. From this we assign entities to topic sub-communities. We then nd entities that have n sub-communities in common. 5
We experimented with di erent ways to build a bag of words from the RDF
data based on approaches in Figure 4. For each problem which we outlined
previously, we used LDA without any modi cations to the algorithm itself. Our goal
with this work was to show how to supplement RDF data to overcome issues
related to spareness, lack of context and the use of unnatural language. We did
see improvement in supplementing the RDF data with repetition of key words,
and using a limited set of synsets and de nitions from Wordnet. Speci cally,
when working with large graphs, using the object literals alone may be su
cient but we have observed better results when including either the predicate
or the predicate and the subject. We have also found that where the graphs
are particularly sparse, using Wordnet [
We used DBpedia data and created two di erent randomly selected data sets. One was used to actually build the topic model, it had 6000 unique resources. The second data set had 100 unique resources. We associated topics to known types and then used the model to infer types for each entity in the second data set. We then used KL divergence to compare topic vectors. From this we mapped types from the rst data set to entities in the second data set. We tested with 200 topics and 400 topics with resources that had an average of seven types that should be recognized. Our ground truth in this case was the type de ntions in the DBpedia data set. We removed the type de ntions for our test data set then evaluated our predictions with what was de ned by DBpedia. Though our test set was relatively small, we were able to see how precision changed based on data variations. As can be seen in Figure 6 we saw the highest precision using predicates and objects and in Figure 7 we saw the highest precision using predicates and objects that included the Wordnet synsets and de nitions. Though it was clear that objects alone did not perform as well as including the predicate, future work will further explore the relationship between supplemental data and the number of topics chosen for the model. 5.2
For this experiment we took a subset of DBpedia data as our knowledge base including 300 unique entities with an average of 19 triples per entity and used this data set to build a topic model. We created a second data set with 100 unique entities obtained from the same data, except we obfuscated the subjects such that subjects could not be directly matched. For example, the unobfuscated subject \Falling in Love with Jazz" became \Jxiiwhw wh Uaka kwki Uxhh". We used this approach as to create a ground truth for entity matching. We used a lookup table to correlate between the obfuscated subjects and the unobfuscated subjects to evaluate our approach. We associated topics with each entity in our knowledge base. We then took our obfuscated data set and inferred topics for each entity. From this we used cosine similarity to compare entities and tried to match entities from the two data sets. Though topic modeling is too coarse to use directly to match instances, it does provide a way to signi cantly reduce the number of candidates that need to be matched. Our experiments showed topic modeling was a reasonable approach for candidate selection reducing on average the number of candidates from 1000s to 100s. However, more work is required to show how this method could be used in conjunction with a entity matching algorithm. 5.3
We tested two di erent alignments that included aligning properties and classes. We used the data from oaei 101 and oaei 103, where the class and property names are identical. We also used the data from oaei 101 and oaei 205, where class and property names to be aligned are not spelled the same. For example, oaei 101#Booklet and oaei 205#Brochure should be recognized as alignable. We used the OAEI ref alignments to evaluate our approach. This alignment document indicates which properties and classes should be aligned. We excluded instance alignments for this evaluation. We did however extract the instance data only to generate a topic model. Our evaluation examined how well we aligned properties and classes. We tested with 25, 50, 100 and 200 topics and saw the best performance with 50 topics. We exercised the di erent variations for the RDF data. The ontologies are good examples of sparseness, by using repetition and supplemental data we were able to get approximately 80% precision, where we selected the top N candidates of either an attribute or class match. We took a sample of the DBpedia data set and performed topic modeling using 50, 100, and 200 topics. From this we associated a set of topics with each entity. We then looked for entities that had n topics in common. Commonality is based on rst identifying topics for each entity that are most relevant given their probabilities. then comparing entities based on this subset of topics. Our data set did not include ground truth for this evaluation. However, as seen in gure 10 our preliminary results found interesting communities, such as a community that included Vini Lopez and Bruce Springsteen who are related by playing in the same band. Future work will perform more comprehensive experiments to evaluate this method further. 6
We described a framework for applying topic modeling to RDF graph data and described how it can be used in a number of linked data tasks, including predicting entity types, instance matching, ontology alignment, context identi cation and community detection. By supplementing RDF data we can address the problems related to spareness, lack of context and unnatural language. We have used di erent problems in Semantic Web research to exercise LDA modeling. For preliminary results over a small amount of data, topic modeling shows promise for a number of tasks. Repetition and Wordnet supplemental data improves performance. More work is needed to determine how we could use in-links and out-links to supplement the data without increasing the noise. Our results, though preliminary, provide some insight into how a basic LDA model might perform given
Fig. 10: Community Detection variations of the text. Our future work looks to modify the LDA algorithm itself to work speci cally with graph based data.
Acknowledgment. This work was supported by NSF grants 0910838 and 1228673.