The datasets that are part of the Linking Open Data cloud diagramm (LOD cloud) are classi ed into the following topical categories: media, government, publications, life sciences, geographic, social networking, user-generated content, and cross-domain. The topical categories were manually assigned to the datasets. In this paper, we investigate to which extent the topical classi cation of new LOD datasets can be automated using machine learning techniques and the existing annotations as supervision. We conducted experiments with di erent classi cation techniques and di erent feature sets. The best classi cation technique/feature set combination reaches an accuracy of 81:62% on the task of assigning one out of the eight classes to a given LOD dataset. A deeper inspection of the classi cation errors reveals problems with the manual classi cation of datasets in the current LOD cloud.
The Web of Linked Data o ers a rich collection of
structured data provided by hundreds of di erent data sources
that use common standards such as dereferencable URIs
and RDF. The central idea of Linked Data is that data
sources set RDF links pointing at other data sources { e.g.,
owl:sameAs links { so that all data is connected into a global
data space [
Since the proposal of the Linked Data best practices in
2006, the Linked Open Data cloud (LOD cloud) has grown to
roughly 1 000 datasets (as of April 2014) [
The most well-known categorization of LOD datasets by topical domain is the coloring of the LOD cloud diagram.1 Up till now, the topical categories were manually assigned to the datasets in the cloud either by the publishers of the datasets themselves via the datahub.io dataset catalog or by the authors of the LOD cloud diagram. In this paper, we investigate to which extent the topical classi cation of new LOD datasets can be automated for upcoming versions of the LOD cloud diagram using machine learning techniques and the existing annotations as supervision.
Beside creating upcoming versions of the LOD cloud
diagram, the automatic topical classi cation of LOD datasets
can be interesting for other purposes as well: Agents
navigating on the Web of Linked Data should know the topical
domain of datasets that they discover by following links in
order to judge whether the datasets might be useful for their
use case at hand or not. Furthermore, as shown in [
In this paper, we present { to the best of our knowledge { the rst automatic approach to classify LOD datasets into the topical categories that are used by the LOD cloud diagram. Using the data catalog underlying the recent LOD cloud, we train machine learning classi ers with di erent sets of features. Our best classi cation technique/feature set combination reaches an accuracy of 82%.
The rest of this paper is structured as follows. Section 2 introduces the methodology of our experiments, followed by a presentation of the results in Section 3 and a discussion of remaining classi cation errors in Section 4. Section 5 gives an overview of related work. We conclude with a summary and an outlook on future work.
Copyright is held by the author/owner(s).
WWW2015 Workshop: Linked Data on the Web (LDOW2015).
In this section, we rst brie y describe the data corpus that we use for our experiments and the di erent feature sets we derive from the data. We than brie y introduce the classi cation techniques that we considered and sketch the nal experimental setup that was used for the evaluation. 2.1
In order to extract our features for the di erent datasets
which are contained in the LOD cloud, we used the data
corpus that was crawled by Schmachtenberg et al. [
In order to create the 2014 version of the LOD cloud
diagram, newly discovered datasets were manually classi ed
into one of the following categories: media, government,
publications, life sciences, geographic, social networking,
usergenerated content, and cross-domain. A detailed de nition
of each category is available in [
Figure 2 shows the number of datasets per category contained in the 2014 version of the LOD cloud. As we can see, the LOD cloud is dominated by datasets belonging to the category social networking (48%), followed by government (18%) and publications (13%) datasets. The categories media and geographic are only represented by less than 25 datasets within the whole corpus. 2.2
For each of the datasets, we created the following eight feature sets based on the crawled data.
Vocabulary Usage (VOC): As many vocabularies target a speci c topical domain, e.g. bibo bibliographic information, we assume that the vocabularies that are 2http://km.aifb.kit.edu/projects/btc-2012/ 3The crawled data is publicly available: http://data.dws. informatik.uni-mannheim.de/lodcloud/2014/ISWC-RDB/
used by a dataset form a helpful indicator for
determining the topical category of the dataset. Thus, we
determine the vocabulary of all terms that are used as
predicates or as the object of a type statement within
each dataset. Altogether we identi ed 1 439 di erent
vocabularies being used by the datasets (see [
Property URIs (PUri): Beside the class information of an entity, information about which properties are used to describe the entity can be helpful. For example it might make a di erence, if a person is described with foaf:knows statements or if her professional a liation is provided. To leverage this information, we collected all properties that are used within the crawled data by at least two datasets. This feature set consists of 2 333 attributes.
Local Class Names (LCN): Di erent vocabularies might contain synonymous (or at least closely related) terms that share the same local name and only di er in their namespace, e.g. foaf:Person and dbpedia:Person. Creating correspondences between similar classes from di erent vocabularies reduces the diversity of features, but on the other side might increase the number of attributes which are used by more than one dataset. As we lack correspondences between all the vocabularies, we bypass this, by using only the local names of the type URIs, meaning vocab1:Country and vocab2:Country are mapped to the same attribute. We used a simple regular expression to determine the local class name checking for #, : and / within the type object. By focusing only on the local part of a class name, we increase the number of classes that are used by more than one dataset in comparison to CUri and thus generate 1 041 attributes for the LCN feature set. Local Property Names (LPN): Using the same assumption as for the LCN feature set, we also extracted the local name of each property that is used by a dataset. This results in treating vocab1:name and vocab2:name as a single property. We used the same heuristic for the extraction as for the LCN feature set and generated 3 493 di erent local property names which are used by more than one dataset, resulting in an increase of the number of attributes in comparison to the PUri feature set.
Text from rdfs:label (LAB): Beside the vocabulary-level features, the names of the described entities might also indicate the topical domain of a dataset. We thus extracted all values of rdfs:label properties, lower-cased them, and tokenized the values at spacecharacters. We further excluded tokens shorter than three and longer than 25 characters. Afterward, we calculated the TF-IDF value for each token while excluding tokens that appeared in less than 10 and more than 200 datasets, in order to reduce the in uence of noise. This resulted in a feature set consisting of 1 440 attributes.
Top-Level Domains (TLD): Another feature which might help to assign datasets to topical categories is the toplevel domain of the dataset. For instance, government data is often hosted in the gov top-level domain, whereas library data might be found more likely on edu or org top-level domains.4 In & Outdegree (DEG): In addition to vocabulary-based and textual features, the number of outgoing RDF links to other datasets and incoming RDF links from other datasets could provide useful information for classifying the datasets. This feature could give a hint about the density of the linkage of a dataset, as well as the way the dataset is interconnected within the whole LOD cloud ecosystem.
We were able to create all features (except LAB) for 1 001 datasets. As only 470 datasets provide rdfs:labels, we only use these datasets for evaluating the utility of the LAB feature set.
As the total number of occurrences of vocabularies and terms is heavily in uenced by the distribution of entities within the crawl for each dataset, we apply two di erent normalization strategies to the values of the vocabulary-level features VOC, CUri, PUri, LCN, and LPN: On the one hand side, we create a binary version (bin) where the feature vectors of each feature set consist of 0 and 1 indicating presence and absence of the vocabulary or term. The second version, the relative term occurrence (rto), captures the fraction of vocabulary or term usage for each dataset.
The following table shows an example of the two di erent feature set versions for the terms ti:
Feature Set Version Term Occurrence Binary (bin) Relative Term Occurrence (rto) Feature Vector t1 t2 t3 10 0 2 1 0 1 0:5 0 0:1 4We restrict ourselves to top-level domains, and not public su xes. k-Nearest Neighbor: k-Nearest Neighbor (k-NN) classi cation models make use of the similarity between new cases and known cases to predict the class for the new case. A case is classi ed by its majority vote of its neighbors, with the case being assigned to the class most common among its k nearest neighbors measured by the distance function. In our experiments we used a k equal to 5 with Euclidean-similarity for non-binary term vectors and J accard-similarity for binary term vectors.
J48 Decision Tree: A decision tree is a owchart-like tree
structure which is built top-down from a root node and
involves some partitioning steps to divide data into
subsets that contain instances with similar values. For
our experiments we use the Weka implementation of
the C4.5 decision tree [
Naive Bayes: As a last classi cation method, we used Naive
Bayes (NB). NB uses joint probabilities of some
evidence to estimate the probability of some event.
Although this classi er is based on the assumption that
all features are independent, which is violated in many
use cases, NB has shown to work well in practice [
In order to evaluate the performance of the three classi cation methods, we use 10-fold cross-validation and report the average accuracy in the end.
As the number of datasets per category is not equally distributed within the LOD cloud, which might in uence the performance of the classi cation models, we also explore the e ect of balancing the training data. We used two di erent balancing approaches: (1) we down sample the number of datasets used for training until each category is represented by the same number of datasets; this number is equal to the number of datasets within the smallest category; and (2) we up sample the datasets for each category until each category is at least represented by the number of datasets equal to the number of datasets of the largest category. The rst approach, reduces the chance to over t a model into the direction of the larger represented classes, but it might also remove valuable information from the training set, as examples are removed and not taken into account for learning the model. The second approach, ensures that all possible examples are taken into account and no information is lost for training, but by creating the same entity many times can result in emphasizing those particular data points. For example a neighborhood based classi er might look at the 5 nearest neighbors, which than could be one and the same data point, which would result into looking only at the nearest neighbor. 3.
In the following, we rst report the results of our experiments using the di erent feature sets in separation. Afterward, we report the results of experiments combining attributes from multiple feature sets.
Table 1 shows the accuracy that is reached using the three di erent classi cation algorithms with and without balancing the training data. Majority Class is the performance of a default baseline classi er always predicting the largest class: social networking.
As a general observation, the vocabulary-based feature sets (VOC, LCN, LPN, CUri, PUri) perform on a similar level, where DEG and TLD alone show a relatively poor performance and in some cases are not at all able to beat the majority class baseline. Classi cation models based on the attributes of the LAB feature set perform on average (without sampling) around 20% above the majority baseline, but predict still in half of all cases the wrong category. Algorithm-wise, the best results are achieved using the decision tree (J48) without balancing (maximal accuracy 80:59% for LCNrto) and the k-NN algorithm, also without balancing for the PUribin and LPNbin feature sets. Comparing the two balancing approaches, we see better results using the up sampling approach for almost all feature sets (except VOCrto and DEG). In most cases, the category-speci c accuracy of the smaller categories is higher when using up sampling. Using down sampling the learned models make more errors for predicting the larger categories. Furthermore, when comparing the results of the models trained on unbalanced data with the best model trained on balanced data, the models on the unbalanced data are more accurate except for the VOCbin feature set. Having a closer look at the confusion matrices, we see that the balanced approaches are in general making more errors when trying to predict datasets for the larger categories, like social networking and government. 3.2
For our second set of experiments, we combine the available attributes from the di erent feature sets and train again our classi cation models using the three described algorithms. As before, we generate a binary and relative term occurrence version of the vocabulary-based features. In addition, we create a second set (binary and relative term occurrence ), where we omit the attributes from the LAB feature set, as we wanted to measure the in uence of this particular set of attributes, which is only available for less than half of the datasets. Furthermore we created a combined set of attributes consisting of the three best performing feature sets from the previous section. ALLrto: Combination of the attributes from all eight feature sets, using the rto version of the vocabulary-based features.
ALLbin: Combination of the attributes from all eight feature sets, using the bin version of the vocabulary-based features.
NoLabrto: Combination of the attributes from all feature, without the attributes of the LAB feature set, using the rto version of the vocabulary-based features. NoLabbin: Combination of the attributes from all feature, without the attributes of the LAB feature set, using the bin version of the vocabulary-based features. Best3: Includes the attributes from the three best performing feature sets from the previous section based on their average accuracy: PUribin, LCNbin, and LPNbin.
We can observe that when selecting a larger set of attributes, our model is able to reach a slightly higher accuracy of 81:62% than using just the attributes from one feature set (80:59%, LCNbin). Still the trained model is unsure for certain decisions and has a stronger bias towards the categories publications and social networking. 4.
In the following, we look at the best performing approach (Naive Bayes trained on the attributes of the NoLabbin feature set using up sampling). Table 3 shows the confusion matrix of this experiment, where on the left side we list the predictions by the learned model, while the head names the actual category of the dataset. As observed in the table, there are three kinds of errors which occur more frequently than 10 times.
The most common confusion occurs for the publication domain, where a larger number of datasets are predicted to belong to the government domain. A reason for this is that government datasets often contain metadata about government statistics which are represented using the same vocabularies and terms (e.g. skos:Concept ) that are also used in the publication domain. This makes it challenging for a vocabulary-based classi er to distinguish those two categories apart. In addition, for example the http://mcu.es dataset { the Ministry of Culture in Spain { was manually labeled as publication within the LOD cloud, whereas the model predicts government which turns out to be a borderline case in the gold standard. A similar frequent problem is the prediction of life sciences for datasets in the publications category. This can be observed, e.g., for the http://ns.nature.com/publications/, which describe the publications in Nature. Those publications, however, are often in the life sciences eld, which makes the labeling in the gold standard a borderline case.
The third most common confusion occurs between the
usergenerated content and the social networking domain.
Here, the problem is in the shared use of similar
vocabularies, such as foaf. At the same time, labeling a dataset as
either one of the two is often not so simple. In [
In summary, while we observe some true classi cation errors, many of the mistakes made by our approach actually point at datasets which are di cult to classify, and which are rather borderline cases between two categories.
Topical pro ling has been studied in the data mining,
database, and information retrieval communities. The
resulting methods nd application in domains such as
documents classi cation, contextual search, content management
and review analysis [
Although topical pro ling has been studied in other settings before, only a small number of methods exist for pro ling LOD datasets. These methods can be categorized based on the general learning approach that is employed into the categories unsupervised and supervised. Where the rst category does not rely on labeled input data, the latter is only applicable for labeled data.
Elle et al. [
An approach to detect latent topics in
entity-relationship graphs is introduced by Bohm et al. [
Fetahu et al. [
While the mentioned approaches are unsupervised, we employ supervised learning techniques as we want to exploit the existing topical annotation of the datasets in the LOD cloud.
In this paper, we investigate to which extent the topical classi cation of new LOD datasets can be automated using machine learning techniques. Our experiments indicate that vocabulary-level features are a good indicator for the topical domain, yielding an accuracy of around 82%.
The analysis of the limitations of our approach, i.e., the
cases where the automatic classi cation deviates from the
manually labeled one, points to a problem of the
categorization approach that is currently used for the LOD cloud: All
datasets are labeled with exactly one topical category,
although sometimes two or more categories would be equally
appropriate. One such example are datasets describing life
science publications, which can be either labeled as
publications or as life sciences. Thus, the LOD dataset classi
cation task might be more suitably formulated as a multi-label
classi cation problem [
A particular challenge of the classi cation is the heavy
imbalance of the dataset categories, with roughly half of the
datasets belonging to the social networking domain. Here, a
two-stage approach might help, in which a rst classi er tries
to separate the largest category from the rest, while a second
classi er then tries to make a prediction for the remaining
classes. When regarding the problem as a multi-label
problem, the corresponding approach would be classi er chains,
which make a prediction for one class after the other, taking
the prediction of the rst classi ers into account as a feature
for the remaining classi cations [
In our experiments, RDF links have not been exploited
beyond dataset in- and out-degree. For the task of web
page classi cation, link-based classi cation techniques, that
exploit the contents of web pages linking to a particular
page, often yields good results [
This research has been supported in part by FP7/2013-2015 COMSODE (under contract number FP7-ICT-611358).