The 2016 edition of the Linked Data Mining Challenge, conducted in conjunction with Know@LOD 2016, has been the fourth edition of this challenge. This year's dataset collected music album ratings, where the task was to classify well and badly rated music albums. The best solution submitted reached an accuracy of almost 92:5%, which is a clear advancement over the baseline of 69:38%.
Linked Open Data [
This year, the Linked Data Mining Challenge was held for the fourth time,
following past editions co-located with DMoLD (at ECML/PKDD) [
The rest of this paper is structured as follows. Section 2 discusses the dataset construction and the task to be solved. In section 3, we discuss the entrants to the challenge and their results. We conclude with a short summary and an outlook on future work.
The 2016 edition of the challenge used a dataset built from music albums recommendations, turned into a two-class classi cation problem.
The task concerns the prediction of a review of music albums, i.e.,\good" and
\bad". The initial dataset is retrieved from Metacritic.com3, which o ers an
average rating of all time reviews for a list of music albums4. Each album is
linked to DBpedia [
The dataset was split into training and test set using random strati ed split 80/20 rule, i.e., the training dataset contains 1; 280 instances, and the test dataset contains 320 instances. The training dataset, which contains the target variable, was provided to the participants to train predictive models. The test dataset, from which the target label is removed, is used for evaluating the built predictive models. 2.2
The task concerns the prediction of a review of albums, i.e.,\good" and \bad", as a classi cation task. The performance of the approaches is evaluated with respect to accuracy, calcuated as:
#true positives + #true negatives Accuracy = #true positives + #false positives + #false negatives + #true negatives (1) 2.3
The participants were asked to submit the predicted labels for the instances in the test dataset. The submissions were performed through an online submission system. The users could upload their prediction and get the results instantly. Furthermore, the results of all participants were made completely transparent by publishing them on an online real-time leader board (Figure 1). The number of submissions per user was not constrained.
In order to advance the increase of Linked Open Data [
In total, three parties participated in the challenge. We compare those results against a baseline approach. 3.1
We provide a simple classi cation model that serves as a baseline. In this baseline
approach, we use the albums' DBpedia URI to extract the direct types and
categories of each album. On the resulting dataset, we built a k-NN classi er (k=3),
and applied it on the test set, scoring an accuracy of 69:38%. The model is
implemented in the RapidMiner platform5, using the Linked Open Data extension
[
During the submission period, three teams completed the challenge by submitting a solution to the online evaluation system and describing the used approach in a paper. In the following, we describe and compare the nal participant approaches. A summary of all approaches is given in Table 1.
By Jedrzej Potoniec In this approach, the authors extract features from several non-LOD datasets, which are then used to build a Logistic Regression model for classi cation of albums. To extract the features, the authors start by scraping the Wikipedia pages for the given albums using the Scrapy tool6. From the collected data, the
authors focus on the album reviews and ratings. Furthermore, reviews and ratings are collected from Amazon7 and Discogs8, while MusicBrainz 9 is used to obtain the number of users owning an album and its average score. The nal dataset contains 94 numerical attributes in total.
To train the classi cation model, the authors use logistic regression, using the RapidMiner platform. Before training the model, a Z-transformation is performed on all attributes, so all attributes have an average of 0 and standard deviation 1. The authors perform 10-fold cross-validation on the training dataset, achieving accuracy of 91:7%. This value is consistent with 92:5% on the test set reported by the challenge submission system.
Furthermore, the authors provide some insights on the relevance of the features for the classi cation task, based on the learned logistic regression coe cients for each attribute. For example, the results show that Metacritic ratings highly correlate with ratings from other sources, like Pitchfork 10, AllMusic11, Stylus12, and others.
The code and the data can be found online13.
By Semih Yumusak, Emir Mun~oz, Pasquale Minervini, Erdogan Dogdu, and Halife Kodaz In this approach, the authors use Linked Open Data features in combination with album reviews to build seven di erent classi cation models. DBpedia is used as a main source for LOD features. More precisely, the authors manually select predicates that might be relevant for the given classi cation task. Along the direct predicate values, aggregate count features are used as well. Besides the LOD features, the authors also use albums' reviews retrieved from Metacritic as textual features. The reviews are rst preprocessed, i.e., lower-case transformation, non-alphanumeric normalizations, stopwords removal and stemming, then standard Bag-of-Words is used to represent each review.
Furthermore, the authors identify that discretizing some of the features leads to better representation of the data, e.g., the award feature of an artist could be marked as \high" if the number of awards is more than one, and \low" otherwise.
In the next step, the authors experiment with seven di erent classi cation models, i.e., linear SVM, KNN, RBF SVM, Decision Trees, Random Forest, AdaBoost, and Naive Bayes. The hyper parameters for each model are determined 7 http://www.amazon.com/ 8 https://www.discogs.com/ 9 https://musicbrainz.org/doc/MusicBrainzn Database/Download 10 http://pitchfork.com/ 11 http://www.allmusic.com/ 12 http://www.stylusmagazine.com/ 13 https://github.com/jpotoniec/LDMC2016 manually via incremental tests, and results extracted from the training set. They evaluate each model on the training dataset using 10-fold cross-validation. The experiments were performed using the sckit-learn library14. The best performance on the training dataset is achieved using Linear SVM with an accuracy of 87:81%. Applying the same model on the test set scores an accuracy of 90:00%, which con rms that the model is not over tted on the training dataset. The authors evaluate the relevance of di erent features group separately, showing that most of the models perform the best when using both LOD and text-based features.
Furthermore, the authors provide some interesting observations about the task. For example, \Bands are more successful than single artists", \Shorter albums are likely to be worse", \The genre of the album indicates if the album is good or bad", and others.
The source les, the crawler code and the reviews, the enriched knowledge base in RDF, and the intermediate les are published as an open-source repository15.
By Petar Petrovski and Anna Lisa Gentile In this approach, the authors use an unconventional method for the task of music album classi cation. They explore the potential role of music album cover arts for the task of predicting the overall rating of music albums and investigate if one can judge a music album by its cover alone. The proposed approach for album classi cation consists of three main steps. Given a collection of music albums, the authors rst obtain the image of their cover art using DBpedia. Then, using o -the-shelf tools obtain a feature vector representation of the images. In the nal step, a classi er is trained to label each album, only exploiting the feature space obtained from its cover art.
To extract the image features, the authors use the Ca e deep learning
framework [
To build a classi cation model, the authors use linear SVM model. The model is evaluated on the training set using 10-fold cross-validation, achieving accuracy of 58:30%. The accuracy of the model on the test set is 60:31%. Hence, the results 14 http://scikit-learn.org/ 15 https://github.com/semihyumusak/KNOW2016 16 bvlc reference ca enet from ca e.berkeleyvision.org 17 http://image-net.org/challenges/LSVRC/2012/ show that using only features extracted from the album cover arts is not su cient for the given classi cation task.
The dataset is available online18, along with the extracted feature vectors and used processes. We made a few more experiments in order to analyze the agreement of the three submissions, as well as the headroom for improvement.
For the agreement of the three submissions, we computed the Fleiss' kappa
score [
To exploit advantages of the three approaches, and mitigate the disadvantages, we analyzed how a majority vote of the three submissions would perform. The accuracy totals at 90:00%, which is lower than the best solution submitted. This shows that the majority vote is highly in uenced by the low scoring submission using the image features, which does not outperform the baseline. We also perform a weighted majority vote, using the achieved accuracy of each approach as the weight. The accuracy totals at \92.50", which is same as the best solution submitted. 4
In this paper, we have discussed the task, dataset, and results of the Linked Data Mining Challenge 2016. The submissions show that Linked Open Data is a 18 https://github.com/petrovskip/know-lod2016 useful source of information for data mining, and that it can help building good predictive models.
One problem to address in future editions is the presence of false predictors. The dataset at hand, originating from Metacritic, averages several ratings on albums into a nal score. Some of the LOD datasets used by the competitors contained a few of those original ratings, which means that they implicitly used parts of the ground truth in their predictive models (which, to a certain extent, explains the high accuracy values). Since all of the participants had access to that information, a fair comparison of approaches is still possible; but in a reallife setting, the predictive model would perform sub-optimally, e.g., when trying to forecast the rating of a new music album.
In summary, this year's edition of the Linked Data Mining challenge showed some interesting cutting-edge approaches for using Linked Open Data in data mining. As the dataset is publicly available, it can be used for benchmarking future approaches as well.
We thank all participants for their interest in the challenge and their submissions. The preparation of the Linked Data Mining Challenge and of this paper has been partially supported by the by the German Research Foundation (DFG) under grant number PA 2373/1-1 (Mine@LOD), and by long-term institutional support of research activities by the Faculty of Informatics and Statistics, University of Economics, Prague.