In content-based recommender systems, the items to be recommended are represented by a set of features based on their content, whereas a user is represented by his profile. To build a recommender both information sources are compared. Most content-based recommenders use quite simple retrieval models, such as keyword matching or the vector space model with basic TF-IDF weighting [ 15 ]. A problem with these models is that they tend to ignore semantic information. To overcome this one can use Explicit Semantic Analysis (ESA) [ 10 ] instead of TF-IDF weighting which allows to represent a document as a weighted vector of concepts. Another way to add more linguistic knowledge is to use for example information from Wordnet as done by [ 6, 3 ]. An alternative is to use language models to represent documents. This was done for example by [ 16 ] when exploring content-based filtering of calls for papers. Besides retrieval models, machine learning techniques where a system learns the user profile and classifies items as interesting or not are also used for content-based recommenders. One of the first to do this was [ 2 ] using a Na¨ıve Bayes classifier.

When it comes to adding semantic information to recommender systems we see that currently leveraging Linked Open Data (LOD) is a popular research strand. [ 11 ] and [ 18 ] were among the first to use LOD for recommendation. The former use this information to build open recommender systems whereas the latter built a music recommender using collaborative filtering techniques. [ 4 ] was the first to really leverage LOD to build a content-based recommender and the first to exploit the semantics of the relations in the link hierarchy. They use LOD information from DBpedia, Freebase and LinkedMDB as the only background knowledge for a movie recommender system and show that thanks to this ontological information the quality of a standard content-based system can be improved. In more recent work, the semantic item descriptions based on LOD have been merged with positive implicit feedback in a graph-based representation to produce a hybrid top-N item recommendation algorithm, SPrank [ 17 ], which further underlines the added value of this kind of data. Moreover, in 2014 in order to spark research on LOD and content-based recommender systems, a shared task was organized by the same authors, i.e. the ESWC-14 Challenge1.

In content-based recommendation, the advances that have been made were made possible thanks to the availability of designated datasets. These include data for predicting music, Last.FM2, and or movies, MovieLens3. Up till now little research has been performed on other genres, such as books. The ESWC challenge, however, made a book recommendation dataset available which is mapped to DBpedia. DBpedia is a crowd-sourced community eo↵rt to extract structured information from Wikipedia and makes them available as linked RDF data [ 14 ]. This dataset will be used as our main data source. In this paper, we focus on the feature construction for a classifier in that we also incorporate semantic features based on the semantic frames present within the items to be recommended. This is, to our knowledge, the first approach that tries to leverage this kind of data and is one way of tackling the issue of Limited Content Analysis within recommender systems [ 4 ]. In order to validate these claims we will compare and combine our best system with a system exploiting LOD.

In this section we give some more information about why we believe exploiting frame information might help with recommendations. First, we introduce some basic concepts and theory after which we explain how we apply a state-of-theart semantic frame parser to our dataset and provide a first analysis. We hypothesize that a plot description tells more about a book than using more global semantic classification based on external semantic information as provided by the LOD cloud. This reasoning can be transferred to other data sources having a large number of textual information. 1http://challenges.2014.eswc-conferences.org/ index.php/RecSys 2http://labrosa.ee.columbia.edu/millionsong/ 3http://grouplens.org/datasets/movielens/

Following the basic assumption that the meanings of most words can best be understood on the basis of a semantic frame, FrameNet [ 9 ] was developed as a linguistic resource storing considerable information about lexical and predicate-arguments semantics in English.

FrameNet is grounded in the theory of frame semantics [ 7, 8 ]. This theory tries to describe the meaning of a sentence by characterizing the background knowledge required to understand this sentence. This knowledge is presented in an idealized, i.e. prototypical, form. A frame is thus a structured representation of a concept. It can be a description of a type of event, relation or entity, and the participants in it. In Table 1 we present an example of such a frame, KILLING. We see it is a semantic class containing various predicates, also known as lexical units (LUs), evoking the described situation, e.g. killer, murder, lethal. Moreover, it illustrates that within FrameNet each frame comes with a set of semantic roles, i.e. frame elements (FEs), which can be perceived as the participants and/or properties of a frame which are of course also lexicalized in the text itself, e.g. Killer: John, Instrument: with only a pocketknife.

FrameNet’s latest release (1.5) contains 877 frames and about 155K exemplar sentences.4 An interesting aspect of the FrameNet lexicon is that asymmetric frame relations can relate two frames, thus forming a complex hierarchy containing both is-a like and non-hierarchical relations [ 22 ]. In this work, we are particularly interested in the former type, also known as Inheritance relations. This type of relation entails that the child frame is a subtype of the parent frame. If we look for instance at our Killing example, of which the taxonomy is visualized in Figure 15, we are able to find out that this frame is a child of the frame Transitive action, which is in turn a child of both the frame Objective Influence and, more interestingly, the frame Event. This taxonomy thus enables us to find even more semantic properties about specific frames. 3.2

3.2.1

Book dataset

For the research described in this paper, we worked with the dataset of the ESWC challenge which is in fact a re4This release is available at http://framenet.icsi. berkeley.edu 5This graph was produced using the FrameGrapher tool, https://framenet.icsi.berkeley.edu/fndrupal/ FrameGrapher

The [Prince], the protagonist, is [named] Alexander.

His [father], [Prince] Baudouin, is [murdered] by T the [King] of Cornwall, [King] [March]. [When] L Alexander [comes] of [age], he [sets out] to Camelot O P to [seek] justice from [King] Arthur and to [avenge] the [death] of his [father]....

Leadership, Appointing, Kinship, Leadership, Killing, SE Leadership, Leadership, Calendric unit, M Temporal collocation,Arriving, Calendric unit, RA Departing,Seeking to achieve, Leadership, Revenge, F Death, Kinship. elaborated version of the LibraryThing dataset6. This dataset contains books that are part of a particular user’s online catalog containing the books he/she has read or owns. For the challenge, the books available in the dataset have been mapped to their corresponding DBpedia URIs [ 17 ]. Based on the available information we were able to download the plot description of each book from its corresponding Wikipedia page (this plot information is lacking in DBpedia). In this way we envisaged to investigate whether knowing more about what is actually happening in a book can enhance the recommendation. We worked with a subset by only including books of which a uniform and unambiguous DBpedia link was available and that actually contained plot information on Wikipedia. In total our final dataset contains 5,063 books with an average plot length of 312 words7.

In order to annotate the semantic frames, each plot was parsed using the state-of-the-art frame-semantic parser SEMAFOR [ 5 ]. This parser extracts semantic predicateargument structures from text using a statistical model and is trained on the FrameNet 1.5 release. It takes as input the text as such, performs some preprocessing steps and outputs on a sentence-per-sentence basis all frames that are present within a text. These frames are represented by one of the 877 possible frame names and also the lexical units and frame elements (both generic and lexicalized form) are output. An example is presented in Table 2. This is the plot description of the book The Prince and the Pilgrim. In the text itself, the lexical units evoking the frames are indicated in square brackets. The frames and LUs which are represented in bold are those frames which actually constitute an Event. Finding out which books are events can be done by exploiting the taxonomy (cfr. supra) which enables us in a way to find out more semantic properties of specific frames. Intuitively, we can state that especially those Event frames give most information about what is happening within a book: the above-mentioned book is clearly a revenge story. However, the other frames might also pinpoint important aspects, e.g. the repetition of the Leadership and Kinship frames could inform us that this novel is about royalty and family.

What this example also illustrates is that the SEMAFOR parser is not 100% accurate. For example, the name of a particular king – King March – is interpreted by the parser as evoking the frame Calendric unit. We should thus keep 6http://www.macle.nl/tud/LT/ 7This dataset will also be made available to the research community in due time in mind that a certain amount of noise is also introduced into our dataset. Moreover, some frames such as Arriving or Temporal collocation, are correctly labeled but do not really contribute interesting semantic information.

For all books in our dataset we parsed the plots using SEMAFOR, after which we also filtered out those frames which can have the Event frame as a parent. Some data statistics regarding these annotations are presented in Table 3, which reveal that the information we have available is rather skewed.

As previously mentioned, we hypothesize that using frames might represent di↵erent information than using semantic information represented in the LOD-cloud. The books dataset we have at hand is particularly useful to verify this claim since all books have been mapped to their DBpedia URIs.

In order to do so, we relied on a manual subdivision of all books in genres based on LOD. This classification was made by [ 23 ] by parsing the abstract (dbo: abstract), the genre ( dbo:literaryGenre, dbp:genre) and the subject (dcterms: subject) of each book against a regular expression pattern of thirty distinct genres. The authors performed this step to allow for more data coverage. However, by doing so they also made a combination of various LOD information categories which enables us to directly compare these with our semantic frames. If we have a look at our running example, The Prince and the Pilgrim,8, we notice that this book is classified under the Fantasy genre.

Based on this genre mapping, we calculated the gain ratio [ 19 ] of our semantic frames representation with relation to the genres, thus considering the frames as features allowing to do genre classification. These gain ratios can then be observed as feature weights, and ranked according to the amount of information they add to discriminating between the thirty possible genres. We start our analysis by first only considering the semantic frame annotations. It became apparent, however, that it might be more interesting to also closer inspect those frames which are Events since these intuitively better represent what is actually happening.

The result of these analyses in presented in Table 4. Because of space constraints, we only represent the five genres representing most books of our dataset. This table each time contains the ten top features (frames and events), i.e. those with the highest gain ratio. The cell colour represents the manual analysis, indicated in light grey are those frames and events occurring only within one particular genre. In darker grey the frames and events which are representative for a specific genre are indicated. Regarding the frames, we see that it is more dicult to find distinctive features correlating with the genre (light grey). In the upper part, only the Science Fiction and Crime genre contain truly representative 8http://dbpedia.org/resource/The Prince and the Pilgrim frames based on our manual analysis (dark grey). If we go to the level of the Events, we see that this already allows for finding more unique events per genre. Again, the Science Fiction and Crime genre are best represented. When we had a closer look at other discriminating features we found the same tendency. In the Crime genre, for example, other Events such as Verdict, Revenge, Execution, Robbery all appeared within the top twenty features.

From this analysis we could deduce that both the frames and events might deliver the same type of information as the LOD, with the events being more representative. However, what becomes clear is that the frames also contribute more information. They can represent what is happening within a book. If we again consider our running example (cfr. Table 2), which is classified as Fantasy, we feel that enriching a recommender with semantic frame, and especially with event information, might account for a better recommendation. This brings us to the actual experiments. 4.

For our experiments we focus on the generation of new, semantic features. In our experimental setting we aim to evaluate the contribution of those features and thus do not explicitly focus on engineering towards a top recommendation performance. 4.1

We opt to add our semantic features to an existing recommender system [ 23 ], which participated, and performed well, in the ESWC’14 Challenge. Though we do apply feature weighting and feature selection as described below, the overall item classification and collaborative-filtering elements of the base system remain unchanged. This allows us to directly compare the predictive power of the frame-based features with the DBpedia-based features used by the original system, in particular as both approaches are die↵rent utilizations of the same information source, i.e. Wikipedia, and dataset, i.e. the ESWC RecSys Challenge data.

We use a reduced version of the dataset, based on a filtering of the 5,063 books that were retained as having sucient plot information available (Section 3.2). This dataset has binary ratings and consists of 53,665 user-item-rating triples (6,162 users, 4,251 items) in the training data and 50,654 triples (6,180 users, 4,311 items) in the evaluation dataset.

Even though this is a binary classification task, we opt to output the positive class likelihood and not the final binary classification in order to avoid making a decision about the cut-o↵ for the likelihood values. Consequently, we evaluate with root-mean-squared error (RMSE) to capture also the degree of confidence between the classification and the goldstandard test dataset9. RMSE is calculated as: v

m RM SE = tuu m1 Xi=1(Xi xi)2 in which Xi is the prediction and xi the response value, i.e. the correct value for the task at hand, and m is the number of items for which a prediction is made. Speaking in practical terms, the lower the RMSE value the better, 9Obtained from the ESWC’14 Challenge Chairs upon request. because the closer the prediction confidence to the actual gold standard.

In addition, again motivated by wanting to avoid to choose a cut-o↵ point for the class assignment, we follow [ 12 ] and evaluate with a receiver operating characteristic (ROC) curve and also compute the area under the curve (AUC) for it. While in contrast to RMSE, the ROC curve is computed only on the relative ordering of the predictions sorted by confidence values, it oe↵rs the advantage of understanding how a classifier would perform given die↵rent cut-o↵ values. In addition, with ROC we can compare against recommender systems that output only an (implicit) ranking and no class confidence values.

The base system by [ 23 ] we extend is a simple contentbased recommender which trains two Na¨ıve Bayes classifiers10 on book features acquired from DBpedia, one global classifier as background model and one per-user classifier to capture individual preferences, trained on a user-neighborhood of variable size. In our experiments, we leave this setting unchanged but only vary the die↵rent features for item representation. We experimented with five di↵erent feature representations, which is explained in closer detail in the next section. 4.2

We report experimental results for the di↵erent feature settings in Table 5. Overall, the two best performing frame features are the Frame elements and the Frames+LUs+FEs, both achieve an RMSE of 0.6036. We see that the best result is obtained when making the combination between the Frame elements and the LOD system, RMSE of 0.5982. Looking at the AUC for the ROC curve, both features still perform very well, but not as good as the DBpedia features alone, which achieve the best overall AUC of 0.5588.

Considering the RMSE values, we observe that the majority baseline is easily outperformed by all die↵rent settings. Looking at the bag-of-words baseline, however, illustrates that having the words of the plot available for recommendation is already a quite dicult to beat baseline. 11We looked at the most frequent LCS nodes and excluded the first 10 generic nodes such as Artifact, Relation, Intentionally a↵ect, Gradable attributes, Transitive action. 12Preliminary experiments revealed that keeping all features as well as doing classifier-based features selection with OneR [ 13 ] with 5-fold-cross-validation on the training data constantly underperformed against this setting. Contrary to our expectations, our settings with only frames or events do not outperform this baseline. We do see that the events as such, which constitute a much smaller feature space, perform slightly better than the frames. The bagof-words baseline is only outperformed when using features actually presenting some sort of word filtering mechanism: the Frame Elements are the lexical representation of words which are evoked by certain frames in the form of semantic roles. Even though these features are extracted from the text, it performs better than the bag-of-words (Words as such) baseline approach (0.6145) which does not make use of any semantic information. Analyzing the RG-ranked feature attributes revealed that also for the other best frame approach Frames+LUs+FEs, the dominant attributes are the Frame elements, these were ranked highest. What is strange is that we do not find a similar trend when performing the same combination with our Event frames. This is probably because the feature space is too small to make a well-informed decision.

Figure 2 presents the ROC curves for our features, for the sake of readability only the most interesting curves are plotted. As to be expected from the AUC values, all curves are very close together. Besides not being far away from the diagonal, for no curve a clear cut-o↵ value is recognizable. We observe that the DBpedia features are slightly better for the left and partially the middle part of the curve, leading to the interpretation that those features are superior for recommender systems which focus on quality. Comparing the best frames-based approach (FEs) with the bag-of-words baseline (Words), we see that FEs are mostly better than just words, with some exception around a false positive rate of around 0.23.

We also compare our system with the hybrid recommender system from Ristoski at. al. [ 21 ] (AUC 0.5848), which was the second best system of the ESWC challenge and performed essentially equally well as the winning system. That system combined many di↵erent features, not only LOD, but also user ratings and explicit collaborative filtering approaches.13 13As that system only outputs scores for the purpose of ranking, we transformed those into confidences by dividing each e t a r e v iit s o p e u r T Looking at the ROC curve, it becomes clear that incorporating more and diverse features is beneficial in this setting. However, we have to note that this system combines di↵erent recommenders using the Borda rank aggregation method, which was not learned on the training data but manually selected while having knowledge about the test dataset (see also the comment below on our own combination model).

If we compare our best semantic frame results with the systems leveraging Linked Data, we see that we achieve a better performance (RMSE of 0.6022) when using the DBpedia features alone and that we get the best overall results when combining both our best system with the Linked Data (RMSE of 0.5982). In this way it appears that combining semantic information from di↵erent sources, i.e. from the linguistically grounded frames features and the explicit, ontology grounded DBpedia features, is beneficial in this setting. The AUC results, however, do not corroborate this finding.

Last, when not learning LOD+FEs together in one model, but separately and combine results with a simple linear combination (these results are presented in brackets in Table 5), as also done by [ 1 ], with = 0.5, we achieve better results (RMSE of 0.5664 and AUC of 0.5571). However, this notable improvement depends in the end on our knowledge of the test dataset, as it influenced our choice of a linear combination, instead of learning the combination of the different classifiers on the training data. Strictly speaking, this is thus not a valid experimental result, nevertheless it indicates there is most likely a better hybrid design with feature combinations that will better utilize the semantic frame features and should yield better results.

score by a constant.

In this paper we have presented an alternative approach to add semantic information to a content-based book recommender system. We directly compared the addition of text internal semantic frame information with text external ontological information based on Linked Open Data (LOD), a popular research strand.

We have shown that parsing the book plots with a state-ofthe-art semantic frame parser, SEMAFOR, delivers valuable additional semantic information. This information could enable a system to fully grasp what is happening within a book. One of the added values of FrameNet is that all frames are related in a taxonomy which allows you to pinpoint those Events forming the key components of a book. Based on a direct comparison between the frames and events and a list of genres derived from DBpedia attributes, we have shown that although these data sources show some similarities, the semantic frames should be able to represent more specific information about what is happening in a particular book.

In order to test this claim in closer detail, we have performed experiments where the focus was on generating new semantic features and find out what these can contribute to a book recommendation system using one global classifier as background model and one per-user classifier. We see that exploiting semantic frame information outperforms a standard bag-of-words unigram baseline and that especially incorporating frame elements and lexical units evoking the frames allows for the best overall performance. If we compare our best system to a system levering semantic LOD information, we observe that our frames approach is not able to outperform this system. We do find, however, that if we combine these two semantic information sources into one system we get the best overall performance. This might indicate that combining semantic information from di↵erent sources, i.e. from the linguistically grounded implicit frame features and the explicit, ontology grounded DBpedia features, is beneficial.

This work has inspired many ideas for future work. Considering the current setup, we are aware that we completely relied on the output of one semantic frame parser, i.e. SEMAFOR. We believe that using a filtering mechanism beforehand, e.g. to filter out those frames and or events which are less meaningful or noisy, or that by applying a di↵erent parser or event extraction techniques new lights can be shed on the added value of this type of information. Also, since we now only relied on Wikipedia to extract book information, we had to reduce an original larger book data. We realize a lot of additional information about books can be found online, for example on Google Books, Amazon, GoodReads, etcetera. Also the same techniques can be used to extract other types of information from both the items and users under consideration for the recommendation task.

As mentioned at the end of Section 5 we would like to further investigate whether another hybrid design might yield better results. In this respect, it would be interesting to plug our semantic knowledge in a collaborative-filtering approach to see whether this can actually help the overall performance. Using our semantic frames we could also inspect in closer detail typical problems recommender systems face such as cold-start and data sparsity.

The work presented in this paper has been partly funded by the PARIS project (IWT-SBO-Nr. 110067). Furthermore, Orph´ee De Clercq was supported by an exchange grant from the German Academic Exchange Service (DAAD STIBET scholarship program). We would like to thank Christian Meilicke for his help in providing the manually derived genres and his help with building the original ESWC recommender system.