Current-generation recommendation algorithms are often focused on generic ratings prediction and item ranking tasks based on a user's past preferences. However, many recommendations are more complex with specific criteria and constraints on which items are relevant. This paper focuses on a particular type of complex recommendation needs: narrative-driven recommendation, where users describe their needs in short narratives, often with one or more example items that fit that need, against a background of historical preferences that may not be spelled out in the narrative, but do play a role in their considerations. Previous work has shown that numerous examples of such complex needs exist on the Web, yet current-generation systems ofer limited to no support for these needs. In this paer, we focus on narrative-driven book recommendation in the context of LibraryThing users posting recommendation requests in the discussion forums. We propose several new algorithms that take advantage of these narratives and example items as well as hybrid systems, the majority of which significantly outperform classic collaborative filtering. We show that narrative-driven recommendation is indeed a complex scenario that requires further study. Our findings have consequences for system design and development not only in the book domain, but also in other domains where users express focused recommendation needs, such as movies, television, games, and music.
Research on recommendation algorithms for ratings prediction and item ranking has resulted in a better understanding of these tasks Knowledge-aware and Conversational Recommender Systems (KaRS) Workshop 2018 (co-located with RecSys 2018), October 7, 2018, Vancouver, Canada. 2018. Copyright for the individual papers remains with the authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors.. and an array of algorithms with state-of-the-art performance. However, not all recommendation needs can be solved by providing a ‘generic’ set of recommendations based on a user’s past preferences. In many cases, recommendation is a more complex problem and often just a single stage in a user’s more complex background need. These needs can place a variety of constraints on which recommendations are interesting and appropriate to the user—and they are unlikely to be satisfied well by the current generation of ratings prediction and item ranking algorithms.
Relatively little research has been done on these complex
recommendation needs and how much of a problem they still pose for
current-generation recommender systems. In 2017, Kang et al. [
However, neither Kang et al. [
Because of the relative novelty of our specific book recommendation scenario, we start by describing it and our methodology in greater detail in the next section. Sections 3 and 4 respectively describe algorithms that utilize the textual narratives and example items to produce relevant book recommendations. Our eforts at hybrid recommendation are described in Section 5. We discuss related work in Section 6 and conclude in Section 7. 2
Bogers and Koolen [
Figure 1 illustrates our complex NDR scenario on the LT forums. Users can post requests for book recommendations to a forum discussion group, describing their recommendation need containing both a narrative description of the kinds of books they want to read and examples of (un)suitable) books and/or authors. These examples form a so-called mini-profile , a temporary representations of that user’s current preferences. Other users can reply to these posts and come with suggestions for relevant books or authors to read and explore. Many of the users interacting in these discussion threads have their own LT profile of their book preferences. On LT, these books are represented with diferent types of metadata and user-generated content. We wish to explore the role these user preferences and book representations can play in generating accurate book recommendations.
We expect that tailored recommendations based on individual requests will provide better recommendations than generating a list of book recommendations using a state-of-the-art CF algorithm based on a user’s entire profile. 2.1
To develop and evaluate our NDR algorithms, we use the
Amazon/LibraryThing (A/LT) dataset. The A/LT dataset has been used
in the INEX Interactive Track [
work on LT, so for all intents and purposes we are recommending books from a collection of 2,654,082 book records, corresponding to 1,904,950 distinct works.
To enable recommendations based on the example authors in mini-profiles, we assigned unique author IDs to each author in the A/LT collection. This was necessary, because the A/LT dataset has inconsistent author naming and no author identifiers. We matched duplicate authors by converting their name strings to lowercase and removing punctuation and then assigned IDs. This reduced the list of authors from 968,703 author strings to 916,479 unique author IDs. We also checked for matches between names with and without middle names and/or initial and collapsed them into the same author ID. While not perfect—John Smith and John D. Smith need not refer to the same person—inspection on a random sample of 100 automatic matches revealed an precision of 89%. Applying this rule further reduced the number of unique authors from 916,479 to 849,578. We then matched all example authors in the mini-profiles to their A/LT author IDs.
In addition to the A/LT dataset, we also use a crawl of LT user profiles with cataloging dates, tags, and ratings. This crawl was conducted in 2012 and 2013 and contains over 66K LT user profiles, with over 29M cataloging transactions and 4.5M distinct books. The LT user profiles dataset has 4,409,399 ratings by 38,174 users, representing less than 15% of the total number of cataloging transactions. Over 42% of users have assigned no ratings at all, and the 58% who have, rate only a small fraction of the books in their catalog. We therefore ignore the ratings and treat each transaction as implicit positive feedback on user preferences. 2.2
We use the same subset of 974 narrative requests from the LT
forums collected by Bogers and Koolen [
When recommending based on these example books and authors, we do not distinguish between positive and negative examples. Instead, we consider all example books as equally relevant sources of recommendation; we are aware of the bias this may introduce. Sentiment analysis could conceivably be used to determine whether examples are cast in a positive or negative light. However, this would also add another layer of complexity and thereby make it harder to tease apart the accuracy of our recommendation algorithms from our example sentiment detection component.
We split the 331 requests into a random 50-50 train/test split with 166 training requests and 165 test requests. Default train/test splits may have more training items, but none of our proposed algorithms have more than one parameter to be optimized. In addition, the recommendation narratives show great variety in terms of narrative
User preferences
for Books represented by has has
Thread length as well as example counts and popularity; a larger test set would allow for better analysis of our algorithms.
The requests were posted in the period 2006–2012, with some users posting multiple requests during this period. At the time of these requests, the same user will have diferent user profiles. This introduces a challenge in training CF models, as their diferent profiles should not be represented as diferent users. For a user u posting request r1 at time t1 with profile p1 and r2 at time t2 with profile p2, we need to either update the model in between requests, or use only the profile at p1. There are some users with multiple requests in our dataset, so to keep our baseline setup simple for training, we represent as user u with multiple requests at t1 and t2 by their profile at time t1. For 53 out of 331 requests, the pre-request profile ended up being based on the earliest request.
We realize that our evaluation uses a relatively low number of requests and only a single train-test split, but wish to emphasize that NDR is diferent from generic ratings prediction and top-k recommendation, and instead a hitherto unaddressed problem. In our scenario, we are dealing with recommendation needs with a strong IR element, where evaluation using 50-100 topics is common. 2.3
The book requests can be evaluated from diferent perspectives, representing diferent recommendation tasks. First, there are the user profiles with all the books that each user added to their personal catalog, with a cataloging date that can be used to determine which books were cataloged before a user posted their request (= pre-request cataloged items) and which books were cataloged afterwards (= post-request cataloged items). For a typical recommendation scenario, the former represent the user’s profile, the latter the books to recommend. The request also receives replies from other users with suggestions of books relevant to the request. Other users often report having consulted the requester’s profile to target their suggestions, although they sometimes suggest books the requester already cataloged. These suggestions represent a recommendation task more focused on the recommendation need. A third perspective is the intersection of the suggestions and the post-request cataloged items. We refer to these as post-request cataloged suggestions (PCS). They represent focused and successful recommendations.
For NDR, the recommendation needs are central, so we use the
suggestions as relevant recommendations, with the PCS as the most
relevant ones. Post-request cataloged items that are not suggested
in the request thread are considered outside the focus of the
recommendation need. We therefore adopted the relevance grading used
in clef 2016 Social Book Search Lab [
We assume this is a high-precision task, so we focus on top-N measures. We are using graded relevance judgements, so we use nDCG@10 as our main metric and report MRR to provide further insights into the rank where users can find the first relevant item. All reported statistical significance tests are one-tailed bootstrap with 100,000 re-samples at levels p < 0.05 and p < 0.01. 2.4
As argued in Section 1, narrative-driven book recommendation ofers a unique and complex recommendation scenario. Generic book recommendation can straightforwardly be addressed using state-of-the-art CF algorithms, such as matrix factorization. However, applying CF to the problem of NDR is unlikely to provide many relevant, on-topic recommendations. To provide evidence for this assertion, we compare our narrative- and example-driven algorithms to a CF baseline.
We used the LightFM toolkit2 by Kula [
1. 2. 3.
(f) (a) (b) up to 100 epochs. Optimal performance was achieved with WARP running for 50 epochs and 300 factors. The narrative requests focus the user’s recommendation need on specific criteria. We expect that basing recommendations on the entire user profile leads to poor performance. However, given the modest number of examples in the mini-profiles, we expect CF to have little signal to work with, akin to the cold-start problem, resulting in poor performance.
For the CBF approaches, we used the language modeling
approach as implemented in the Indri toolkit. Optimal retrieval
parameter values3 for the diferent versions of the A/LT collection
are based on earlier experiments that share a similar setup in that
the narrative was also used as a query [
One source of information about the user’s recommendation need is the narrative posted to the LT forums. As shown earlier in Figure 1, each request consists of a narrative and, as part of that narrative, a set of example books and authors. In this section, we propose a NDR algorithm that uses the textual representation of the narrative to identify and recommend related books. Figure 2 shows a schematic illustration of our approach.
For each request, we take the narrative description and use CBF to match it against the collection of 2.6 million books. Here, the requester’s narrative is treated as a temporary representation of 3An overview of these parameter values can be found at http://anon.ymiz.ed/url.
MRR 0.024 0.059△ 0.026 0.052 0A.u0t4h8o▲red 0.111▲ 0b.0o6o7k▲(s) 0.164▲ 0.074▲ 0.181▲ 0.074▲ (a) (f0).180▲ their user profile. Each book in the A/LT colle(cct)ion is represented using diferent types of metadata fields and m a(dt)ched against the narratives using retrieval algorithms. In our(ee)xperiments, we distinguish between six categories of metadata representations. One Authored category of metadata (a) contains the core m(eg)tadata, such asbotoitkl(es), author, and publisher. Curated metadata (b), contains index terms and Dewey codes form another category, while Tags (c) are the set of distinct tag1s. assigned to each book by the LT users. The1A.mazon reviews (d) fo2r.m another collection representation; the c2o.mbination of reviews and tags (e) represents the UGC associated with each book. Fi3n.ally, combining all metadata fields (f) is th3e. sixth collection representation. 3.1
Table 1 shows the results of the narrative-based recommendation algorithm. Diferences are tested for statistical significance using one-tailed bootstrap with 100,000 resamples. CBF based on narratives outperforms CF, which is not surprising, given the focused nature of the task. CF will recommend items from similar users, but not necessarily within the confines of the information need. Within the CBF approaches, user-generated content clearly outperforms other metadata, possibly because it reflects the language of requesters better than e.g. curated metadata. Reviews perform significantly better than tags. The combination of reviews and tags is also significantly better than tags alone, but not than reviews alone. Reviews use a broader vocabulary than tags to match narrative requests, and maybe cover book aspects that are both relevant in reviewing and in searching. We found no correlation between the performance of diferent NDR approaches and the length of narratives or the number of examples. Furthermore, we looked at the popularity of recommended items, in terms of how many LT users cataloged them. CF tends to recommend items from the more popular end of the spectrum than the suggestions (including the successful suggestions) and the examples, while CBF tends to recommend more items from the long tail, especially when using curated metadata. Here another advantage of UGC is revealed, as it favors more popular books compared to curated metadata, in that popular books have more UGC than obscure books, so have 1. 2. 3.
EDR-1 a higher probability of being retrieved, but favors more obscure books compared to CF in that it penalizes the topic drift in the huge amounts of UGC of popular books. to calculate book-to-book similarities as option (g). Sections 4.1 and 4.2 describe the two algorithms in more details as well as their efectiveness compared to traditional CF and NDR. 4
The second source of information about the user’s recommendation need is the mini-profile they provide, i.e., the example books and/or authors they mention as relevant to their current need. We proposed two algorithms that each uses one of the example types as input: (1) example-driven recommendation using similar books (EDR-1), and (2) example-driven recommendation using similar authors (EDR-2). Figure 3 shows a schematic illustration of both algorithms.
Both algorithms can use diferent representations of the books and authors, corresponding to the six metadata groups introduced in Section 3: (a) traditional metadata, (b) curated metadata, (c) tags, (d) reviews, (e) UGC (= tags and reviews combined), and (f) all representations combined. In addition, we used the user-item matrix 4.1
Our first example-driven algorithm EDR-1 uses the example books in the mini-profiles and is visualized on the left-hand side of Figure 3. As mentioned in Section 2, we do not distinguish between positive and negative examples, but consider all example books as relevant sources of recommendation. The EDR-1 algorithm consists of two basic steps: (1) identifying unseen books that are similar to each of the example books, and (2) merging these sets of similar books to get a single ranked list of book recommendations.
(1) Identifying similar books. In the first step of EDR-1 for each example book mentioned in a mini-profile, we can identify similar books using one of two data sources: metadata representations of each book, or item-to-item similarities calculated on the user-book matrix. These roughly correspond to the diference between CBF and CF. With regard to metadata-based matching, the metadata representations of each example book are matched against the 2.6 million book representations in the A/LT collection. Matching can be done for each of the six representation collections (a)-(f) introduced in Section 2. Each example book representation is converted into an Indri-safe query and matched against each of the collection’s book representations, producing ranked lists of book recommendations. Since the collection is indexed at the level of individual ISBNs, the same book can be represented by multiple XML representations, one for each ISBN. In each case, we pick the representation that contains the largest amount of text as the canonical representation of a book. Not every LT work could be mapped to an ISBN identifier; 52 out of 740 (7.0%) of all of our mini-profile example books could not be mapped. This means that for requests without mappable books no recommendations could be generated.
Another issue is that for the richer collections, such as (f) with all metadata fields combined, these book representations quickly become too long for eficient document retrieval. For instance, the median word count of book records for the (f) representation is 27,988 words, while the longest representation contains 210,790 words. Query-document matching in IR toolkits like Indri is optimized for relatively short queries, so to speed up matching against 2.6 million book representations, we capped each book representation at an empirically determined upper limit of the 500 most frequent words in each representation.
To identify similar books based on user preferences we calculated item-to-item similarities from the user-book matrix using the CF model trained with LightFm, as described in Section 2.4.
(2) Merging sets of similar books. Of the 298 requests with
example books, 120 requests (40.3%) contain only a single example book.
In these cases the ranked list of relevant book representations
directly corresponds to the list of book recommendations after being
mapped back from ISBNs to LT works. The remaining 59.7% contain
multiple example books, which means they yield multiple results
lists per request, with many books occurring on several lists. We
merge these lists using the CombSUM fusion method introduced
by Fox and Shaw [
One parameter that could influence recommendation accuracy is the number of similar books to include from each results list. Including a larger number of results—the top 500 vs. the top 10—is likely to result in more books occurring in multiple lists. Another perspective is to see this as the number of nearest (book) neighbors to recommend. We denote this parameter as k and optimized its value on our training set of 166 requests, separately for each of our seven collection representations (a)-(g). We varied the value of k in steps of 50 up to k = 1000. The line chart on the left-hand side of Figure 4 shows how varying k influences recommendation performance. Overall, performance tends to be fairly stable with regard to k. The optimal values of k for EDR-1 are included in Table 2. 4.2
Our second example-driven algorithm EDR-2 uses the example authors in requests, and is shown on the right-hand side of Figure 3. It consists of three stages: (1) identifying authors that are similar to each of the example authors; (2) merging these sets of similar authors to get a single ranked list of similar authors; and (3) replacing each similar author by their authored books and merging these to get a single ranked list of book recommendations.
(1) Identifying similar authors. In the first stage of EDR-2 for each example author mentioned in a mini-profile, we identify similar authors by matching their metadata representations against those of all authors in the A/LT collection. To make this possible, we re-indexed the A/LT collection at an author-level by aggregating all of an author’s book representations into a single author-level representation. For each book, we again took the associated ISBN with the largest amount of text. These book representations were then aggregated into a single author representation, one for each of the six metadata representations (a)-(f) introduced in Section 2. This resulted in an author-centric collection containing 849,578 author representations. Each example author’s representation is converted into an Indri-safe query and matched against each of the collection’s author representations, producing a ranked list of author recommendations. As with EDR-1, we capped each author representation at a limit of the 500 most frequent words.
(2) Merging sets of similar authors. The 121 requests with example authors contained an average of 2.0 author per request. As for EDR1, we used CombSUM fusion to merge normalized author lists belonging to the same request. We also examined the influence of the number of similar authors k to include from each results list; the results of this optimization can be found in Because there are fewer authors than books in the A/LT collection and because expanding authors to their authored books results in a much larger number of recommended books, we varied k for EDR-2 in smaller step sizes. The line chart on the right-hand side of Figure 4 shows how k influences performance. There are more pronounced peaks for EDR-2 than for EDR-1, but , performance tends to be fairly stable with regard to k. The optimal values of k for EDR-2 are also included in Table 2.
(3) Inserting & merging authored books. To arrive at lists of recommended books, we expanded each recommended author by their authored books and assigned each book the similarity score of that author. Because of errors in mapping ISBNs to work IDs, some books occurred multiple times for the same request; we again summed all of a book’s individual scores to produce a single list of recommended books, capped at 1000 results. 4.3
Table 2 shows the results of EDR-1 and EDR-2, with scores over all test topics and over only those that have example books and example authors respectively. EDR-1 is less efective than NDR but more than EDR-2, with UGC again more efective than other metadata. This is possibly due to requests having fewer authors than books or that (especially prolific) authors introduce more topic drift. EDR-2 is more efective than CF for requests with example authors when using UGC. 5
In this section we look at hybrid systems that combine NDR and
EDR. More specifically, we apply results fusion, where the outputs
of diferent recommendation algorithms are combined into a single
results list, which is a form of weighted hybridization according to
the hybrid recommendation taxonomy by Burke [
Table 3 contains the results of the hybrid recommenders, which we compare against the best single approach from the previous sections, i.e. the NDR (g) system which uses UGC. We only report the best performing weights. We discuss the results with a per-request analysis to explain the diferences. Combining NDR and EDR-1 (f) results in a significant improvement in nDCG@10, although not in MRR. The narratives and examples complement each other as representations of the information need. The combination increases the number of requests with non-zero scores from 55 to 68, and also improves more request scores than it hurts.
Combining NDR with EDR-1 with user preferences ((g)) strongly improves precision for 20 requests, but slightly hurts it for 35 requests, which is why scores are higher, but not significantly so. Again, user preferences are good for early precision, but beyond that the signal gets so weak that popular items pollute the ranking. The combination with EDR-2 (f) run leads to only 17 requests having a diferent score, but significantly hurts performance. The similar author recommendations appear to cause topic drift. Combining all three approaches leads to small but statistically insignificant improvements, mainly reducing the number of requests scoring zero. Combining EDR-1 and EDR-2 leads to small improvements over the individual EDR approaches, again, mainly by reducing the number of zero-scoring requests.
EDR-1 (Recommendation using similar books) 0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 1000 Metadata Curatedmetadata Tags Reviews Reviews+Tags Allfields
EDR-2 (Recommendation using similar authors) 0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 1000
Metadata Curatedmetadata Tags Reviews Reviews+Tags Allfields
Overall, the hybrids exploit the complementarity of the data sources, leading to more requests with relevant recommendations, although it is hard to improve over the competitive NDR baseline. However, our experiments do show that combining recommendation based on the narrative and the examples can lead to significantly improved performance.
Our work on book recommendation is far from the first: numerous
book recommenders have been proposed over the past years. Most
of the related work has focused on exploring and comparing CF,
CBF and graph-based algorithms for generic book ranking [
Our focus on expressing recommendation needs through textual
narratives is reminiscent of the work by Adomavicius et al. [
Narrative descriptions of needs and interests are typical of
conversational recommendation, where one person describes the kind
of items they like and what they would be interested in, while
others provide suggestions and explanations. [
The narrative requests we use for recommendation also bear
similarities to online product reviews in terms of their composition
and complexity. Reviews cover diferent aspects of a product with
the author describing likes and dislikes. The overall review score
of a product can be seen as personal ranking of the importance of
aspects. O’Mahony and Smyth [
Our work on example-driven recommendation also has a few
parallels with earlier work. For instance, Liu et al. [
Our focus on complex recommendation mirrors the increased
focus in IR on complex search tasks and how best to support them
[
In this paper we explored techniques to address narrative-driven recommendation needs. This is a novel, complex scenario in which users have specific recommendation needs that they express in a request containing both a natural language description as well as a mini-profile containing relevant example of books and authors. The combination of these complex user needs and the richness and heterogeneity of the item descriptions poses many challenges for designing recommender systems that support this task.
For the narrative part of the requests, we experimented with CBF techniques on a range of book metadata and user-generated content. We found that the latter is more efective as it more closely matches the vocabulary of the user’s narrative and strikes a balance between popularity and specificity. For example-driven recommendation (EDR) on the mini-profiles we experimented with both CBF and CF approaches, and found that example books ofer more focus and thereby better recommendations than example authors. Again, UGC is more efective than curated metadata and CF-based item-to-item recommendation. Finally, we looked at hybrid systems that combine NDR and EDR and found that NDR uses the more important signal, but that carefully weighted combinations can lead to significant improvements, especially in providing more requests with at least some relevant recommendations.
Overall we found that NDR is a challenging task that requires multiple data sources and algorithms to solve. Nevertheless, there are several issues that future work should address. One of them is the detection of and diferentiation between positive and negative examples using sentiment analysis techniques; currently all examples are treated as positive examples, but this likely afects performance negatively. Other algorithms, such as those from work on conversational recommendation, or graph-based algorithms on multi-partite networks containing user, book, author and tag nodes could perhaps provide a more integrated approach to EDR. Finally, testing our algorithms on other domains, such as movies, games and music would be required to determine the generalizability of our findings.