Research on recommender systems is a challenging task, as is building and operating such systems. Major challenges include non-reproducible research results, dealing with noisy data, and answering many questions such as how many recommendations to display, how often, and, of course, how to generate recommendations most effectively. In the past six years, we built three research-article recommender systems for digital libraries and reference managers, and conducted research on these systems. In this paper, we share some experiences we made during that time. Among others, we discuss the required skills to build recommender systems, and why the literature provides little help in identifying promising recommendation approaches. We explain the challenge in creating a randomization engine to run A/B tests, and how low data quality impacts the calculation of bibliometrics. We further discuss why several of our experiments delivered disappointing results, and provide statistics on how many researchers showed interest in our recommendation dataset.
Recommender systems is a fascinating topic for both researchers and industry. Researchers find in recommender systems a topic that is relevant for many disciplines: machine learning, text mining, artificial intelligence, network analysis, bibliometrics, databases, cloud computing, scalability, data science, visualization, human computer interaction, and many more. That makes research in recommender systems interesting and creates many opportunities to cooperate with other researchers. For industry, recommender systems offer an opportunity to provide additional value to customers by helping them finding relevant items. Recommender systems may also provide a justification to store user-related data, which may be used for generating additional revenue. A long version of this article is available online https://arxiv.org/abs/1704.00156 In addition, recommender systems may even become a major part of the business model, as companies such as Netflix, Spotify, and Amazon demonstrate.
Over the past six years, we built, operated, and researched three research-article recommender systems in the context of digital libraries and reference management. The work was often rewarding, but also challenging and occasionally even painful. We share some of our experiences in this article. This article is not a research article but a mixture of a project report, lessons learned, text-book, and summary of our previous research, enriched with some novel research results.1
The primary audience of this article are researchers and developers who think about developing a real-world recommender system for research purposes, or for integrating the recommender system on-top of a real product. While our focus lies on recommender system in the context of digital libraries and reference managers, researchers and developers from other disciplines may also find some relevant information in our article. As this paper is an invited paper for the “5th International Workshop on Bibliometricenhanced Information Retrieval”, we particularly discuss our work in the context of bibliometric-enhanced recommender systems and information retrieval. 2 2.1
In 2009, we introduced SciPlore MindMapping
Docear2 is the successor of SciPlore MindMapping, pursuing the same goal, i.e.
enabling researchers to manage their references, annotations, and PDF files in mind maps
Our latest recommender system is Mr. DLib4, a machine-readable digital library that
provides recommendations as-a-service to third parties
To build and operate a recommender system, more than just knowledge about
recommender systems and related disciplines such as text mining and machine learning is
required. Server administration, databases, web technologies, data formats (e.g. XML
or JSON), and data processing (crawling, parsing, transforming) are probably the most
important ones, but also knowledge about software engineering in general (e.g. agile
development, unit testing, etc.), scalability, data privacy laws, and project management
is helpful. Niche knowledge that does not directly relate to recommender systems may
also be beneficial and lead to novel recommendation approaches. For instance,
knowledge in bibliometrics could help to develop novel re-ranking algorithms for
content-based research-paper recommender systems
(No) help from the literature
There are hundreds of research articles about recommender systems in Academia
Even if evaluations were sound, recommendation effectiveness may vary a lot. In
other words, only because a recommendation approach performed well in one scenario,
does not mean it will perform well in another scenario. For instance, the TechLens team
proposed and evaluated several content-based filtering (CBF) and collaborative
filtering (CF) approaches for research-paper recommendations. In one experiment, CF and
CBF performed similarly well
Some authors used bibliometrics to enhance recommender systems in digital
libraries. Popular metrics were PageRank
Our own research confirms that recommendation approaches perform very
differently in different scenarios. We recently applied five recommendation approaches on
six news websites
0.40 ino 0.30 isc 0.20 reP 0.10
0.00
User-based CF
Mst. ppl. sequence
Item-based CF
Content-based
Most popular
There are several potential reasons for the unpredictability. In some cases, different
evaluation methods were used. In other cases variations in algorithms or user
populations might have had an impact
To build an effective recommender system, A/B testing is essential. This means, the recommender system needs a pool of recommendation algorithms to choose from, and a logging mechanism to record, which algorithm was used and what the user feedback was (e.g. how many recommendations were clicked or downloaded). Unfortunately, in most situations, a simple A/B test with only two alternative algorithms will not be enough. In Mr. DLib, we implemented a “randomization engine” that first picks from several recommendation classes randomly, and then varies the parameters of the recommendation algorithms. The recommendation classes are content-based filtering (90% chance), stereotyping (4.9% chance), most-popular recommendations (4.9% chance), and random recommendations as baseline (0.2% chance). For each of the recommendation classes, the randomization engine chooses some parameters randomly. For instance, when content-based filtering is chosen, the system randomly selects whether to use “normal” terms, or key-phrases5. When key-phrases are chosen, the system randomly selects if key-phrases from the abstract, title, or title and abstract are used. Then the system randomly selects if unigrams, bigrams, or trigrams are used. Then the system randomly selects if one, two, three, … or twenty key-phrases are used to calculate document similarity.
Once the recommendations are calculated, the randomization engine chooses
randomly if the recommendation candidates should be re-ranked based on bibliometrics.
Re-ranking means that from the top x recommendations those ten documents are
eventually recommended that have, for instance, the highest bibliometric score. Again, there
are many parameters that are randomly chosen by the randomization engine. The engine
selects the bibliometric (plain readership count, readership count normalized by age of
the document, readership count normalized by the number of authors, etc.), the number
of recommendation candidates to re-rank (10 to 100), and how the bibliometric and text
relevance scores should be combined (bibliometric only, multiplication of scores, and
some more variations). While we currently only work with readership data from
Mendeley, we plan to obtain additional data from sources such as Google Scholar, Scopus,
or Microsoft Academic
Developing such a randomization engine is a non-trivial task and we are not perfectly satisfied with our solution. It occurs too often that a fallback algorithm must be used because the randomly assembled algorithm cannot be applied because, for instance, a document does not have 20 key-phrases in its title. 5 Keyphrases are the most meaningful phrases describing a document. Keyphrases are extracted with stemming, part-of-speech tagging and other mechanisms that we describe in an upcoming paper. 3.4
Crucial for content-based recommendations is the quality of metadata. When using publicly available datasets, the quality may often be good because datasets were designed and maybe even curated to be published. In the real-world, however, data quality is often low. For instance, some of the most productive “authors” in Mr. DLib’s database are “et al.”, “and others”, and “AnoN”. Obviously, these are no real authors. When this data is used e.g. to recommend papers of co-authors, or to re-rank recommendations based on h-index, the resulting recommendations will be of suboptimal quality.
Cleaning data from third parties is a labor-intensive and usually boring task (it is
much easier to motivate a colleague to implement a novel recommendation algorithm
than convincing a colleague to spend some weeks cleaning data in a database). Data
cleaning becomes particularly challenging, when the data comes from a third party and
the data are updated occasionally by the partner. In that case, one would need a process
to decide how to deal with the updated data and judge if the new data from the partner
is better than the manually changed data in our system. From our experience in other
projects, we know that manually cleaning data usually causes many problems.
Therefore, we decided to do no manual data cleaning in Mr. DLib, and only apply a few
heuristics such as ignoring “et al.” when calculating bibliometrics.
On the Internet, users tend to be impatient: the longer they wait for content, the less
satisfied they become
It may be that a user who must wait too long, leaves the web page and does not even see the recommendations.
The re-ranking of recommendations based on bibliometrics requires rather a lot of calculation (primarily due to the randomization engine and because we store many statistics when calculating the bibliometrics). Consequently, when evaluating the effectiveness of bibliometric re-ranking, one need to additionally consider if the additional effectiveness if worth the additional time users need to wait. 4.2
In the recommender-system community, it is often reported that a recommender system
should try to avoid making ‘bad’ recommendations as this hurts the users’ trust. In other
words, it is better to recommend 5 good and 5 mediocre items than recommending
9 excellent but 1 bad item. To avoid bad recommendations, some relevance score is
needed that indicates how good a recommendation is. Ideally, only recommendations
above a certain threshold would then be recommended. However, at least Lucene has
no such threshold that would allow a prediction of how relevant a recommendation is7.
The Lucene text relevance score only allows to rank recommendations for one given
query and compare the relevance of the results returned for that one query. In a recent
analysis, we found that Lucene relevance scores and CTR correlate, but still it is not
possible to avoid “bad” recommendations
Even if Lucene had an “absolute” text relevance score, this score would only be able to prevent bad recommendations to some extent. We see a high potential in bibliometrics to support recommender systems in not recommending bad items. 4.3
Another question that may seem simple to answer is how many recommendations to
display? One? Two? ...Ten? We experimented in Mr. DLib with varying numbers of
recommendations between 1 and 15
In offline evaluations, it is common to tweak datasets. For instance, Caragea et al.
removed papers with fewer than ten and more than 100 citations from the evaluation
corpus, as well as papers citing fewer than 15 and more than 50 papers
We re-ranked content-based recommendation with readership data from Mendeley
There are a few potential reasons why our experiments delivered disappointing results (besides the possibility that the recommendation approaches are just not effective). For instance, in case of the bibliometric re-ranking, we assume that click-through rate might be not appropriate to measure the re-ranking effectiveness. However, even while there might be plausible reasons for the results, the fact remains that many of our experiments delivered rather disappointing results, but this is probably rather the rule than the exception in a real-world scenario. 8 The analysis is still in process, and the final results will be published soon. 5.3
One advantage of working on a real-world recommender system is the possibility to
release datasets, which then can be used (and cited) by other researchers. Some datasets
such as MovieLens are very popular in the recommender system community. The
MovieLens dataset was downloaded 140,000 times in 2014
Not all datasets become that popular. In 2014, we released a dataset of Docear's
recommender system
Building and operating real-world recommender systems is a time-consuming and challenging task, as is research on such systems.
To develop recommender systems, knowledge from various disciplines is required such as machine learning, server administration, databases, web technologies, and data formats. When building our own recommender systems, we could find no guidance in the literature. Most published research results had questionable evaluations and even if evaluations were sound, recommender systems seem to perform just too differently in different scenarios. Consequently, we used Apache Lucene as recommendation framework and began from scratch. Evaluating different recommendation approaches requires a randomization engine that selects and assembles recommendation algorithms automatically. In addition, a detailed logging mechanism is required that records which algorithm created which recommendation and how the user reaction was to the different algorithms. Another challenge lies in dealing with sub-optimal data quality from partners. Due to time constraints, we decided to not manually improve the data but just work with what we got.
Running a recommender system requires fast generation and delivery of recommendations. Otherwise, users become dissatisfied and click-through rates decrease. Although it is widely known that recommender systems should avoid making bad recommendations, this is not easily accomplished in practice, at least if Lucene is used as recommendation framework. Lucene has no absolute relevance score and hence no mechanism to recommend only items above a certain relevance threshold.
Research on real-world recommender systems is probably more frustrating than research in a lab-environment, mostly because data cannot be tweaked that easily. Consequently, we had to accept that many experiments with novel recommendation approaches failed. Similarly, while some recommendation datasets such as MovieLens are widely used in the community, we could not yet manage to establish our datasets as interesting source for other researchers.
Despite all these challenges, research on real-world recommender systems is a rewarding and worthwhile effort. We feel that working on real systems provides much more relevant research results. In addition, offering a real-world open-source project attracts many volunteers, students, and project partners. This, in turn, enabled us to conduct research in many areas, and be quite productive in terms of publication output. 7
This work was supported by a fellowship within the FITweltweit programme of the German Academic Exchange Service (DAAD). In addition, this publication has emanated from research conducted with the financial support of Science Foundation Ireland (SFI) under Grant Number 13/RC/2106. We are also grateful for the support received by Sophie Siebert, Stefan Feyer, Felix Beierle, Sara Mahmoud, Gabor Neusch, and Mr. DLib's partners.