For the past few years, we used Apache Lucene as recommendation framework in our scholarly-literature recommender system of the reference-management software Docear. In this paper, we share three lessons learned from our work with Lucene. First, recommendations with relevance scores below 0.025 tend to have significantly lower click-through rates than recommendations with relevance scores above 0.025. Second, by picking ten recommendations randomly from Lucene's top50 search results, click-through rate decreased by 15%, compared to recommending the top10 results. Third, the number of returned search results tend to predict how high click-through rates will be: when Lucene returns less than 1,000 search results, click-through rates tend to be around half as high as if 1,000+ results are returned.
Mendeley, or Citavi. Docear has approximately 50,000 registered users and uses
mindmaps to manage PDFs and references. Since 2012, Docear has been offering a
recommender system for 1.8 million publicly available research papers on the web.
Recommendations are displayed as a list of ten research papers, showing the title of the
recommended papers (Fig. 1). Clicking a recommendation opens the paper’s full-text
(PDF) in the user’s web browser. Between 2012 and 2015, the recommender system
delivered around one million recommendations to more than 7,000 researchers. For
more details on the recommender system please refer to
All presented results are based on data that we collected between May 2013, and
October 2014. During this time, Docear’s recommender system delivered 418,308
recommendations to 4,674 unique users. We use click-through rate as measure for the
effectiveness of delivered recommendations. Click-through rate (CTR) describes the ratio of
clicked and delivered recommendations. For more details on click-through rate and its
suitability as evaluation metric please refer to
Lucene’s Relevance Scores Lucene provides relevance scores for each recommendation. This information could be used, theoretically, to recommend only documents with a relevance score above a certain threshold. However, on the Web it is often reported that these scores cannot be used to compare relevancies of recommendations between different queries, or to conclude from the relevance score how relevant the search result or recommendation is overall.1 Our data shows a slightly different picture.
In our data, the highest relevance score for a recommendation was 19.01, median was 0.16 and mean was 0.22. Fig. 2 shows that CTR was lowest (3.36%) for recommendations with a relevance score below 0.01, and highest (6.16%) for relevance scores of 1 and above. For recommendations with relevance scores between 0.1 and 0.8, CTR remained mostly stable around 5%. Overall, there is a notable trend: CTR increases, the higher Lucene relevance scores become.
Our observation contradicts the common claims that Lucene’s relevance score cannot be used to estimate a search result’s absolute relevance. If, for instance, an operator of a recommender system decided that a click-through rate of at least 4% was desirable, then recommendations with a relevance score below 0.25 should probably be discarded. 1 https://wiki.apache.org/lucene-java/ScoresAsPercentages 8% 7% 6% 5% 4% TR 3% C 2% 1% 0% Similarly, our result might lead to the conclusion to only recommend documents above a certain relevance threshold, e.g. 1. However, recommending only documents with a relevance score of 1 and above is probably not sensible as only a small fraction of recommendations had a relevance score of 1 and above (0.60%). Similarly, it might seem sensible to not recommend documents with relevance scores below 0.025 as these documents had very low CTRs. However, only a small fraction of recommendations (4.27%) had relevance scores below 0.025, so this decision would barely affect the overall click-through rate.
25% 20% 15% 10% 5% n o it u b iitrs D To increase the diversity of recommendations, Docear’s recommender system randomly chose 10 recommendations out of the top50 results returned by Lucene. However, this leads to lower click-through rates. Recommendations originally being ranked 1 by Lucene received CTRs of 6.83% on average and recommendations on rank 2 received CTRs of 6.08% on average (Fig. 3). For ranks 3 to 10, CTR remains stable around 5.3% and then CTR constantly decreases the lower the original rank. ed s lisyapd itanndo fro em eubNm rceom 25% 20% 15% 10% 5% 0%
1 # Dspld. Recs 2.10% CTR 6.83%
2 2.13% 6.08%
3 2.07% 5.29%
4 2.14% 5.41%
5 [6;10] [11;20] [21;30] [31;40] [41;50] 2.10% 10.61% 20.40% 19.64% 19.46% 19.35% 5.25% 5.33% 4.72% 4.69% 4.46% 4.17%
Original Lucene Rank
ed s
ilsyapd itandno
fro em
eubNm rceom
Overall, recommendations being in Lucene’s top10 results, achieved CTRs of 5.55%
on average, while the top50 achieved CTRs of 4.73% on average. This means, selecting
randomly 10 recommendations from the top50 candidates decreases recommendation
effectiveness by around 15%, compared to showing recommendations from the top10
only. The recommender system shuffled recommendations before they were displayed.
This means, position bias cannot have influenced the results
# Dspld. Recs (Terms) # Dspld. Recs (Cit.) CTR (Terms) CTR (Citations) vance score, discarding them will probably not notably affect the overall recommendation effectiveness. Second, recommending ten recommendations out of the top50 results might be sensible. Although this process decreases the overall recommendation effectiveness by 15%, the recommendation diversity or number of total recommendations is increased. Third, the number of recommendation candidates returned by Lucene is suitable to approximate the recommendation effectiveness. If Lucene returns less than 1,000 results for term-based recommendations, the click-through rate probably will be around half as high as if 1,000 candidates are returned. In the case of less than 1,000 results, it might make sense to not display the recommendations or generate recommendations again with another recommendation approach.
For the future, we suggest to repeat our analyses in different scenarios, for instance,
with news recommenders or other literature recommender systems, to see if Lucene
behaves in the same way as in the scenario of Docear. Currently, we are developing a
recommender system as-a-service that would allow us to conduct such analyses with
different partners
This work was supported by a fellowship within the FITweltweit programme of the German Academic Exchange Service (DAAD). In addition, this publication has emanated from research conducted with the financial support of Science Foundation Ireland (SFI) under Grant Number 13/RC/2106.