We investigate the problem of choice overload { the di culty of making a decision when faced with many options { when displaying related-article recommendations in digital libraries. So far, research regarding to how many items should be displayed has mostly been done in the elds of media recommendations and search engines. We analyze the number of recommendations in current digital libraries. When browsing fullscreen with a laptop or desktop PC, all display a xed number of recommendations. 72% display three, four, or ve recommendations, none display more than ten. We provide results from an empirical evaluation conducted with GESIS ' digital library Sowiport, with recommendations delivered by recommendations-as-a-service provider Mr. DLib. We use click-through rate as a measure of recommendation e ectiveness based on 3.4 million delivered recommendations. Our results show lower click-through rates for higher numbers of recommendations and twice as many clicked recommendations when displaying ten instead of one related-articles. Our results indicate that users might quickly feel overloaded by choice.
More and more information is available online for academic researchers in
digital libraries [
Developing Mr. DLib (Machine-readable Digital Library)4 [
There have been several studies investigating choice overload with respect to
consumer goods (for example [
Other related studies looked into the number of search results to be
displayed. Jones et al. conclude that the screen size is a determining factor with
respect to how many search result items users interact with [
In our work, we focus on a di erent domain. We will investigate the problem
of choice overload in digital libraries { in contrast to movie recommendations
7 All data relating to this paper is available on http://datasets.mr-dlib.org,
including a table of the delivered and clicked recommendations, the information about
the investigated digital libraries, and the gures presented in this paper.
and search results. The question of how many recommendations to display and
choice overload has not been studied in the domain of digital libraries, to the best
of our knowledge. In a recent literature survey on more than 200 articles about
research paper recommender systems, no one discussed or researched this topic
[
In order to investigate choice overload in digital libraries, we rst examine how many items other recommender systems in digital libraries display. We investigated 63 digital libraries8 and reference managers with search interfaces9. We considered recommendations that are being displayed when an item from the search results is selected. In a few cases, the number of displayed recommendations of related items was dependent on the size of the browser window. For the numbers given in the following, we assumed a full screen browser window on a laptop computer (13" display with 1280x800 resolution).
In a second step, we conducted an experimental evaluation to investigate how di erent numbers of recommendations a ect the click rates on related-article recommendations. Click-through rates are a good way to study the users' actual behavior when displaying recommended items in real situations. We analyze data from 3.4 million recommendations. The data was obtained from users of the academic search engine Sowiport10, which is run by GESIS - Leibniz-Institute for the Social Sciences 11, which is the largest infrastructure institution for the Social Sciences in Germany. Sowiport contains about 9.6 million literature references and 50,000 research projects from 18 di erent databases, mostly relating to the social and political sciences. Literature references usually cover keywords, classi cations, author(s), and journal or conference information, and if available: citations, references, and links to full texts.
Sowiport co-operates with Mr. DLib, an open Web Service to provide
scholarly literature-recommendations-as-a-service (Figure 2). This means that all
computations relating to the recommendations run on Mr. DLibs servers, while
the presentation takes place on Sowiports website. Our recommender system
shows related-article recommendations on each articles detail page in Sowiport
(see Figure 1). Whenever such a detail page is requested by a user, the
recommender system randomly chooses one of four recommendation approaches
to generate recommendations: 1. stereotype recommendations, 2. most
popular recommendations, 3. content-based ltering (CBF), and 4. random
recommendations. We measured the e ectiveness of the recommendation approaches
8 Most of them listed on https://en.wikipedia.org/wiki/List_of_digital_
library_projects
9 See http://datasets.mr-dlib.org for detailed results.
10 Some explanations about Sowiport and Mr. DLib are from [
Our expectation is that at rst, by increasing the number of displayed recommendations, there will be an increase in the CTR. By displaying more and more recommendations, we expect to reach a maximum in CTR at some point. After that, when displaying more recommendations, we expect the CTR to drop, indicating choice overload. Similarly, for the clicks, we would expect to nd a maximum at a certain number of displayed recommendations. Plotting the CTR and the average clicks we show this expectation in Figure 3. Here, the CTR, the orange line, increases with the number of displayed recommendations, reaches a maximum at four, and decreases afterwards, indicating choice overload. The clicks, the gray line, reach a maximum at ve displayed recommendations. In that case, we would decide for four (maximum CTR) or ve (maximum clicks) recommendations.
Alternatively, we would have expected results as in Figure 4. Here, the CTR declines with increasing number of displayed recommendations. The absolute clicks have a maximum at four. The question then would be: What is better { a higher CTR or the maximum of absolute clicks? Depending on the answer, we would choose a recommendation set size of between one and four. As we will show in the following section, the results are quite di erent from our expectations. 1.00% 0.90% 0.80% 0.60% 0.70% 0.50% 0.60% TR0.50% C0.40% 0.30% 0.20% 0.10% 5 0.00% 1
Number of Recommendations in Existing Digital Libraries 19 (30%) of the 63 digital libraries displayed recommendations for related items. Figure 5 shows the distribution of recommendations for those 19 libraries. Most libraries (72% of those that display recommendations) display three, four, or ve recommendations, none is displaying more than 10 or less than three. It is also notable that they all always show a xed number of related-articles, one could also imagine displaying varying numbers. Looking for related items in the database, there can be varying scores of relevance. One way of displaying a varying amount of related-article recommendations would be to take into account such relevance scores, e.g., giving 10 recommendations if there are 10 highly 28%
28% 16% 4%
4% 1 2
3 4 5 6 7 8 Number of Related-Article Recommenations 8% 9 12% 10 relevant related-articles, and giving only two recommendations, if there are only two related-articles that have a relevance score above a certain threshold.
We do not know how the numbers of related articles in the reviewed recommender systems were chosen by the operators. We assume that they either just arbitrarily chose the numbers, or did some experiments but did not publish the results. In the following, by measuring CTRs, we want to investigate and measure the e ect the number of displayed recommendations has and if the CTR indicates choice overload for certain numbers of displayed recommendations. 4.2
Experiment with Varying Number of Recommendations The solid orange line in Figure 6 shows the CTR by the number of displayed recommendations. The higher the number of recommendations, the lower the overall CTR is. The dashed line shows the average absolute number of clicked recommendations (per 1,000 recommendations). The bigger the recommendation set size is, the higher the number of absolute clicks is.
When only one recommendation was displayed, the CTR was 0.84% on average. This means, when our recommender system delivered 1,000 times one recommendation each, 8.4 recommendations were clicked. For two displayed recommendations, the CTR was only 0.49% on average. This means, when our recommender system delivered 1,000 times two recommendations each (2,000 in total), 9.8 recommendations were clicked (half clicked on the rst one, half on the second one). Overall, if one or two recommendations are displayed, does not make a big di erence in the absolute number of clicks, it only increases by 17% (8.4 to 9.8). When fteen recommendations were shown, the CTR was at the minimum of 0.14% on average, while the absolute number of clicks was at the maximum of 21.4.
Comparing these results with our expectations given in Section 3, we can see that the CTR unexpectetly decreases rapidly and has a clear maximum at 0.90% 0.80% 0.70% one displayed recommendation. Furthermore, contrary to our expectations, the absolute clicks keep increasing instead of having a maximum value at a few recommendations. The results show an under-proportional increase in average clicks on the displayed recommendations. Displaying twice as many recommendations does not double the clicks. In order for the absolute number of clicked recommendations (per 1,000) to double from 8.4 (for one displayed recommendation) to 17, the number of displayed related-articles has to be raised to 10 or 11. When 15 recommendations are displayed, only 2.5 as many recommendations are clicked compared to displaying a single recommendation. Regarding choice overload this implies that having more recommendations to choose from does, in general, only create a small incentive for the user to click on more of the displayed recommended items. There are some points to consider when interpreting these results. Many documents in Sowiport only have sparse information, thus they might not be interesting for the users. Another possible option why the experimental results are so unclear about the number of recommendations to give is that the relevance of the recommendations might have been too low, so that many users did not click further recommendations after clicking the rst one. In that case, the research should be repeated when we are able to deliver better recommendations. The session-length is another aspect to consider. For instance, if one user visits two pages and on gets 15 recommendations on each page, we assume the CTR will be higher than for a user who looks at 10 pages and gets 15 recommendations on each page. An additional aspect of sessions is that so far, we did not lter for recommendations that have already been shown to a user. So, if a user looks at 15 detail pages and gets 10 related-article recommendations on each, most likely there will be duplicate recommendations, and hence the CTR decreases the more recommendations are shown. Another aspect to consider in the results' interpretation is how users might use Sowiport. If the user clicks on a recommendation, a new tab is opened. If the recommendation was good, she might forget about the other open tab { especially if there are further good recommendations shown in the new tab. 5
The average clicks on displayed recommendations under-proportionally increase with the number of displayed items. In order for the clicks to double, the size of the recommendation set has to be increased from one to 10 or 11. These numbers might imply that the users quickly feel overloaded by choice. The results di er from our expectations. Just based on these numbers, we could conclude that we should only display one recommendation because the CTR is highest { or to keep displaying even more than 15 recommendations until the number of absolute clicks does not increase anymore. Further research will be necessary to determine a good number of recommended items to display.
Our results are based on Sowiport. Further research is necessary to con rm if our ndings also apply to other digital libraries. We therefore plan to repeat our research, for instance, with JabRef and the library of the Technical University of Munich. Future work could also include using other evaluation methods and metrics than CTR (e.g., a user study, or user ratings, or tracking which recommended item were actually exported or saved) or making a survey and asking the operators of the other digital libraries how they decided the number of displayed recommendations. Furthermore, a question to be discussed is what recommender systems in digital libraries should try to achieve, e.g., maximizing CTR, maximizing the number of clicked recommendations, etc.
Acknowledgments. This work has received funding from project DYNAMIC12 (grant No 01IS12056), which is funded as part of the Software Campus initiative by the German Federal Ministry of Education and Research (BMBF). This work was also supported by a fellowship within the FITweltweit programme of the German Academic Exchange Service (DAAD). This publication also has emanated from research conducted with the nancial support of Science Foundation Ireland (SFI) under Grant Number 13/RC/2106. We are further grateful for the support provided by Sophie Siebert. 12 http://www.dynamic-project.de