We describe the experimental recommendation platform created in collaboration with the Social Science Research Network (SSRN). This system allows for researchers to test recommendation algorithm on SSRN's users and quickly collect feedback on the e cacy of their recommendations. We further describe a test run performed using EigenFactor recommends and compare its performance to SSRN's production recommender.
The Social Science Research Network (SSRN) is a world wide collaborative of over 272,100 authors and more than 1.7 million users that is devoted to the rapid worldwide dissemination of social science research. Like many publishers of this scale SSRN is contending with an rapid increase in the amount of scholarly literature currently being written. To help their users nd relevant articles SSRN has implemented a recommender system based on co-downloads, a collaborative ltering mechanism. Although the co-download system performs well, SSRN sought to further improve recommendations while also driving research in the area of bibliometrics. The authors, in collaboration with SSRN, built an experimental platform to test a novel, citation network based recommendation algorithm: EigenFactor Recommends.
Much of the research into scholarly article recommendation has su ered from
poor datasets and di culty testing with real users. As discussed in depth by Beel
et al[
In addition to improving access to online validation, the authors wanted to
validate their algorithm, EigenFactor Recommends. This algorithm is a novel
citation based recommender which exploits the hierarchical nature of academic
literature. Some of the earliest work on citation based recommenders for
scholarly literature was done in the late 1960s and early 1970s, notably bibliographic
coupling[
In this section we describe the experimental platform we constructed in collaboration with SSRN. This platform is, in the author's opinion, unique in that it allows for easy testing of di erent recommendation algorithms on live users of a very large publisher of academic literature. To fully understand this platform we will start by describing what a user sees when visiting SSRN, then discuss the appearance log and click log, and nally detail the experiment module.
Figure 2 shows a mock up of an article view on SSRN; this is the page a user is presented when viewing an article, such as http://ssrn.com/abstract=1636719. The current article's title, author list, publication date and abstract are shown in box 1. Box 2 contains various statistics about this article, while box 3 shows the recommendations for this article. The algorithm used to generate these recommendations is selected according to the weights de ned in the experiment module, and all recommendations listed in box 3 are generated by the same algorithm. Initially only up to three recommendations are shown, but if more are available clicking the more button will show any additional recommendations (box 4), up to ten total.
Whenever a user views an article several entries are generated in the appearance log, an example of which is shown in gure 2. Each entry in this log le corresponds to a single recommendation being shown on an article view page, including information about when the recommendation was generated, what algorithm generated it, what position this recommendation occupied in the \recommends" box, the article that was viewed (source) and the recommended article (target). A boolean ag is also present denoting if SSRN's fraud system considered this activity fraudulent. Note that since each entry in this log corresponds to a speci c recommendation a single article view could generate up to ten entries. Furthermore, recommendations \under the fold" only have an entry in the appearance log if a user actually viewed them { that is a user must have clicked the more button.
If a user clicks on a recommendation they are taken to that article's view. This will result in a new set of entries being generated in the appearance log, but also in the click log, an example of which is shown in gure 3. This log contains the same data as the appearance log, but also includes the ID of the user and a ag indicating if the le was downloaded or not. Note that for this experiment SSRN - Social Science Research Network Paper Title
Author #1
Author #2 Januaray 1st, 2015 1
Views Downloads Download Rank References
Recommended 1) Title by Author 2) Title by Author 3) Title by Author 4) Title by Author 5) Title by Author 6) Title by Author 7) Title by Author 8) Title by Author 9) Title by Author 10) Title by Author 2 3 4 More all user data was stripped from the dataset as it was not used. By correlating the data from the appearance log and the click log we can reconstruct user actions and through careful analysis determine the e cacy of various recommendation algorithms.
The nal part of this platform is the experiment module, an administrative tool that allows us to easily add recommendations to SSRN and con gure the amount of tra c each algorithm receives. For example, one could direct 85% of tra c to a production algorithm while splitting the remaining 15% of tra c between several experimental algorithms. The module also allows us to download the appearance log and click log, while providing some very basic analysis of the recommendation algorithm's performance. One current limitation of this system pertains to how recommendations are provided. Recommendations are provided in CSV le with a paper ID followed by a list of recommended paper IDs. This means recommendations are limited to item-to-item recommendations; we are unable to provide customized recommendations on a per-user basis. Fig. 2. Example of data included in the appearance log. Each entry in this log corresponds to a single recommendation being shown on an article view page (Figure 2 box 3). Each entry includes an ID which is unique to this log and not correlated with any other logs, a timestamp for the event, what algorithm generated this recommendation, what position the recommendation occupied in the recommendation list, the article that was viewed (source) and the recommended article (target). A boolean ag is also present denoting if SSRN's fraud system considered this activity fraudulent.
We ran an experiment on SSRN for a one week period, from January 27th 2014 through February 3rd, 2015 (inclusive). During the experiment a user could receive recommendations from one of four di erent algorithms: control, co-download, EigenFactor expert or EigenFactor serendipity. The experimental platform was con gured so that 85% of users were given recommendations from the co-download algorithm, while the remaining algorithms were each given 5% of tra c. The co-download algorithm is the default for SSRN when not running experiments, so it was given a larger portion of tra c to mitigate the risk of giving bad recommendations to users.
The experiment recorded a total of 1416404 recommendations, of which 239974 (16.94%) were considered fraudulent by SSRN's fraud detection system. These events are excluded from subsequent analysis, leaving 1176430 recommendations. A detailed breakdown of these numbers is available in table 1. The control algorithm consists of recommendations chosen at random from the SSRN corpus. As table 2 shows users view the abstract of these recommendations (clicks) at a slightly lower rate compared to either EigenFactor algorithm (0.65% vs 0.86-0.95%), and they download at a much lower rate of 1.72% vs 4-6% for real recommendations. It is possible that users nd the title's of the recommended papers interesting, but after reading the abstract realize they don't relate to the original paper. This behavior is inline with our expectations for a random control. The current production recommender used by SSRN is a collaborative ltering algorithm based on co-downloads. The algorithm works by tracking user downloads. To generate a recommendation for a paper the algorithm selects all users who have downloaded the source paper, then gathers a list of all the papers those users have downloaded. These papers are then counted and sorted by count, descending. The papers that have the most downloads are the top recommendations. Note that this algorithm is undirected, it doesn't know if paper 1 was downloaded then paper 2, only that 1 and 2 were both downloaded. The current implementation also limits the co-downloads to papers downloaded within the last two years, which helps to provide more recent recommendations. #!/usr/bin/env python from collections import Counter users = #set of users who downloaded paper i co_dl = Counter() for user in users: for paper in user.get_downloads()
co_dl[paper] += 1 return co_dl.most_common(3) 3.3
EigenFactor (EF) recommends is a variant of the EigenFactor algorithm
combined with the MapEquation algorithm [
EigenFactor recommends is di erent than the co-downloads approach in that
it uses the citation network, rather than usage data3. The recommendations are
based on the hierarchical structure of the SSRN corpus. The multi-level structure
is extracted using a variant of InfoMap [
Two variants of EigenFactor recommends were tested, expert and serendipity. Expert works be selecting the highest scoring articles in the most local cluster (i.e., the endleafs of the hierarchical tree). Serendipity also operates on the most local cluster, but instead selects a paper at random within this local clusters. Typically, these endleaf nodes consist of hundreds of papers.
To generate both variants of EigenFactor recommends, we rst constructed
the full citation network from SSRN. This network included 2,414,097
individual citations over 156,570 papers4. From this network, we produced 218,825
recommendations for both the expert and serendipity variants. These
recommendations were then uploaded into the experiment module and made available
to users on SSRN.
3 The method is also di erent, which is explained in the West et al. paper [
This experiment was the rst use of the SSRN recommendation system. As such, it was not only an experiment on collaborative ltering algorithms vs citation based algorithms, but also a test run of the system itself. Initial analysis uncovered several issues with both data collection and experimental design. 4.1
Although the metrics below are more thoroughly described in the Experimental Platform section, a brief summary is provided. There are three di erent metrics we capture: appearances, clicks and downloads. Appearances refers to a recommendation being shown to the user when they visit a page on SSRN. In gure 2 this is the box titled \Recommended" on the right hand side. Each recommendation counts as an appearance for that algorithm, so if three recommendations are shown that would count as three appearances for the algorithm that generated those recommendations. Recommendations for a given page view are all generated from the same algorithm, which is selected based on the weights provided in the experiment module. Clicks tracks when a user clicks on a recommendation. Doing so will take you to the abstract of the recommended paper. The nal metric we track is downloads, which is a measure of when a document is downloaded from a recommendation. Only downloads that are due to a recommendation are recorded in this metric. For each of these metrics counts represents the count of events while % is the percentage of that event type a speci c algorithm accounts for.
We also present three useful statistics for each algorithm: C/A, D/C and D/A. C/A is the number of clicks divided by the number of appearances. This is e ectively a click-through rate, and also could be considered the probability that a recommendation will be clicked. D/C is downloads over clicks, which is the percentage of clicks that lead to a download. The nal value is D/A, downloads over appearances, which can be thought of as the probability that a recommendation will be downloaded.
One issue we encountered was the number of recommendations shown for each algorithm. Due to an oversight when providing data to the experimentation platform, EigenFactor and the control algorithm only displayed a single recommendation while co-downloaded displayed several, potentially up to 10. This is apparent when one looks at the data that was captured for all positions and compares it to data only at position one (see table 3). Part of this is an overcounting problem, which can be recti ed, but a more serious issue arises that is intractable: since co-download gets to show three di erent recommendations it has a higher chance of a user clicking one (or more) of its recommendations. This gives co-download an advantage of 0.77% for C/A as shown by table 3, though it does result in a worse download rate (D/C). As table 4 shows the position of a recommendation has some impact on the click through and download rates. For recommendations this is not unexpected: if recommendations are presented in order of strength and the algorithm is e ective one would expect higher ranked recommendations to be selected more often. Furthermore, recommendations below the \cuto " on the article view page (anything with position greater than three) are clicked and downloaded at a much lower rate, as we would also expect. There is, however, an interesting question of primacy e ect: does a recommendation being listed rst increase the rate at which it is downloaded, independent of the recommendation quality? Table 4 provides some evidence that this primacy e ect is occurring. The clicks % shows a large discrepancy in the number of clicks given to item one vs two and three. Furthermore, positions two and three show a substantially higher download rate, which implies more interest in items in position two and three when they are clicked. 4.4
Table 5 shows a list of the most clicked recommendations by algorithm. Although no in-depth analysis about these titles has been performed, it is clear that nance related articles are very heavily represented. During this experiment several issues with experimental design and the data collection platform were discovered. Although unfortunate, this is unsurprising. Though data quality issues mean few strong conclusions can be drawn, we did see evidence that the co-download algorithm results in a signi cantly higher click through rate (3.94%) over either of the EigenFactor algorithms (0.95% and 0.86%). It is, however, unclear why co-download performed nearly three times better, and this will be an area for future investigation.
However, once a user is viewing an abstract co-download has a slightly lower download rate compared to the EigenFactor algorithm (4.55% vs 5.77%). It is very interesting that the EigenFactor serendipity algorithm has the highest download rate, though given the small number of downloads overall this could just be noise in the dataset. One could view this as EigenFactor having higher deviation in the recommendations it generates. It may recommend articles with interesting titles less often, but when it does that article is downloaded at a higher rate.
Finally, evidence of a primacy e ect was found in the co-download data set. The next experiment will seek to validate this with additional data in the control set.
We are already planning a larger scale experiment to validate these ndings, running the algorithms for at least a month. We will also be xing all issues that were discovered in this process and more aggressively validating the data collection process.