Mendeley Suggest is a popular academic paper recommender, serving over 1.5M researchers in 2018. We attempt to assess the extent Mendeley Suggest helps its users in their research in two areas: helping researchers keep up with the most prominent development in the field and help researchers find relevant literature. Our findings indicate that the recommender significantly increases the chance that a user finds important research and decreases the amount of time she needs to spend on searching. We observe that the efect is much greater than the number of accepted recommendations and propose that it is due to an increase in reading activity that Mendeley Suggest recommendations spur. Time-series analyses are presented to back up this hypothesis. Our results highlight the potential of academic paper recommenders in furthering science.
• Information systems → Collaborative filtering ; Digital libraries and archives; • Applied computing → Digital libraries and archives.
2 2.1
Several research papers have investigated proxies for citations
garnered by published articles, such as the work of Haustein et al.
[
Additionally, there is also substantial existing literature studying
the efects of recommender systems, both analytically and for in-use
cases. For example, Fleder et al. [
3https://www.mendeley.com
4https://www.elsevier.com/__data/assets/pdf_file/0011/117992/Mendeley-Manualfor-Librarians_2017.pdf
5https://en.wikipedia.org/wiki/CiteULike
that the use of a recommender system enhances the consumers’
satisfaction with the website and provides a more efective product
search process. Zhou et al. [
Mendeley is a free reference manager and an academic social network where users can manage their interests by creating a personal repository, called library, of articles which they find useful. Mendeley also provides a reader equipped with highlight and annotation functionalities on desktop, web, and mobile.
All Mendeley users automatically have access to Mendeley
Suggest (MS), an article recommender that uses collaborative and
content-based approaches [
The MS recommender comprises diferent types of recommenders, which tackle the various disciplines and levels of seniority of researchers who use Mendeley. The primary recommender is based on a collaborative filter which makes use of similarities between users’ libraries, i.e. predicting whether a user is interested in a paper based on whether similar users have the document in their libraries. One of the drawbacks of a collaborative filtering approach is its susceptibility to the cold-start problem, wherein newly added articles cannot be immediately recommended and new users cannot be served recommendations. To circumvent this problem, Suggest also has a content-based recommender, based on ElasticSearch morelike-this queries, and weighted by the popularity of articles.6
In addition to the recommenders, MS also applies dithering and
impression discounting [
We attempt to quantify the value MS brings to its users along two dimensions: coverage and time. If we know the set D = {(p, u)} of all the papers {p} that each researcher {u } ought to read, we could measure how much of them she covers at a certain point in time, both through recommendations and other means, and we would hope that MS users reach higher coverage in a shorter amount of time compared to non-users. Although this ideal cannot be attained, we will later propose relaxations that capture some perspectives of the set.
3.1
Before presenting our experimental design, we will introduce a few terms used in this paper that requires specification beyond what is given by common sense.
MS works by giving users recommendations on papers to read, presented, for example, as a list on a web page or on a tab integrated in the Mendeley mobile application. We track anonymized interactions with MS via, among others, two types of events: recommendation viewing and recommended addition. Viewing a recommendation entails a user clicking on a link in the recommendation list, upon which a document page will be opened. At this point, the document is not added to the user’s library yet. The user can actively do so by clicking on a button that says “Add to library”. She can also add the same paper through other means (e.g. importing a PDF or pasting a bibtex entry) which are not captured as a recommended addition.
The routine of a user includes collecting documents to build up her library. We will refer to this activity as additions, which can include articles recommended by MS. For all papers, we have the timestamp of the last time they are added to a user’s library. We also track events related to annotations performed on documents in users’ library. When a line is highlighted or a note edited, we record the timestamp and action type for analytic purposes.
Throughout the paper, we assume the same notion of articles in
MS, Mendeley libraries, and the literature as represented by Scopus.
Similarly, in the scope of this paper, citations are treated as a given.
Behind the scene, they are extracted via the machinery internal to
Scopus [
A common technique to measure the performance of recommender systems is A/B test. A control group A and an experimental group B are typically served two versions of a system that difer in a single feature. Although highly efective in measuring short-term direct efect, sustaining a long A/B test is often dificult in a commercial setting because of its negative efect on customer experience. More importantly, the approach is only suited to study versions of a recommender but not the very efect of using it because we cannot, in normal circumstances, bar users from using the product to create a control group.
As an alternative, we study groups of users difering in Mendeley Suggest usage. By measuring at a user-group level during an extended period of time, we can capture both direct and indirect efects of our recommender system.
Measured by the number of recommendation views between January 2018 and July 2019, the distribution of Mendeley users resembles a Zipfian curve, with most users opening less than one article per week. To study the efect of diferent degrees of usage, we divide this population into four chunks:
S-heavy Users who clicked on the most recommendations, belonging to the top 5%, S-frequent Users who are less active than the first group but belong to the top 25%. This group of users viewed more than 2.5 recommendations per week during the period we observed. S-infrequent The remaining users who clicked on at least one recommendation, and S-non-user Mendeley users who did not open any recommendation. To reduce computational complexity, we extract a random sample of 400,000 users.
We only include in S-non-users people who added at least one article to their Mendeley library since 2018. There can be various reasons an active user of Mendeley does not use Mendeley Suggest. Since the platform is most known for its reading and reference managing functionalities, a user might simply never encounter Mendeley Suggest. She might also have decided not to use it in the past. We leave an in-depth examination of the non-user group for future work.
Figure 1 shows the relative library size of user types, normalized to that of non-users. It can be observed that higher MS usage coincides with higher Mendeley usage overall, except between infrequent Suggest users and non-users. This is a factor afecting coverage that we will comment on later. 3.3
As the first relaxation of the ideal paper assignment set D, we propose to study the set D1 of recent and most-cited articles in the literature. Arguably, it is important for a researcher to be aware of the latest major development in her field, regardless of whether she is going to use it directly in her research.
To construct D1, we sort articles published in 2018 onward according to the number of times they are cited. For each field as codified by Scopus’s All Science Journal Classification Codes (ASJC) 7, an excerpt of which can be found in Table 1, we extract the 100 most cited articles that is unambiguously in the field (i.e., being assigned to only one ASJC code). A sample of the papers we extracted can be seen in Table 2. We do not possess an up-to-date mapping from researchers to their field of research, therefore, we treat articles from every field equally. Given the contrast between the broad scope of ASJC codes and the narrow specialization of researchers, we do not expect a researcher to have read many of the extracted articles. We choose not to calculate Peason correlation because counting the number of articles might mistake broad-mindedness (or a lack of focus) for the coverage of useful literature.
The extent that a group of users capture the latest literature is therefore defined as the proportion of its members who added to
Oncology
coverage1 = |{user who added at least one extracted article}| |{all users}|
We checked that papers from D1 are reachable by MS, with the number of articles recommended to at least one user spreads relatively evenly across fields between 1 and 100 (mean=55, stddev=29). 3.4
In the second perspective, we attempt to measure the efectiveness of MS in helping users find papers that they might want to cite later on. To evaluate this, we construct the set D2 = {(u, p)} of papers {p} that Mendeley users {u } cited between January 2018 and July 2019. This information is available to us via a feature in Mendeley that allows users to claim their Scopus profile. Publications of an author and the out-going citations were automatically extracted and can be readily queried via Scopus.
The coverage of citable articles for a group of users is proportional to the number of papers p they added to their Mendeley libraries before the publication of any of their articles citing p: coverage2 = |{pairs of ⟨user, added paper that is later cited⟩}| |{pairs of ⟨user, cited paper⟩}|
Because users can combine Mendeley with other means of reference management and distribute references across co-authors, we do not expect the coverage to reach 100%. Ideally, we would like to measure literature added to Mendeley library before the submission of a paper but this data is not available to us. The delay between submission and publication might artificially increase coverage. However, we expect it to be the same across groups of users. 4
In this section, we will present the results of experiments outlined in the previous section and their implications. 4.1
Figure 2 shows the adoption curves of diferent groups of users w.r.t. our set of most cited recent papers. It is clear that the more a researcher uses MS, the more likely she finds the latest important paper.
The diference cannot be explained by the level of activity alone. Although S-heavy users added only twice as many articles into their Mendeley libraries compared to S-non-user (see Figure 1), they reached a coverage of 0.3843 compared to 0.0057 of S-nonuser in July 2019 (68 times higher). Moreover, S-infrequent users who added to their library 50% less articles than S-non-user (see Figure 1) still got 23 times higher chance of staying up-to-date with
Analytical and numerical solution of non-Newtonian second-grade fluid flow on a stretching sheet
The diagnosis and management of nonalcoholic fatty liver disease: Practice guidance from the American Association for the Study of Liver Diseases #cit. 26 the most important research (comparing a coverage of 0.1274 with 0.0028 in July 2019).
This result demonstrates the benefit of using MS, even in a nonfrequent basis. 4.2
The results in the previous sections are surprising when we consider the relatively small amount of viewed recommendations per week (see Section 3.2). Upon conducting a quick analysis, we found that, for S-infrequent Suggest users, the number of all additions to Mendeley library is 172 times that of additions recommended by Suggest. In the case of S-frequent and S-heavy, there are, respectively, 46 and 24 library additions for each recommendation by Suggest.
We hypothesize that the indirect efect of MS is much bigger than the direct one. In one scenario, upon reading a relevant paper, a researcher might follow forward and backward citations to gain a more exhaustive understanding of her field. Alternatively, a researcher might discover a new topic by serendipity, broadening her coverage. If this is the case, we expect an increase in additions to library when people use MS.
To validate this hypothesis, we study the usage pattern of Sinfrequent users in the first quarter of 2019. As mentioned in Section 3.1, we have records of anonymized addition events in Mendeley. For the analysis, they are divided into two categories: those that occur on the same day as a recommendation viewing event and those do not. The results can be seen in Figure 4. In line with our prediction, days that people use MS see 1.55 times more articles added to their library. infrequent users in Q1 2019 in two scenarios: when they use and do not use MS. Numbers of additions are normalized such that the average activity without using MS is 100%. Q1 2019 in two scenarios: using and not using MS. Numbers of events are normalized such that the mean activity level without using MS is 100%.
We also check if MS usage coincides with deeper reading by looking at annotation events. Following the same procedure, our analysis shows that, on average, people annotate much more around the time they use the recommender. Although Suggest is observed together with increased annotating in only 42 days as opposed to the 47 days that it sees less activity, the peaks are much higher than the depth of the troughs (Figure 5). We repeated the experiments with S-frequent and S-heavy and obtained similar results although the efect is less pronounced: articles added together with MS usage are 1.14 and 1.13 times as many as without. 5
In the current research, we study the impact of Mendeley Suggest on scientific researchers. Through various analyses, we showed that MS increases the chance that a researcher finds important and relevant literature, in a more timely manner. We propose a mechanism to explain this efect in which a researcher does not stop at adding a recommended article to her library but read the content in depth and explore further to deepen and broaden her grasp of the literature. Evidences from Mendeley usage log are presented to support our hypothesis.
The results of our research highlight the positive efect a scientific article recommender can have on researchers’ professional lives. Considering that MS is composed of standard techniques such as nearest-neighbor collaborative filtering and ElasticSearch-based content recommendations, without a reranking step, there is much room for improvement.
A limit of the current research is its observational nature. There are alternative explanations that, given the limited resources the authors possess, we could not eliminate. For example, the correlation between MS usage and reading activities might be because users tend to open recommendations when they have more time to read. Further research is needed to disentangle factors and reach a clearer picture of the recommender’s impact.