In recent years the number of academic publication increased strongly. As this information ood grows, it becomes more di cult for researchers to nd relevant literature e ectively. To overcome this difculty, recommendation systems can be used which often utilize text similarity to nd related documents. To improve those systems we add scientometrics as a ranking measure for popularity into these algorithms. In this paper we analyse whether and how scientometrics are useful in a recommender system.
The number of academic publications doubles approximately every ten years [
To handle this information ood, recommender systems come into account.
They identify the informational needs of researchers and recommend the best
tting literature. Unfortunately, neither the automatic identi cation of the
informational needs, nor the search for relevant literature are trivial tasks. The
extent of this challenge can be derived from the amount of research in this area.
In the last sixteen years 90 methods were developed and investigated by around
300 academic researches and published in over 200 publications [
Since it is important to recommend papers which are relevant, it is necessary to improve recommender systems. In this paper we focus on the use of scientometrics to rank the recommendations.
Scientometrics are introduced as a 'quantitative study of science and
technology' [
On the contrary, altmetrics, which takes its name from alternative metrics,
use information from social media, blogs et cetera [
Until now recommendation systems rank their recommendation only with respect to content similarity. The assumption in this paper is that a paper with a good reputation is more worth reading, thus should be recommended. To measure the reputation we will use scientometrics. The scienti c question is which scientometrics and their combination with the similarity ranking is most liked by the user and thus the best. To measure how much users like a recommendation the Click-Through-Rate (CTR) will be analyzed. 2 2.1
Papers can be ranked by di erent criteria. Lewandowski and Behnert [
The text statistics can describe how similar two documents are based on the
content. A famous measure is, for example, TF-IDF. Text statistics also can
focus on the document length or anchor text and emphasized text. The popularity
pays heed to the usage of a document, how often is it read, cited, downloaded
or bought, and how highly are they rated. The freshness ensures that one gets
recent documents. The locality and availability considers, if the user currently
has access to the document, since he would not want to get recommendations
he can not read. It also considers costs, since free documents are easier to
access. Also one can use content properties to rank papers, which includes the le
format, the language and the amount of meta data. People like to read their
own language and use common le formats like PDF. The last category is the
user's background, where the user is categorized, for example, by his academic
background or his eld of study. The rationale behind this is that a computer
scientist most likely will not read a document related to history [
In this paper we focus on the popularity category to rank papers. We use scientometrics which we assume indicate the popularity and rank the paper accordingly. 2.2
Scientometrics are used to calculate the reputation of journals, authors,
institutions or papers. After that people can for example identify authors with high
or low reputation. Scientometrics can help to identify patterns which are highly
cited [
It is also common to rank results in search engines based on their popularity.
This concept is for example used in Google's Page Rank Algorithm, where the
popularity is expressed by hyperlinks [
Cooperationpartner GESIS requests recommendations for a document send recommendationset
MrDlib data corpus of different list of relevant reranking list of shown 9.5 million documents recommendocuments algorithms (10-100) algorithms dations (6) focus of the paper
Mr. DLib (Machine Readable Digital Library, http://mr-dlib.org/) is a
researchpaper recommender system and for academic purposes only [
CT R =
Number of clicked recommendations
Number of delivered recommendations We gathered data from 17th October 2016 to 8th February 2017. Currently we display six recommendations for each recommendation request. In total we analysed 38,740,893 recommendations with 53,441 clicks, which corresponds to a CTR of 0.138%. 3.2
In this section we present our test results of the re-ranking approaches. They are split with respect to the three di erent attributes. All data is available at http://datasets.mr-dlib.org/. The re-ranking method describes how the scientometric data and the text relevance score are combined. We ranked the recommendations considering 'text relevance (TR) only' or 'scientometrics only' combined with di erent weightings. While the text relevance calculated by Solr is within a certain range, the scientometric values can range from 0 to several 1000. To balance the impact of the text relevance and the scientometrics, root and logarithmic transformation are applied on the scientometrics. Each combination was sorted ascending and descending to ensure there is a di erence in sorting them according to the metrics.
As seen in gure 2, the ascending ordered scores are always lower than the descending. This was expected since we assume that scientometrics are a measure for popularity and a popular publication has higher quality or more interesting ndings thus is more worth reading. Also the ranking with the scientometrics scores higher than the text relevance only approach. An unclear occasion is the good performance of the ascending ordered 'scientometric only'. Its performance was nearly identical to the descending ordered 'text relevance only'.
The statistical signi cance is given for the associated descending and ascending orders - except for the 'scientometric only'. There is no statistically signi cant di erence between the descending orders. By metrics, we refer to the scientometric indicator that we used for ranking recommendations. We analysed absolute readership count Rcount, readership count normalized by the age of the paper in years Rage and readership count normalized by the number of authors Rauth. The formula for readership normalized by the age of the paper Rage is
Rage =
Y earnow
This is expected to show better performance, since good papers need time to become famous.
The formula for readership normalized by the number of authors Rauth is Rauth =
#authors
This normalization is expected to show also better performance. Paper with many authors are likely to be more famous because they are wider spread.
We considered in our experiments primarily those metrics which were easy to calculate while incorporating as many as possible di erent aspects of the scientometrics in the literature review. Another important criteria was that the metrics would only use the data that we had. By evaluating this subset of metrics, we will have an idea of which groups of metrics perform best in a real-world setting. With this information we can focus on this group in the future.
Surprisingly the absolute readership count Rcount scores highest. These results are statistically signi cant.
The discrepancy of the CTR to the re-ranking method results from not including 'text relevance only' data. The re-ranked candidates are the number of candidates which will be picked from an algorithm to re-rank them according to the ranking parameters. The more candidates are considered for re-ranking, the less the text relevance is considered. It seems that more candidates leads to a better performance although a medium sized list decreased it. The statistical signi cance is given for the associated descending and ascending orders except for 30 and 100 candidates. Between the di erent descending orders roughly half of the combinations are statistically signi cant.
In this paper we evaluated di erent re-ranking approaches to see whether and how scientometrics can improve academic paper recommendations systems. With our current data we can conclude that scientometrics do improve the ranking of documents in a recommendation system compared to a text relevance only approach. This is shown in gure 2, where the CTR of the descending rankings that include scientometrics score higher than the 'text relevance only' approach. However, this improvement is rather small. Furthermore, a smaller list of reranking candidates seem to lead to a better CTR. The metric which achieved the best score is the absolute count without normalization.
However, the good scoring of the 'scientometrics only' with ascending ordering was not expected. This might be due to the low scientometric data coverage of 17.82%. If too many documents of the pre-generated list do not have associated readership data the re-rank will not e ect the sorting. Thus, descending and ascending orders will have the same sorting and same performance.
To achieve a better coverage in the future we will calculate author metrics and apply them back to the papers by building a sum or the average. This will lead to a coverage of 46.27%. Furthermore, a fall-back mechanism could be implemented, which will choose a higher coverage metric if the current metric has to less data.
Another reason for the small improvement of the scientometric-including rankings might be the style of the evaluation. When the recommendations are displayed only title and authors are shown. The user decides only based on this information if the recommendation might be useful. However, our approach takes popularity into account, which is an assumption for quality. To enhance the evaluation we should log if the user used the document after taking a look at it e.g. if he clicked several links like cite, export, favourite or search. This measure might be more suitable for evaluating a popularity approach.
The next step is gathering and calculation of citation metrics. We will soon
be able to evaluate the scientometrics in JabRef and collect more data [
This work was supported by a fellowship within the FITweltweit programme of the German Academic Exchange Service (DAAD). Moreover we want to thank Joeran Beel, Martin Glauer and Christoph Doell for the support.