Usage and citation metrics are indicators of interest in documents by users in information retrieval (IR) systems. Our aim is to create an impact relevance variable for ranking functions in legal IR systems. In this paper, we study the development of user clicks and citation counts over time for documents in a Dutch legal search engine, and the relation between citation counts and user clicks. Based on a set of 95,074 documents we nd a Spearman correlation with 24 months of citation data of = 0:39 after 1 month of usage data, and = 0:47 after 12 months.
Legal Information Retrieval (IR) systems still rely heavily on algorithmic and
topical relevance. This does not encompass all aspects of relevance for the user,
as described by Saracevic [
Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). BIR 2020, 14 April 2020, Lisbon, Portugal. 2
This paper presents the analysis of usage and citation data in a legal search engine, for the future purpose of transforming raw counts into a ranking variable that re ects the impact relevance of documents for the users. We address the following research questions: 1. How soon after publication are citation metrics informative to be included in ranking algorithms? 2. To what extent are usage and citations correlated?
In this research we will analyse citation and usage (click) data from the Legal Intelligence IR system, the largest legal IR system in the Netherlands. The contributions of this research are (a) an analysis of legal citation data to determine how soon after publication citation data is informative to be included in a ranking function, and (b) an analysis of the correlation between usage and citations of legal publications. 2 2.1
In IR, the theory of relevance has several dimensions, including algorithmic
relevance, topical relevance, cognitive relevance, situational relevance, and, in
particular for legal IR, bibliographic relevance [
3 As discussed by Mart [
To measure this broader impact, citations alone do not provide enough
information, since not all legal professionals are also author, and the impact of the
publications on these legal professionals will not be visible in citations. Gar eld
himself acknowledged that: \there are undoubtedly highly useful journals that
are not cited frequently" [4, p. 476] \that does not mean that they are therefore
less important or less widely used..." [4, p. 476] An example he uses is Scienti c
American, a journal readers read to keep up to date, but tend not to cite. The
impact of these sources can not be captured by citation measurement. Haustein [
Legal IR systems are a hybrid of academic search and professional search, as they
are used by both legal scholars and legal practitioners [
As Kousha and Thelwall [
For the above reasons, we aim to combine metrics for document usage and
citations. However, because some readers are also authors, a correlation between
usage and citations counts is expected. Perneger [
The KNAW, the Royal Netherlands Academy of Arts and Sciences, has
indicated that it can take up to two years for documents in the humanities gather
su cient citations for research evaluation [
From the document index of the legal search engine, we select all documents that were added to the system between January 1st and June 30th 2017. Documents have both a publication date and a date on which they were added in the system. In most cases, these dates will be the same, but in some cases there are small di erences (for example when the document is published in folio before being made available online or vice versa). To be able to accurately asses the usage of the documents, we decided to use the date added rather than the publication date. This resulted in a set of 536,635 documents.
For each of these documents, we retrieve a unique document identi er and a reference number. Using the reference number, we conduct a search in the document index, counting how many documents refer to this document in their main text. Using the document identi er, we extract the usage data (clicks) from the search engine logs. Citation data After accumulating all citations (excluding self-citations), we see that only 104,048 documents have received citations. This means that (536; 635 104; 048 =) 432,587 documents (81%) did not receive any citations. This might be because some document types (such as books) do not have a reference number that can easily be used for citation extraction5. However, based on citations in other elds, it is also to be expected that a large number of documents does not generate citations.6 Of the documents with citations, 68,781 documents have only one citation. For the analysis how citations aggregate over time, we will use the remaining 35,267 documents that have gathered more than 1 citation since publication (since documents with 0 or 1 citation(s) will generate a at line). We look at the period up until 24 months after publication.
Usage data After accumulating all usage data for up to 24 months after
publication, we see that only 131,494 documents have received usage actions. This
means that (536; 635 131; 494 =) 405,141 documents (75%) did not receive
any clicks. Similar to the citations above, this highly skewed distribution is as
expected. For the analysis of how usage changes over time, we look at documents
that have gathered more than 1 usage interaction (click) since publication. This
gives us a set of 95,074 documents.
4 Usage data is available from 2017 on. For that reason, it was not useful to use older
documents (before 2017).
5 But the citations mentioned in the books are available.
6 See, for example Brody et al. [
To analyse how soon after publication citation data becomes relevant for use in ranking algorithms, we computed the time between the month the cited document became available and the month the citing documents became available7. Because we are interested in the pattern of aggregation of citations, this plot only shows documents that have more than 1 citation. We plotted the aggregated number of citations over time for the mean, median, rst and third quartile. 6
Figure 1 shows that documents gather citations quite quickly, and are informative for use in IR much more quickly than after 2 years, as the KNAW suggested. Even the documents with low number of citations receive their rst citations in the rst months after publication.
The data shows a large di erence between the mean and the median. This
is likely caused by a large number of documents with limited citations, and a
small number with a very large number of citations. This is as expected based on
bibliometric theory [
Figure 2 shows the correlation between citation counts at each month after the documents are made available and citation counts at 24 months. A month after publication (for documents published in January 2017 this means citation data up until the end of February 2017, since some documents were published at the very end of January) we nd a Spearman correlation of = 0:65. We chose Spearman correlation because the data, like all citation data, does not follow a normal distribution but a long-tail distribution with extreme outliers. However, as gure 2 shows, initially the Pearson correlation gives similar results.
Two months after the cited document has become available, the Spearman correlation is = 0:71. For research evaluation purposes, this correlation may not be su cient. But for information retrieval, where we would like to be able to Usage and Citation Metrics for Ranking Algorithms 7 reasonably estimate the impact of a document as early as possible, a correlation of = 0:71 at two months is valuable. It is also possible to update the data regularly8, so increases in citation counts can be incorporated as they occur. 4.2
Similar to the citation data, we see a di erence between the mean (7.11 after 1 month) and the median (2.00 after 1 month) in gure 3.9 This is again caused by a long-tail distribution, and is seen throughout the 24 months.
Figure 4 shows a Spearman correlation between usage after 1 month and usage after 24 months of = 0:63. The Spearman correlation between usage after two months and usage after 24 months is = 0:69.10 8 e.g. monthly 9 The bump visible in the line of the mean between 9 and 11 months, and the decrease visible in the line of the median at 12 months, are the result of errors in the underlying data. In future work, we will research the cause of these errors and correct for them. 10 The bump visible in the line of the Spearman correlation between 9 and 11 months is the result of errors in the underlying data, as was also visible in gure 3. In future work, we will research why these data errors are visible only in the Spearman correlation, and not in the Pearson correlation. 8
In gure 4 the di erence between the Spearman correlation and Pearson correlation is more pronounced. In this research we work with the Spearman correlation because the data has a longtail distribution with extreme outliers. We compute the Spearman correlation between the usage at each month and the citations after 24 months (95,074 documents, see Section 3.2). Figure 511 shows both the Spearman and Pearson correlation coe cients, though we focus on the Spearman correlation because of the longtail distribution of the data with extreme outliers, and because the order of magnitude for usage is di erent than for citations.
The Spearman correlation between 1 month of usage and 24 months of citations is = 0:39. The highest correlation found between usage and 24 months of citations is = 0:47 after 12 months. 11 The dip visible in the line of the Spearman correlation between 9 and 11 months is the result of errors in the underlying data, as was also visible in gure 3 and gure 4. In future work, we will research why these data errors are visible only in the Spearman correlation, and not in the Pearson correlation.
The development of the correlation between usage and citations is as
expected. Brody et al. [
As indicated by Haustein [
Given the two di erent avors of impact that usage and citations represent, both variables have to be considered in order to include relevance impact in a ranking algorithms. However, since usage and citations are correlated (albeit moderately), it would be unwise to add the two factors as separate boost factors in the ranking algorithm of the search engine, since that would overestimate the impact of the publication. Possible solutions are (a) taking the average of the two impact values, (b) taking the lowest of the two values, or (c) taking the highest of the two values. In a large number of situations the average would give an adequate representation of the impact of a document. However, with the example of the Scienti c American in mind, which is highly read but not often cited, there is a risk of disregarding sources which readers use to keep up to date with the eld. In Dutch legal publications this might be overviews (`Kronieken') of recent remarkable case law. Using the lowest of the two values would also disregard these publications. For that reason we choose the highest of the two scores to incorporate as variable in the ranking function, thereby allowing both documents that are used for research and documents that are used to keep upto-date to appear high in the ranking. 5
This research demonstrates that legal publications gather citations from the moment they are published. We nd a correlation of = 0:71 between the citation counts at 2 months and the citation counts at 24 months after publication of the document. We nd a correlation of = 0:69 between the usage after 2 months and the usage after 24 months after publication of the document. This suggests the early citation and usage data can be used as a predictor for later citations/usage for ranking in legal IR.
Usage and citations show di erent forms of impact but are correlated (Spearman's correlation between = 0:39 and = 0:47). This means that usage and citations measure di erent avors of impact. This also means that a usage boost should not be added on top of a citation boost, since that would overestimate the impact of certain publications. As solution we suggest to take the highest of the two values.
In future work we will incorporate these metrics in a ranking algorithm. This will include an impact relevance variable that has limited in uence at the beginning, when the correlation with later usage/citations may not yet be reliable enough, and increases in in uence as the data becomes more reliable.