Citation Metrics for Legal Information Retrieval Systems

Abstract

This paper examines citations in legal information retrieval. Citation metrics can be a factor of relevance in the ranking algorithms of legal information retrieval systems. We provide an overview of the Dutch legal publishing culture. To analyze citations in legal publications, we manually analyze a set of documents and register by what (type of) documents they are cited: document type, intended audience of documents, actual audience of documents and author a liations. An analysis of 9 cited documents and 217 citing documents shows no strict separation in citations between documents aimed at scholars and documents aimed at practitioners. Our results suggest that citations in legal documents do not measure the impact on scholarly publications and scholars, but measure a broader scope of impact, or relevance, for the legal eld.

Keywords

• Legal information retrieval metrics Citations

Introduction

Legal Information Retrieval (IR) has a number of characteristics that distinguish it from retrieval tasks in other domains. One of those aspects is the de nition of relevance, which includes bibliographic relevance, source authority, and source quality as important criteria [ 24 ]. E ective legal IR is not only important for practitioners (lawyers and legal professionals), but also for scholars. In his reective article `Rethinking the Law School', the dean of Leiden University (then dean of Leiden Law School) stated that \For the massive number of research results available via the Internet today, researchers need some guidance on both the content and the quality." [19, p. 243]. Bibliometric-enhanced IR systems may improve the e ectiveness of legal search, by using citation metrics as a form of impact measurement, which can be added as a factor of relevance in ranking algorithms.

But the challenge in legal bibliometrics, and therefore legal bibliometricenhanced IR, is that the legal domain di ers from other research domains in two manners: (1) its strong national ties and (2) the often strong interconnection ? Also a liated with Legal Intelligence as business analyst. between research and practice. In the Dutch legal domain this is demonstrated by the fact that scholars and practitioners use the same legal IR systems, and by the lack of distinction between legal scholarly and professional publications. This lack of distinction is one of the reasons why a citation-index, which in other elds measures the impact of scienti c publications, has not yet been established within the Dutch legal domain. We are currently in the process of creating this index for Dutch legal publications, covering all Dutch legal publishers and government publications. This also requires the creation of a framework for using citations in IR systems for the legal domain.

Citation-index measurements in the hard sciences are mostly associated with the notion of scienti c impact. However, because the Dutch legal domain has a di erent publication culture than other research domains, one of the questions that arises is what citations in legal publications represent. Can we separate the impact of a publication on the scholarly debate from its impact on the rest of the legal domain, or is this inseparable? Only when it is established what we are measuring when we create a citation index, can we use that information to improve ranking algorithms in legal IR systems.

In this paper we explore the type of citations that occur in the legal citation network. We manually analyze a set of documents { legal cases, practitioner oriented journal publications, and scholarly oriented journal publications { and register by what (type of) documents they are cited: document type, intended audience of documents, actual audience of documents and author a liations. This exploration leads to a framework for using citation metrics to improve ranking in legal IR. The following research questions are addressed in this paper: (1) what is the theoretical background for citation metrics in legal IR? (2) Does data analysis of citations in legal publications support this theoretical background? 2 2.1

Background

Citations as a Form of Impact Measurement

The use of citations as a proxy for impact was introduced by Eugene Gar eld. Gar eld stated that \Since authors refer to previous material to support, illustrate, or elaborate on a particular point, the act of citing is an expression of the importance of the material. The total number of such expressions is about the most objective measure there is of the material's importance to current research." [7, p. 23] De Bellis, referring to the work of Merton, stated that: \Citing, speci cally, is the same as peer-reviewing, just on a smaller scale. Hence bibliographic citations are atomic components of the cognitive and reward system of scienti c communication." [2, p. 30][ 14 ] De Bellis also stated that \Being cited by other authors is not simply a matter of intellectual lineage. When the score gets high, it is likely that the cited document is exercising an impact on citing sources" [2, p. 32] And that: \This forward-pushing potential, in turn, is the hallmark of scienti c quality" [2, p. 32] Another description of the meaning of citation measurements comes from Kurtz and Henneken: \The measurement of an individual's scholarly ability is often made by observing the accumulated actions of individual peer scholars. A peer scholar may vote to honor an individual, may choose to cite one of an individual's articles, and may choose to read one of an individual's articles." [12, p. 696]

What these authors have in common is that they consider the total number of citations a proxy for the impact of the publication on other scholarly publications and scholars.1 2.2

Dutch Legal Publications

The topic of legal research and the Dutch legal publishing culture has been extensively described by Stolker [ 19 ]. He argues that because law is a national research topic, and publications are often bound, by topic and language, to national publications, an analysis of such publications should be done on a national level. Stolker also notes that the legal publishing culture has a strong tradition in book publishing. Even though law journal articles are becoming more important, a perspective of legal publications is not complete without considering books.

He describes three types of legal journals: \journals primarily focusing on the scholarly debate; journals merely focusing on dissemination (notes/annotations and short commentaries); and journals { probably the majority { doing both." [19, p. 257] He also con rms that law journals often do not have external peer review, but are reviewed by the editorial board. This indicates that the theory and methods for impact measurement from the hard sciences, cannot simply be copied for the legal domain.

In his book, Stolker cites several sources [ 9 ][ 21 ] that are critical of citation metrics as a form of impact measurement for legal publications, which might explain why a legal citation index has not been created up until this point.2 However, he focuses exclusively on impact measurement for research evaluation systems, not for use in IR systems, while Gar eld [ 8 ] originally focuses on applications in library management and the creating of reading lists for scholars and students. Use for research evaluation is mentioned, but does not appear to be Gar eld's original focus.

Snel [ 17 ] states that there are three reasons for citing in scholarly legal publications: to provide context for the research, to legitimize statements made in the research, and to allow others to check the quality of the research. His article is aimed at scholars, and contains advice for writing sound scholarly publications. But he mentions non-scholarly publications as possible sources to reference to [17, p. 255]. To provide societal context for the legal research, he writes that authors may refer to newspaper articles. To legitimize their statements, they may refer to law articles, and legal cases [17, p. 256]. This demonstrates that citing non-scholarly sources is accepted practice in scholarly legal publications. 1 Work by e.g. Teufel [ 22 ] narrows this down by looking at the words surrounding the citation, to see whether the author cites in a positive or negative manner, but this falls out of the scope of this paper. 2 A proof of concept was presented by Wirt Soetenhorst in 2017 [ 18 ], but a literature search has not returned any information that this citation index has been completed. 2.3

Citations in IR

An example of using citations as ranking criterion in academic search, including potential negative e ects, is the work by Beel and Gipp [ 1 ]. They investigated the role of citations in Google Scholar and found that citations have a signi cant in uence on the ranking, though more so for title searches than for other searches [1, p. 442,444]. It appears that since their research, Google has slightly adapted the algorithm to also include how recently the article has last been cited.3 This is most likely done to mitigate the Matthew e ect, where highly cited documents, which are likely older to have been able to generate such a high number of citations, remain at the top at the expense of newer publications. By displaying these highly cited documents at the top, they are more likely to be cited, creating a self-reinforcing e ect. Beel and Gipp [ 1 ] named this Matthew e ect as one of two main points of criticism for using citations in ranking algorithms in their paper.

Use of citations in legal IR systems can be seen in, for example, the American legal IR system Westlaw4. As Jackson and Al-Kofahi [ 10 ] indicate though, the more factors like citations play a role in ranking, the harder it is for a user to understand why certain results appear in certain positions. This appears to be one of the reasons why Dutch legal IR systems have focused on thesauri and synonyms to improve their systems. 5

Furthermore the scale of the Dutch jurisdiction, and thereby the size of Dutch legal IR companies and the datasets they have available to them, do not compare to the US and Westlaw. And Westlaw's techniques cannot simply be copied to other jurisdictions, because of the large di erence between common law jurisdictions (like the US) who focus mainly on case law, and civil law jurisdictions (like the Netherlands and most continental European countries), who focus on legal codes, with case law as an interpretative tool [ 25 ].

In the Netherlands Van Opijnen [ 15 ] and Winkels and colleagues [ 25 ][ 26 ][ 27 ][ 28 ] have applied citation analysis to Dutch law and case law, but did not include literature.

Beel and Gipp state that \citation measures impact but not quality in general." [1, p. 440] Gar eld, though a proponent of using citations as a form of impact or `signi cance' [8, p. 473] measurement, also noted that: \citation frequency re ects a journals value and the use made of it, but there are undoubtedly highly useful journals that are not cited frequently" [8, p. 476] \that does not mean that they are therefore less important or less widely used than journals that are cited more frequently. It merely means that they are written and read primarily for some purpose other than the communication of original research ndings." [8, p. 476] An example he uses is Scienti c American, a widely read journal that he states users read to keep up to date. 3 https://scholar.google.com/intl/en-US/scholar/about.html 4 http://lscontent.westlaw.com/images/content/L-355700_

West-Search-brochure .pdf 5 https://clin28.cls.ru.nl/#abstract-36

This indicates that for legal IR systems, it is important to research whether citations measure impact on scholarly publications and scholars, as is the case in hard sciences. But it also means that it would be important to research whether impact on scholarly publications and scholars is an important factor in relevance assessments by users of those systems before implementing it, considering that the user group consists of both scholars and practitioners. 3

Methods

To discover whether citations in legal publications signify impact as in the hard sciences, we manually analyze the type of documents that cite the documents researched. For 9 documents from the year 2014 { 3 legal cases, 3 practitioner oriented journal publications, and 3 scholarly oriented journal publications { we have registered what (type of) documents they are cited by.

For these citing documents, we register what type of document it is, what the target audience of the publication is, what the actual audience of the publication is, and what the a liation of the author(s) is, with the aim of estimating whether this citing document is aimed at and used by scholars or practitioners.6 3.1

Document selection criteria

Bornmann et al. [4, p. 214], citing Boyack [ 5 ], have remarked that the distribution of citation counts over documents is skewed.7 This means that a large portion of documents receive no citations, whereas a small number of documents receive a large number of citations [ 13 ]. For that reason, a random selection of documents would not be informative for our study, because the majority of randomly selected document has no or very few citations. We selected the documents for our analysis as follows.

We chose documents from the year 2014 because of the time it takes for documents in the social sciences to gather citations [ 21 ]. The citing documents were from the period 2014-2018, the most up-to-date data available at the time of the research.

To be able to research what types of documents cite, we needed to select documents from 2014 that were likely to have been cited. Based on the assumption that documents that are sought often are also read often, and therefor likely to be cited, we used the 2015 query logs from the Legal Intelligence system. We sorted the queries by frequency of occurrence. We manually went through this 6 In the Dutch legal domain the factor whether or not a document is peer-reviewed is not a reliable indicator of whether an article is scholarly or practice oriented, because peer-review is not a common practice. In most cases there is a form of softpeer review by the editors of the journal, who may be scholars themselves, but may also be practitioners [ 11 ].

The Dutch legal journals are also not classi ed in A-, B- and C- journals, as is done in economics [20, p. 32]. 7 See also Bonaccorsi et al. [ 3 ] list and looked at all queries that are clearly related to a case (journal identi er, ECLI number or party/case name) or journal article (journal identi er or title (more than one word)) published in 2014. Case law and journal articles from other years were skipped (such as Haviltex), to avoid a citation bias based on time since publication. The documents selected are the rst documents in the query list that met these criteria.

The journal articles mentioned at the top of the query list for all users were practitioner oriented publications, so we created a separate query log with queries from users a liated with a university. From that log, we selected the rst three journal articles from 2014 that were published in a journal that (also) publishes scholarly articles by an author that is a liated to a university. The documents selected were:

Case law: { ECLI:NL:HR:2014:799 (query: Cancun) (DocumentID: 12923916) { JAR 2014/298 (DocumentID: 14290648) { ECLI:NL:HR:2014:3351 (DocumentID: 14223358)

Practitioner oriented articles:

{ NJB 2014/1962 (query: Bescheidenheid en moed ) (DocumentID: 14151738) { ArbeidsRecht 2014/25 (query: Informatieverstrekking aan derden in het licht van goedwerkgeverschap: is zwijgen de norm? ) (DocumentID: 13002758) { NJB 2014/1225 (query: de andere kant van de ZSM-medaille) (DocumentID: 13330606)

Scholarly articles: { NJB 2014/804 (DocumentID: 12987162) { NJB 2014/2056 (DocumentID: 14177758) { Strafblad 2014/68 (DocumentID: 14331724)

3.2

Manual document analysis and classi cation

For each of the selected documents, we searched for documents citing it based on the unique document identi er. We extracted the identi er of the citing document, the title, the name(s) of the author(s) and the document type. We then classi ed each of the documents as being directed towards practitioners or towards scholars, using the following criteria: { We classi ed government publications and case law as directed towards practitioners. Because these publications are created as a byproduct of the practice of the legislature, the executive, and the judiciary; they were not written for the advancement of scholarship. News articles, annotations to case law and noti cations of publications (short summaries with references to new books or case law) were also classi ed as directed towards practitioners, because of the practical nature of these publications. { Dissertations are considered to be directed towards scholars. { Journal articles and books can be directed towards either practitioners or scholars, or both. For journals and books, we checked the (self-reported) publisher information to nd out whether the journal or book in its entirety was directed more towards scholars or practitioners. We classi ed a publication as aimed towards practitioners if the title or description mentioned things like `practical information' or `for practice'. If a journal states that it has both scholarly and other articles, we considered the annotated case law and similar documents from the journal as directed towards practitioners, and articles that analyse several cases and/or literature, as a scholarly article. In case of uncertainty, the documents were marked as practitioner oriented. { We retrieved the a liation of the authors from the author information in the document or through Google. Government publications, such as case law and parliamentary documents, are classi ed as written by practitioners, despite the high/scholarly level of quality of some of these publications, for the reasons mentioned above. An author can have multiple a liations. To give a publication the bene t of the doubt, we classi ed the author as scholarly if one of the a liations was a university. For documents with more than one author, the publication was classi ed as scholarly if one of the authors was a liated with a university.

To verify whether scholarly publications were also actually used by scholars, we analyzed the ratio of usage from users associated with a university (scholarly users) versus usage from practitioners. We rst queried for usage from all users with a user ID a liated with a university company ID (this data includes students as the function of the user is not included in the data), followed by users with a user ID a liated to other organization types (such as government, courts, law rms and corporations).

The usage is measured in online usage actions, not unique users; the same user can use a document on multiple occasions. A usage action can consist of opening a document, saving it or sharing it with a colleague. The usage measures only online usage in the Legal Intelligence system. It is possible that users have alternative methods to access information, for example through paper versions of books and journals. We have no reason to assume that this would apply more to one group than to the other. 4

Results

For both the selected documents and the citing documents, we consider a document to be scholarly when the publication form (and audience) in the Document Class is Scholars, at least one of the authors is a liated to a university, and the Usage shows use by users a liated to a university.8

Because the group of users a liated to a university is much smaller than the group of users a liated to other organization types, it is not possible to 8 Information about the citing documents can be found at https://github.com/ G-Wiggers/Citation-Metrics-for-Legal-Information-Retrieval-Systems. determine a xed value or ratio above which a document can be considered scholarly. Therefore the threshold is set to at least one usage action from a user a liated to a university. It is, of course, very well possible that an author a liated with a law rm can write a scholarly article. However, the number of people a liated with multiple organization types, including a university, shows that many authors who wish to conduct scholarly research would be able to be a liated (part time) to a university as well.

The stringent cumulative conditions are chosen to ensure the least possible false positives in scholarly publications. This allows us to separate between the purely scienti c impact of publications, as measured by citation indexes in the hard sciences, and broader impact on the (practitioners in the) legal eld.

Table 1 quanti es the usage and citations for the nine documents that we analyzed. Based on the analysis of these documents { 9 cited documents and 217 citing documents { it appears that there is no strict separation in citations between documents aimed at scholars and documents aimed at practitioners. Though the practitioner documents are mainly cited by other practitioner documents, and only rarely by scholarly documents, this appears to be the case for scholarly publications as well. This might be caused by our chosen classi cation method, which has strict criteria before a publication can be classi ed as scholarly, meaning few documents are.

If this analysis proves to hold on a larger sample size, this would indicate that citations in legal documents do not measure the impact on scholarly publications 9 This article was cited in 1 document, which summarized the content, and was therefore classi ed as a publication for practitioners. and scholars, but measure a broader scope of impact, or relevance, for the legal eld. A citation and summary of a scholarly publication in a journal could signify that the document is (also) relevant for practitioners. A highly cited case could signify that a novel problem was solved (e.g. the rst case that dealt with the question whether a digital item is a good in the sense of property law), or that the court veered from a previous ruling, and could therefore be cited in both scholarly and practitioner publications. A citation from a journal article in a reference work could signify that the paper had a novel contribution to legal scholarship and was of high quality. Though the reasons for citing di er, they are all indications of relevance for the legal domain as a whole. (See Figure 1)

It is noticeable that documents that are highly used, are often highly used for both groups 10. This seems to indicate that both groups are looking for some similar features in publications. This appears to be the case for journal articles, but also for (annotated) case law and law commentaries, document types considered to be aimed at practitioners. 10 The full dataset can be found at https://github.com/G-Wiggers/ Citation-Metrics-for-Legal-Information-Retrieval-Systems

The di erence in use between publications could in part be explained by the access rights system of the IR system. Though the IR systems works the same for every user, only results from publications the user has a subscription to are shown in the results list. All government publications are freely accessible to all users, as well as open access publications. It appears that certain journals have a higher subscription rate than others, and that digital availability of books is limited to a small share of the user group. However, though this might explain the di ering levels of use, it remains noticeable that the level of use appears to be similar between the user groups. This is likely related to Gar eld's remark that scholars do not read only works transmitting novel research, but that they are also interested in publications that can keep them up to date.

If this usage analysis holds for a larger sample size, this could mean that both scholarly and practitioner users of legal IR systems are looking at similar features when determining whether a document is relevant or not. This concurs with the ndings in our previous research on factors that in uence the perception of relevance in legal IR systems [ 24 ].

This means that though citations provide valuable information about the relevance of legal publications for ranking algorithms in legal IR systems, citations alone do not provide enough information. Based on the data of this research a combination with usage metrics appears to be an obvious combination, though combinations with certain metadata, such as issuing instance and source, might also prove useful. Similarly, a citation analysis to determine the level of depth or speci city of a document, as suggested by Kurtz and Henneken [ 12 ] might improve the ranking in legal IR systems, given the factors of relevance for this user group [ 24 ]. 6

Conclusion

In this paper we addressed two research questions:

(1) What is the theoretical background for citation metrics in legal IR? The literature review shows that in the hard sciences, citations are considered to mainly measure the impact of a publication on other scholars and scholarly publications. It also shows that legal publications use non-scholarly publications to support their claim. The analysis shows that these scholarly publications in turn are used in non-scholarly publications. This indicates that citations in legal publications do not measure impact in the same way as in the hard sciences, but measure a broader form of impact.

(2) Does data analysis of citations in legal publications support this theoretical background? The results of this research suggest that citations in legal documents indeed do not measure the impact on scholarly publications and scholars, but measure a broader scope of impact, or relevance, for the legal eld. Though there appear to be multiple reasons for citing, they all appear to indicate a form of relevance for the legal domain.

Both scholarly and practitioner users of legal IR systems appear to look at similar features when determining whether a document is relevant or not. This means that although citations provide valuable information about the relevance of legal publications for ranking algorithms in legal IR systems, citations alone do not provide enough information. Further research will investigate which additional information is needed to improve ranking algorithms in legal IR systems.

Because of the small sample set, the results of this paper cannot be extrapolated to all publications in the Dutch legal domain. Because of the national characteristics of legal domains, this example from the Netherlands can also not be extrapolated to other countries. It does, however, provide some valuable rst insights in the possibilities of using citation metrics in Dutch legal IR systems, which allows for further exploration. 7

Acknowledgements

The authors thank the employees of Legal Intelligence, in particular dr. T.E. de Greef and P. van Boxtel for their cooperation in this research.

References

1. Beel , J. , & Gipp , B. ( 2009 ) Google Scholar's ranking algorithm: the impact of citation counts (an empirical study) . In Third International Conference on Research Challenges in Information Science (RCIS) , 439 { 446 . IEEE.

2. De Bellis , N.: History and Evolution of (Biblio)Metrics; in Cronin, B . & Sugimoto , C. R. (eds.) ( 2014 ), Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact , 23 { 44 ( 2014 )

3. Bonaccorsi et al.: Do social sciences and humanities behave like life and hard sciences? Scientometrics 2017 ( 112 ), 607 { 653

4. Bornmann , L. , Bowman , B.F. , Bauer , J. , Marx , W. , Schier , H. , Palzenberg , M. : Bibliometric Standards for Evaluating Research Institutes in the Natural Sciences , in Cronin , B., Sugimoto , C. R. (eds.): Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact , 201 { 223 ( 2014 )

5. Boyack , K.W. : Mapping knowledge domains: characterizing PNAS . Proceedings of the National Academy of Sciences of the United States of America , 101 , 5192{ 5199

6. Brody , T. , Harnad , S. : Earlier Web Usage Statistics as Predictors of Later Citation Impact . Journal of the Association for Information Science and Technology , 57 ( 8 ), 1060 - 1072 ( 2006 )

7. Gar eld, G.: Citation Indexing: its theory and application in science, technology, and humanities . John Wiley & Sons, Inc., New York ( 1979 )

8. Gar eld, G.: Citation Analysis as a Tool in Journal Evaluation . Science 178 ( 4060 ), 471 { 479 ( 1972 )

9. Van Gestel , R. , Vranken , J. : Assessing Legal Research: Sense and Nonsense of Peer Review versus Bibliometrics and the Need for a European Approach . German Law Journal 12 ( 3 ), 901 { 929 , ( 2011 )

10. Jackson , P. , Al-Kofahi , K. : Human Expertise and Arti cial Intelligence in Legal Search , in: Geist, A. , Brunschwig , C.R. , Lachmayer , F. , Schefbeck , G . (eds.): Strukturierung der Juristischen Semantik . Weblaw, Bern ( 2011 )

11. Kaltenbrunner , W. , Rijcke de, S.: Quantifying ' Output' for Evaluation: Administrative Knowledge Politics and Changing Epistemic Cultures in Dutch Law Faculties . Science and Public Policy 44 ( 2 ), 284 - 293 ( 2017 )

12. Kurtz , M.J. , Henneken , E.A. , Measuring metrics a 40year longitudinal crossvalidation of citations, downloads, and peer review in astrophysics . Journal of the Association for Information Science and Technology , 68 , 695{ 708 ( 2017 )

13. Merton , R.K. : The Matthew e ect in science, II. Cumulative advantage and the symbolism of intellectual property . Isis 79 , 606 { 623 ( 1988 )

14. Merton , R.K. : The normative structure of science . In R.K. Merton (ed.): The sociology of science: theoretical and empirical investigations , 267 { 278 ( 1973 )

15. van Opijnen, M. : Op en in het web . Hoe de toegankelijkheid van rechterlijke uitspraken kan worden verbeterd PhD University of Amsterdam ( 2014 )

16. Perneger , T.V.: Relation between online "hit counts" and subsequent citations: prospective study of research papers in the BMJ . BMJ 329 ( 7465 ), 546 { 547 ( 2004 )

17. Snel , M.V.R. : Hoera, een lijstje! Over bronvermelden . Ars Aequi 3 , 254 { 260 ( 2018 )

18. Soetenhorst , W.J.: Een juridische citatie-index: het proof of concept is voorhanden . Nederlands Juristenblad 17 ( 915 ), 1184 { 1186 ( 2017 )

19. Stolker , C. : Rethinking the Law School: Education, research, outreach and governance . Cambridge University Press, 2015 .

20. Stolker , C. : Een discipline in transitie: Rechtswetenschappelijk onderzoek na de Commissie Koers . Recht en Methode in onderzoek en onderwijs 1(3 ), 13 { 43 ( 2011 )

21. Royal Netherlands Academy of Arts and Sciences: Judging research on its merits { An advisory report by the Council for the Humanities and the Social Sciences Council . Royal Netherlands Academy of Arts and Sciences, Amsterdam ( 2005 )

22. Teufel , S. : Argumentative zoning for improved citation indexing. Computing attitude and a ect in text: Theory and Applications . Springer, Dordrecht 159 { 169 ( 2006 )

23. West , J.D. , Vilhena , D.A. : A Network Approach to Scholarly Evaluation, in Cronin , B., Sugimoto , C. R. (eds.): Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact . MIT Press, Cambridge Massachusetts 151{ 165 ( 2014 )

24. Wiggers , G. , Verberne , S. , Zwenne , G-J. : Exploration of Intrinsic Relevance Judgments by Legal Professionals in Information Retrieval Systems . In: Proceedings of the 17th Dutch-Belgian Information Retrieval workshop , 5{ 8 ( 2018 )

25. Winkels , R. , Ruyter , J. , & Kroese , H.: Determining authority of dutch case law . Legal Knowledge and Information Systems , 235 , 103{ 112 ( 2011 )

26. Winkels , R., de Ruyter, J.: Survival of the ttest: network analysis of dutch supreme court cases . In International Workshop on AI Approaches to the Complexity of Legal Systems . Berlin, Springer. 106 { 115 ( 2011 )

27. Winkels , R. , Boer , A. , Plantevin , I. : Creating Context Networks in Dutch Legislation . In: Proceedings of JURIX , 155 { 164 ( 2013 )

28. Winkels , R. , Boer , A. , Vredebregt , B., van Someren , A. : Towards a Legal Recommender System . In: Proceedings of JURIX 271 , 169 { 178 ( 2014 )