AI research finds itself in the third boom of its history, and in recent years, AI-related themes have gained considerable popularity in new disciplines, such as law. This paper explores what legal research on AI constitutes of and how it has evolved, while addressing the issues of information retrieval and research duplication. Using Latent Dirichlet Allocation (LDA) topic modeling on a dataset of 3931 journal articles, we explore three questions: (a) Which topics within legal research on AI can be distinguished? (b) When were these topics addressed? and (c) Can similar papers be detected? The topic modeling results in a total of 32 meaningful topics. Additionally, it is found that legal research on AI drastically increased as of 2016, with topics becoming more granular and diverse over time. Finally, a comparison of the similarity assessments produced by the algorithm and a human expert suggest that the assessments often coincide. The results provide insights into how a legal research on AI has evolved over time, and support for the development of machine learning and information retrieval tools like LDA that assist in structuring large document collections and identifying relevant articles.
• Computing methodologies → Topic modeling; • Applied computing → Law. information retrieval, topic modeling, legal research
Artificial intelligence (AI) research finds itself in the third boom of its history, fuelled by increased funding, scientific breakthroughs such as deep learning, and widespread public speculation relating to the scope and impact of these breakthroughs. While decades ago the interest in AI research was generally limited to specific disciplines (e.g. computer science, philosophy), during the past years AI-related themes have gained considerable popularity in new disciplines, such as law.
With over 2500 publications already by the year 2015 referring
to ‘artificial intelligence’ [
This paper aims to explore what legal research on AI constitutes of and how it has evolved while addressing the issue of information retrieval and the risk of research duplication. We develop a methodology that distinguishes topics in a collection of documents (in this case journal publications), allows exploring the evolution of the topics over time, and detects similarity between documents, with the purpose of providing solutions for reading and analyzing a number of publications in bulk in ways that humans cannot. Consequently, we aim to answer the following research questions (RQ):
RQ1: Which topics within the field of legal research on AI can be distinguished (‘What’)? A methodology would provide insight in how legal research on AI is structured, and it would allow classifying publications in sub-topics, which would enhance information retrieval.
RQ2: What (i.e. about which topics) has been written when (‘What - When’)? This question contributes to the understanding of which topics have emerged, remained, or lost the interest of legal scholars. The analysis of the What - When question will provide information about the evolution of legal research on AI.
RQ3: Can similar papers be detected? Considering the sharp increase of publications, it may be becoming increasingly dificult to find publications on similar research questions, which may even result in reproduction of scholarship because prior publications are overlooked. This question explores a methodology that allows detecting thematically similar documents in a given corpus. 2
One of the innovations in this paper is to use unsupervised
machine learning to categorize legal research on AI and to map how
legal scholarship on this topic has developed. The idea of smart
literature reviews has received prior attention, on the grounds of
how manual searches for existing literature - especially in matured
domains where scholarship is abundant - is not eficient and might
have a negative impact on the quality of new research [
To the best of our knowledge, legal scholarship as such has
not yet been analyzed using LDA. Within the narrow confines of
the field of research labeled as AI and Law, a lot of legal research
on information retrieval has been published in the past decades.
Notwithstanding that the volume of scholarship on information
retrieval pertaining to the discipline of computer science alone is
vast to say the least [
Still, the question of how to capture and visualize the
development of an entire legal sub-field like legal research on AI has
barely been explored. One of the avenues of exploration has been
shaped by Bench-Capon et al., who have previously focused on
the proceedings of the International Conference on AI and Law,
ifrst held in 1987, to make a 25-year retrospective of the research
generated therein by describing the scholarship progress through
illustrative papers selected from various editions [
However, given the limitations of legal databases or the way they are used by researchers, making comprehensive overviews of existing literature remains a considerable hurdle. This is all the more so in the past four years, when the production of legal scholarship on artificial intelligence seems to have grown considerably (see Figure 3 in the Appendix). This paper aims to fill this research gap by proposing and testing an unsupervised machine learning approach to the clustering of literature in this field, namely topic modeling. 3 3.1
The corpus includes a total of 3931 journal articles obtained from the HeinOnline database. Absent a centralised, comprehensive, open access repository for international legal scholarship, we focused on one of the commercially available databases. HeinOnline is one of the leading international databases on legal materials, which contains over 170 million pages of literature and indexing over 2700 law journals. In the section ‘Law Journal Library’, section type ‘Articles’, the corpus covers literature available in the database between 1960 and 2018. Unlike arXiv, HeinOnline does not have a section on ‘artificial intelligence’. The total number of retrieved articles reflects the results of a boolean search using the keywords ‘artificial intelligence’, namely all articles which include both terms.
Resulting articles therefore discuss a wide array of aspects relating to artificial intelligence, and do not, as such, focus on specific technical or legal issues. For the purpose of this study, we assume that even one reference to the keywords is suficient to include an article in the corpus.
The articles in the corpus follow a power law distribution, where a relatively large number of publications is written by a low number of authors, and few publications by many authors (see Figure 4 in the Appendix). The same distribution applies to the number of publications per author(s) with roughly 28 authors having more than 5 publications (see Figure 5 in the Appendix). 3.2
Topic Modeling
3.2.1 Latent Dirichlet Allocation. Latent Dirichlet Allocation (LDA)
[
LDA is a generative, probabilistic model for a collection of
documents, which are represented as mixtures of latent topics, where
each topic is characterized by a distribution over all words of the
collection of documents. The basic representation unit of the
documents is the word, i.e. all distinct terms are extracted from the
document collection along with their frequencies (per document),
which is the so called ‘Bag-of-Words’ model [
On a conceptual level, the algorithm tries to discover topics that
can represent the collection of documents. Each document is
generated from a mixture of these topics and each topic is generated
from a probability vector (distribution) over all words. Assuming
such a generative model for any collection of documents, LDA’s
goal is to try and ‘backtrack’ this process, i.e. find a set of topics
that are likely to have generated the whole document collection.
3.2.2 LDA Variants. In this paper, we used the model of Blei et al.
[
Nevertheless, there are extensions of the LDA model towards
topic tracking over time [
Bruggermann et al. applied DTM on the RCV1 Reuters corpus
(810.000 documents) with weekly time epochs [
Pre-processing of the corpus involved filtering for articles written only in the English language and afterwards removing common English stop-words (the, of, etc.) as well as removing some very common corpus-specific words (subject, supra, part, etc.). Moreover, we considered the presence of either unigrams (one words) or bigrams (two words) so as to be able to capture some concepts like ‘dispute resolution’.
Subsequently, LDA was used to identify topics in the corpus.
For identifying the number of topics ( ), we used perplexity [
The topic validation consisted of two steps. First, three researchers inspected the 20 terms for each topic in order to label the topic (e.g. ‘military technology’). Second, paper titles were inspected to determine whether the paper titles supported the assigned label - if not, the label would, if possible, be adjusted. In this respect, the researchers were presented with a list of paper titles for each topic. The validation process indicated that the vast majority of the LDA-produced topics were substantively meaningful.
Table 1 reveals the topics distinguished by the LDA model. It includes the topic IDs (not meaningful), the ten terms that have the highest weights in relation to the topic, and the labels the human coders assigned to the topics.
Three miscellaneous topics were identified: id21, id25, and id33. Both the inspection of the 20 words and the titles did not result in a substantively meaningful label or description for these topics. It was decided to not remove the words in these topics and to not re-run the topic modeling algorithm, as it was expected that the removal of words could introduce selection bias in the corpus. Moreover, the identification of the three miscellaneous topics does not afect the relevance or interpretation of the other topics.
The results show a wide range of topics, varying from tax to
military technology to copyright. Within each topic, diversity could
still be observed. For example, for the topic of algorithmic
decisionmaking and quantitative methods, paper titles such as ‘An FDA for
Algorithms’ [
An important issue concerned the initial selection of journal articles. The articles were selected based on the search string ‘artificial intelligence’. The number of occurrences of this string presumably is, however, not an entirely accurate proxy for measuring whether the paper actually is about artificial intelligence. It might be that papers on public policy or competition & markets mention the term ‘artificial intelligence’, but do not primarily focus on artificial intelligence or even on technology or digital matters. An additional selection was therefore required. Consequently, three of the researchers (authors) independently went through the topic list and the related words as displayed in Table 1 in order to determine whether the labels and words are likely to be related to artificial intelligence, digital, and/or technology. A perfect agreement could be observed for the vast majority of topics. In the few instances of disagreement between the coders, the disagreements were resolved through a brief discussion. Ultimately, a total number of 18 topics was selected (Table 1, in bold). The 18 topics were used in subsequent analyses. 4.2
To answer this question we needed to determine which of the 18 topics that were selected in the previous section, have gained or lost attention over time. The first step is to extract the dominant topics of each paper by using the document-topic vectors (as described in section 3.2.3). More specifically, we denoted for each document the first three dominant topics 1−3, based on their relative weights (contributions) in descending order in the document-topic vector d . To assess the relevance of each document’s topics, two researchers manually reviewed the first five topics – sorted by means of contribution – from a sample of documents. They agreed that for most documents, the relevance dropped significantly after the third topic, since the latter topics provide no substantive contribution to the paper. Based on the results of the inspection, the first three topics were denoted as dominant. We define the frequency of a topic as the number of times that a topic is dominant in all articles.
Linking the topic frequencies to the year of publication, the count of papers addressing each topic every year was computed. Figure 1 depicts how the 18 selected topics evolved over time (1960 - 2018). id 0 1 2 3 4 5 6
Figure 2 shows how the topics have evolved in relation to other
topics across six major periods in AI history (taken from [
As a measure of relative topic popularity during a certain period, the ratio of papers concerned with each topic to the total number of papers written in a certain period is computed. Caution needs to be exercised when interpreting the relative topic popularity. Considering the relatively low number of publications before approximately 1986 (see Figure 1), just before the second AI winter, not much weight should be given to the topic popularity in those years, as small changes in the number of publications can have substantial impact on the popularity of one topic relative to other topics.
For example, as visualized in Figure 2, scholarly output produced during the first AI boom concerned, among others, knowledgebased systems (13.16 %) and algorithmic decision-making and quantitative methods (7.90 %), however, trends in legal technology and information retrieval were prominent topics in this period, forming the subject-matter of 45% and 21% of the papers, respectively. As time advances, topics like financial regulation & technology , autonomous vehicles, surveillance, software licensing and Internet governance appear.
Another illustration is reflected by the developments visible around the deep learning era. The topics that gained popularity (compared to the previous period, namely the third AI boom) with rise of deep learning are regulation of innovation, privacy, algorithmic decision-making and quantitative methods, financial regulation & technology, military technology and autonomous vehicles. The largest decreases in popularity in this period are those of knowledge-based systems and trends in legal technology. The popularity of the other topics underwent popularity changes below the average for this period. The interest in most topics grew during the deep learning era.
In addition, we observed several interesting trends regarding the evolution of pairs of topics (e.g. algorithmic decision-making and quantitative methods with knowledge-based systems and information retrieval with trends in legal technology), as well as individual topics, e.g. privacy, which emerged as a topic during the first AI winter , 1The label ‘ADM and quantitative methods’ refers to the topic algorithmic decisionmaking and quantitative methods (id14). but interest in which saw several relatively insignificant changes before the deep learning era, when its popularity underwent a high increase. 4.3
As mentioned in Section 3.2.3, each document is represented by a
vector d , where each component of the vector shows the weight
of each topic in that specific document. For any pair of documents
we can use cosine similarity [
The detection of publications that have similar topics enhances information retrieval and reduces the risk of reproduction of scholarship because prior publications are overlooked. Having computed the similarity between all document pairs in our corpus, we obtained some 7,436,296 similarity scores after adjusting the algorithm to avoid computing the similarity between the same document pair twice.
To explore the substantive meaning of the ensuing similarity scores, the results produced by the cosine similarity algorithm were compared to substantive similarity as defined by a legal expert one of the authors. The goal of the inspection was to explore (1) the extent to which papers were similar and (2) whether diferences in similarity scores produced by the cosine similarity measure are substantively meaningful diferences. For this, five papers for diferent topics were selected - the seed papers. Each of these five seed papers were compared with five papers with diferent similarity scores - the comparison papers: one similarity score in the .55-.65 range, one in the .65-.75 range, one in the .75-.85 range, one in the .85-.95 range, and one in the .95-1.00 range. As a result, a total of ifve seed papers and 25 comparison papers were inspected.
Without knowledge as to which similarity range the comparison paper would fall under, the legal expert ranked the similarity for each pair of papers (seed paper - comparison paper) for each topic separately. A Spearman’s rank correlation test showed a high correlation (r = .62, p < .001) between the ranks based on the machinegenerated similarity scores and the ranks provided by the expert. The comparison took place on four levels: the paper title (i.e. to what extent do the paper titles suggest similarity?), the research question level (i.e. are the research questions similar?), the focus or sub-topic level (i.e. which sub-topics does the paper focus on, which angle or perspective does it take?), and the citation level (i.e. is there a reference from one paper to the other?).
The inspection of the paper suggests that papers with similarity
scores of .85 or lower may share substantive similarity, but in a
limited way at best (e.g. pairs of publications where one article [
When inspecting citations between papers with very high similarity scores, references from one paper to another were sometimes found, and sometimes not. In some instances when no citation was found this could be due to the fact that papers were published in subsequent years and authors did not have the opportunity to cite the published version of the similar paper. Nevertheless, there are instances where a reference was expected in the papers, but not found. 5
The landscape of legal research on AI has undergone considerable changes with respect to the volume of scholarship tackling topics dealing with AI. In this context, we were interested in what exact topics authors focused on, and how these topics shifted over time. For this, we applied LDA topic modeling to identify topics and analyze how journal articles were distributed across topics as well as across time (1960-2018).
The main finding for the first research question, namely what topics can be distinguished in the corpus of legal papers on AI, is that 35 topics can be identified, 32 of them being meaningful (and three miscellaneous topics). Overall, the model performed considerably well in identifying latent topics in our corpus (see Table 1 above).
The second research question dealt with the evolution of topics throughout the diferent periods which can be identified in the history of AI. The topic river displayed in Figure 1 above shows fundamental changes in legal research on AI from two perspectives: on the one hand, the total number of papers referring to artificial intelligence sees a sharp increase since 2016, and on the other hand, the diversity of topics also increases throughout time. This can be mostly contextualized by the occurrence of new technologies (e.g. the Internet, leading to a new topic on Internet governance), but also by the granular development of existing technologies.
As for the third research question, namely how similar papers can be detected, we further calculated similarity scores between pairs of articles and compared the scores for a selection of pairs to the scores produced by humans. Consequently, it was explored whether the similarity scores produced by the machine coincide with the expert assessment. A correlation test revealed a high correlation between the orders produced by the machines and by the human. The highest agreement levels were found for the papers with the highest similarity scores. The similarity predicted by the machine did, however, also sometimes deviate from the expert assessment, although there will undoubtedly also be disagreement between human experts when assessing the similarity of documents.
The results do not only provide insight into how a legal research within a broader theme has evolved over time, they also provide support for the development of LDA tools that assist in structuring large document collections and in finding relevant papers. To aid researchers in either exploring or keeping up with such vast and complex research themes explored in parts of legal scholarship which do not necessarily overlap or interact, it is necessary to consider how a method such as that presented in this paper (topic modeling) can be used to further visualize this body of legal research. To this extent, we are currently working on a dashboard which we aim to make available to scholars (from any discipline) with an interest in exploring the evolution of legal literature on AI, or particular topics within it. Such a dashboard would make it easier, on the one hand, for legal researchers to get the bigger picture of all the fields of law / journals / scholars tackling topics of interest relating to AI, and on the other hand for researchers from other disciplines (e.g. computer science) to have a bird’s eye view on the vast legal literature which might have a direct impact on their research. Such publicly available resources could even be a new way to stimulate more awareness of legal and ethical implications of technology on society, and be of interest to civil society as well.
Of course, this paper does not go without limitations. First, the corpus is not necessarily representative of all legal articles or legal publications in general. Although HeinOnline papers do presumably constitute a significant part of journal articles in law, there are other publishers and repositories that contain a number of publications that may composed substantively diferent than the publications in HeinOnline. Second, another limitation regarding the corpus concerns the initial selection. We selected journal articles that included the keywords ‘artificial intelligence’. Had we selected diferent keywords, the corpus might have looked diferently. Additionally, diferent topics can be expected when running additional topic models for a subset of the corpus. Furthermore, the analyses that explored the increase or decrease of publications over time used the three most dominant topics to determine whether topics have become less or more relevant. The results might be diferent if also less relevant topics are taken into consideration, although such analyses would be empirically dificult to conduct, as it would require a measure to determine when certain topic weights are deemed insuficiently relevant.
We are currently exploring the possibility of applying and comparing diferent topic modeling algorithms and incorporate the latest NLP representation models (namely word embeddings either using word2vec or BERT). Moreover, we want to apply our methodology to a diferent legal collection and identify whether our findings can be confirmed in a diferent corpus. Finally, after fully validating our approach, we are planning to release it as an open source website where people can explore and visualize our ifndings in an intuitive way.
A