Thanks to the recent developments in distributed representation learning and the large amounts of published and digitized legal texts, computational linguistic analysis of legal language becomes possible and eficient. However, most of these open language resources and shared tasks are in English. For the languages that have little open legal texts like Japanese, a word embedding model trained on the specific language usages is accompanied by the concern of less accuracy and representativeness. Based on the observation that legal language shares a modest common vocabulary with general language, we examined the validity of using the pretrained general word embedding model for processing legal texts by an intrinsic evaluation constructed on pairs of synonyms and related terms which were extracted from a legal term dictionary. We first investigated the settings of hyperparameters of the embedding models trained on legal texts. Then we compared the performances of our domain-specific models with general models. The pre-trained Wikipedia model conducted a better performance than domain-specific models on detecting semantic relations. This model also showed a higher compatibility with legal texts than the general model trained on newspaper articles. Although researchers tend to indicate the importance of domain-specific representation models, a general model can still be an alternative solution when there is little language resource.
Owing to the emergence of Word2Vec [
Resources of general language usages are abundant in major
languages. However, when processing texts in specialised domains,
vocabularies of these domains can be very diferent from general
language. Besides those so-called “technical terms” appeared in
every specialised domain, there are also words called “sub-technical
terms” that “activate a specialised meaning in the legal field,
being frequently used as general words in everyday language” [
Unfortunately, in comparison with English, there are less open
source data of legal texts in Japanese. Either the documents are
not in machine-readable data format, or they are not even open to
public. As estimated in [
Therefore, in this paper, we examine whether general embedding models, specifically, a Japanese word embedding model pretrained on Japanese Wikipedia and a model trained on newspaper articles, can be used when processing legal texts. We start with constructing a similarity and relatedness task as an intrinsic evaluation of trained embedding models. Pairs of synonyms and related terms are extracted from a Japanese legal term dictionary. We train domain-specific embedding models on two legal text datasets with diferent settings of hyperparameters and investigate the best configurations. The comparison of the general embedding models and the domain-specific models are then conducted by the intrinsic evaluation.
Although the performance of a model mostly depends on downstream tasks, we believe that it is also important for researchers to have an awareness of the distributed representations inside of the embedding models when trying to use them to achieve better scores in specific tasks and to solve the real world problems. 2
NLP tasks related to legal issues, including legal information retrieval, document classification, question answering methods and so on, have been increasingly attracting attention from both computational linguists and legal professionals.
To improve the performances of theses tasks with the
assistance of semantic analysis, there were two word embedding
models specifically trained on legal texts. One was the pre-trained model
built in a Python library called LexNLP [
COLIEE (the legal question answering Competition on Legal
Information Extraction/Entailment) [
The evaluation of the word embeddings trained from diferent
textual resources has been conducting in the biomedical domain.
Roberts [
In general, related work tends to indicate the importance of domain-specific distributed representation models for processing specialised texts. 3
Our dataset consists of three corpus, a dictionary of legal terms (hereinafter, referred to as dictionary), “the fact of the crime” parts of the judgements obtained from Westlaw Japan 4 judicial precedent corpus (referred to as judgements), and newspaper articles contained in Mainichi Newspaper Corpus (referred to as newspaper). Basic statistics of our corpus are given in Table 1. Detailed descriptions of each corpus are given below.
Dictionary. The technical term dictionary adopted in this work was Yuhikaku Legal Term Dictionary (4th edition). The dictionary consists of 13,812 entry words with the definitions written by experts and carefully edited. We simply referred the word “legal term” (or “term”) to the entry words that were recorded in the dictionary instead of getting involved in the sophisticated discussion about the meaning of the word. In the dictionary, a synonym of term t is 1https://archive.org/details/Law2Vec. 2http://www.luxli.lu/university-of-athens/. 3https://code.google.com/archive/p/word2vec. 4https://www.westlawjapan.com/. given when t has no definition and is labeled with a See tag, while a related term of t is labeled with a See Also tag when the experts thought more information was needed
Judgements. We obtained 2,306 judgements passed on criminal cases in district courts nationwide from 2008 to 2017. Legal English is known as legalese because of its tedious and puzzling language usage. Legal Japanese also shared these problems. Therefore, in order to conduct a moderate comparison with newspaper articles in the aspect of contents and document lengths, we extracted “the fact of the crime” part from each judgement.
Newspaper. When a case happened, it is often reported as an article in the social section of the newpaper. Additionally, the language usage in a newspaper article can be considered as a general usage, or at least less specialized than legalese used in legal texts. We obtained all the articles from a one-year (2015) corpus. 1748 legal terms were observed in these articles. 4
Before processing, we applied Japanese morphological analyzer Chasen5 to split the sentences into words and remove signals and numbers. The similarity measured between two vectors in this paper were all cosine similarity.
The examining procedure was in two steps. First, we built a term pair inventory for performance evaluation. Term pairs were separated into synonym pairs and related pairs. Domain-specific models were then trained with hyper parameter tuning on this inventory. Second, we focused on the common term pairs existed in both general models and domain-specific models. The performances of the models were examined on both synonym detection and related term detection. 4.1
We extracted 1440 pairs of synonyms and 6641 pairs of related terms by exploiting the indicative tags provided in the dictionary. These pairs constructed the gold standards of synonym detection task and related term detection task for evaluating each model’s ability of catching semantic relations between terms.
We evaluated the performances of models by counting how many semantic relations were correctly caught by each model. Specifically, we first obtained top n most similar words of term t from the model. n was set to {1, 5, 10}. If the synonym or the related term was in these most similar words, we treated the trial as a correct one. The performance was represented by accuracy as the ratio of correctly predicted pairs to all synonym or related term pairs. 5version: 0.996, neologd 102.
The Validity of General Embedding Models for Processing Japanese Legal Texts
We applied pre-trained Wikipedia Entity Vectors as our general word embedding model 6. It is a 300-dimension Skip-Gram Negative Sampling (SGNS) model. With the same training configuration of it, we trained another general model on newspaper articles for the comparison within general models. We then trained our domainspecific models on the dictionary and the judgements, respectively and together. The size of the source data and vocabularies are given in Table 2.
The performance of word embedding models can be improved
by hyperparameter tuning. Since the efects of diferent
configurations can be diverse, we investigated hyperparameter settings as
in [
The best accuracy scores of models on the two tasks under diferent configurations are shown in Table 4, 5. The models trained on judgements failed in detecting both synonym and relatedness relations. The best accuracy of those judgement models was 0 (0.0%) 6https://github.com/singletongue/WikiEntVec. Wikipedia data until 2018.10.01. for synonym pairs, and 19 (0.3%) for related term pairs. In both tasks, additional legal texts (i.e., judgements) did not improve the performance of our domain-specific models, which indicated that our legal text dataset is biased to the dictionary dataset and the more data does not always lead to the better performance.
The default training configuration of gensim is {dimension =
100, window size = 5, min.count = 5, negative sample = 5}. The
selected configuration after a hyperparameter tuning on an English
domain-specific model training [
The most suitable configuration for the models trained on the dictionary across the variation of top_n was {dimension = 300, window size = 15, min.count = 3, negative sample = 10}. It is similar to the configuration of the Wikipedia model which is {dimension = 300, window size = 10, min.count = 3, negative sample = 10}.
We selected the same values of parameters as the Wikipedia model as the training configuration of our domain-specific model with which the general models would be compared on the next stage.
As shown in Table 4, 5, the Wikipedia model achieved higher performances on detecting semantic relations of legal terms, even those relations were obtained from the legal domain. This result can be due to the absence of low frequency terms in the dictionary corpus. Therefore, we further conducted two detection tasks on the common pairs among the domain-specific model, the Wikipedia model and the newspaper model. There were 18 common synonym pairs and 465 common related term pairs. Results of the experiment are shown in Table 6, 7.
The Wikipedia model achieved the best accuracy score among three models, while the same general embedding model, the newspaper model, was the worst. The performance diference between the Wikipedia model and the newspaper model also confirmed that the performance of general models are efected by the diversity of general language resources. The similar results of the examination on the common term pairs to the examination on all term pairs indicated that the Wikipedia model is superior to the domain-specific dictionary model for catching the intrinsic semantic relations of legal terms. 6
Since the usefulness of an embedding model mostly depends on the downstream tasks, we don’t argue that which embedding model is better or worse for legal NLP tasks. The purpose of this research is to investigate whether a general corpus could be used when the training on the specific domain is not practicable. The word embedding model built on Wikipedia showed a considerable performance on the intrinsic evaluation. The legal domain is diferent from other specialised domains in the aspect of the ratio of overlapping words with general language. This characteristic is helpful when there are not enough domain-specific language resources. In this paper, we provided some evidence that domain-specific word embedding models are not always outperform general models and not all the domain-specific texts are useful when constructing the semantic relations among technical terms. The using of general word embedding models, especially the models trained on a balanced large-scale corpus, therefore can be considered as an alternative way to processing those domain-specific texts.
The authors would like to thank YUHIKAKU Publishing Co., Ltd. for providing the legal dictionary dataset. We are also grateful to the reviewers for their valuable comments and suggestions.