The combination of Natural Language Processing and Articial Intelligence for the eld of Law is a growing area, with the potential of radically changing the daily routine of legal professionals. The amount of text generated by those professionals is outstanding, and to this point, still unexplored by Computer Science. One of the most acclaimed research eld covering both knowledge areas is Legal Prediction, in which intelligent systems try to predict speci c judicial characteristics, such as the judicial outcome or the judicial class or a given case. This research intends to create a classi er to predict judicial outcomes in the Brazilian legal system. At rst, we developed a text crawler to retrieve judicial outcomes from the o cial Brazilian electronic legal systems. Afterward, a few judicial subjects were selected, and some of their features were extracted. Later, a set of di erent classi ers was applied to predict the legal considering these textual features.
1.1
The combination of Natural Language Processing and Arti cial Computer Science has been revolutionizing many di erent elds of expertise. Sub elds of Computer Science, like Natural Language Processing (also known as NLP), have steadily improved a myriad of professional and scienti c activities. NLP helps researchers to understand how computers can process and analyze large amounts of natural language data and their meanings. Even simple NLP mechanisms, such as dictionaries and word counts, can o er interesting underlying facts that cannot always be noticeable without e ective processing.
Law is one of the knowledge areas that are the most dependent on text data. Millions of legislation workpapers, court decisions, and appeals are produced daily, and many di erent job specializations, such as lawyers, judges, defendants, and plainti s, have various necessities that could be supplied by intelligent systems.
Over the last years, researchers have been dedicated to predicting judicial case outcomes using NLP application software and Machine Learning methods over those textual cases. See Section 1.3 below. However, no research with this intention has been done in Brazilian Portuguese for Brazilian courts, as of 2019.
Branting et al. [
Hyman et al. [
As a primary objective, this research intends to develop a framework to predict judicial outcomes in the Brazilian state of S~ao Paulo Justice Court. The S~ao Paulo Justice Court is the most signi cant legal court on the planet, considering the number of cases per year. We believe that designing a predicting model that o ers consistent results for this judicial court this model could be transferred, after ne-tuning, to any other court. Firstly, we will develop a text crawler to retrieve data from the legal outcomes. To pre-process these data, we combine NLP tools to extract characteristics from the text, selecting what is believed to be the primary information that can lead to valid predictions. After this step, we will insert these pre-processed data into machine learning frameworks. Finally, we evaluate all the methods and their respective results against the real judicial outcomes of the court. 1.3
Recent advancements have signi cantly improved the state-of-the-art in the eld
of legal prediction. In the most in uential work, Aletras et al. [
In another in uential research, Sulea et al. [
Recent researches have used machine learning successfully to improve Law
decisions. Gokhale and Fasli [
Not directly related to the scope of legal prediction used in this paper,
some recent research has also been conducted in Brazilian Portuguese. Aires et
al. [
Liu and Chen [
A corpus of judicial sentences, along with their outcomes, was collected from eSAJ, the electronic system of the S~ao Paulo Justice Court (TJSP). To restrict the number of documents retrieved, a few previously de ned judicial subjects were selected, which are: second-degree murder (in Brazilian Portuguese, homic dio simples ), and active corruption (in Brazilian Portuguese, corrupc~ao ativa).
Only subjects with very well de ned outcomes will be selected. As well de ned outcomes, we intend to choose those judicial outcomes with the condemnation or absolution of the defendant. Many di erent legal subjects, such as divorce papers, do not have explicit terms for condemnation or absolution. Therefore, it is of utmost importance to nd those judicial subjects with clear and well-established results. We also applied numerical labels to features as condemnations and to the absolutions. These binary labels are used to develop a mathematical/statistical model that one may predict the conclusion of the cases mentioned above based on their textual structure. 2.2
We have implemented a web text crawler to retrieve data from eSAJ, using Python. As the user can select from many di erent elds to exhibit the judicial opinions, such as classes, subjects, judges, and processes numbers, the crawler must be able to choose from those di erent query choices to compose a raw text le. The following elds were collected: judicial class, judicial subject, judge, county, release date, and full text of the judicial sentence.
In addition to the data collected from the text, additional elds were derived from the information collected: gender of the judge, and a boolean eld indicating whether the county was the capital city of S~ao Paulo, or a state city.
To classify each of the judicial outcomes, we have created the binary labels for condemnation (+1) and absolution (-1). We have also used the professional guidance of Brazilian lawyers, with the purpose of better understanding the texts. The language adopted in the eld of Law worldwide might be notoriously obscure for a layman.
The data collection poses a compelling challenge, as the amount of data displayed on the website of the TJSP is indeed substantial. As of June 1st, 2018, a simple search for the Brazilian Portuguese correspondent of rape (estupro), for example, returns 5,658 hits. Another search for the Brazilian Portuguese term for drug (droga) returns 138,956 hits. Each hit is a judicial opinion of its own, containing many sentences and text topics. As each judicial class under the Brazilian law system contains di erent text topics, e.g. the topics in a text from the class divorce papers (in Brazilian Portuguese, documentos de divorcio) di er substantially from texts from the class release permits (in Brazilian Portuguese, alvara de soltura).
Table 1 illustrates the number of documents retrieved from the eSAJ system. The data retrieved was pre-processed to remove unnecessary information. We began by tokenizing the text and eliminating stopwords. For this task, we used the Natural Language Processing Toolkit in Python, called NTLK. NLTK can deal with di erent languages other than English, such as Brazilian Portuguese, which makes this framework a strong candidate for this research. Unusual characters, such as hyphens or parentheses, were also removed. After those steps, we have proceeded with the stemming of the resulting text. For the testing rounds, we used a 10-fold cross validation. 2.4
In this step, we transformed the data into a mathematical sequence that can be passed through machine learning algorithms. We used TFIDF (term frequency{inverse document frequency) to transform each of the sentences into numbers that are processed through various machine learning (ML) algorithms. 3 3.1
We have already tested the two di erent datasets with various ML methods. The methods selected were: Logistic Regression (LR), Linear Discriminant Analysis (LDA), Gaussian Nave Bayes (GNB), K-Neighbor (KN), Support Vector Machines (SVM) and, Regression Trees (RT). All the tests were initially run by splitting the training set with 75% of the whole data set, and the test set with the remaing 25% .
For the rst data set, the homicides legal texts, we found the calibration plot in Fig. 1. In calibration plots using Platt Scaling, the closest to the perfect calibrated curve, the better. Therefore, from this gure, we can infer that Regression Trees show better results. Support Vector Machines presented the lowest performance, predicting values very di erently to those that would be expected based on the reviewed literature.
At last, we run a k-fold cross validation, with 10 epochs, for each of the algorithms. The results are shown in Table 2 and Fig. 2. LDA and Linear Regression, in these tests, were the best choices among all the algorithms. For the second data set, with the corruption legal texts, we found the calibration plot shown in Fig. 3. In Platt Scaling, the closest to the perfect calibrated curve, the better. As the previous results with the Homicides data set, we can infer that Regression Trees are the best choice. K-Neighbors showed the lowest performance, predicting values very di erently to those that would be expected. All the tests were initially run by splitting the training set considering 75% for the training set and 25% for the test set.
Those results show that, at least for the two databases being evaluated, Regression Trees are the best choice for predicting the outcomes of legal texts. In both data sets, it showed to the the best choice, or being among the best choices available. 4
After the tests done, we can infer that Regression Trees are the best method to predict results in both data sets being analyzed. Even though the other algorithms showed varying results. SVM, as an example, showed a good performance in the corruption database, but the lowest value in the homicides database. Regression Trees have always kept good predicting outcomes. Those results match
Fig. 4. Candlesticks for the algorithms with 10 k-fold
other results found by other researches in the legal area, in many di erent
countries, such as the one conducted by Kastellec [
Other researchers also used the same method, such as Rios-Figueroa [
Those researches show that, even though legal systems are extremely di erent around the world and throughout di erent languages and countries, such as Brazil, USA, Italy and Zimbabwe, they do have similar characteristics that can be e ectively measured by the correct algorithms.
As future steps, we plan to adopt di erent methods of converting word embeddings into whole texts, so that we can also utilize methods, such as neural networks, and, eventually, compare these with the ones mentioned in this work.