This paper presents a novel pipeline for transforming flat-labeled text collections into a hierarchical structure, which involves leveraging simple yet efective similarity methods that account for both lexical and semantic criteria to associate labels from disparate sources. Our approach employs a custom similarity measure, the Reinforced Edit Similarity, to identify probable correspondences based on lexical similarities. A subsequent semantic alignment and validation phase is then performed using an automatic classification mechanism. Preliminary results attest to the efectiveness of our proposal. These results are obtained from the research group of the University of Torino in the NGUPP project.
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License more accurate content-based classification. We focus Attribution 4.0 International (CC BY 4.0).
Legal informatics concerns the automatic processing of information to support legal activities. One of the most relevant issues encountered in the legal informatics field, dard criteria for the classification and analysis of legal documents [1, 2, 3]. In a juridical system, courts typically specialize in issuing specific types of judgments based courts focus their attention on a set of specific subjects (e.g. civil criminal, family rights, labour, immigration...), for which a deeper and more granular organization of the judgements’ labels is noted. This implies, on the one hand, the dificulty of having a national structure of such ing of labels on more in-depth topics. As a consequence, such structures used by courts that are close to each other, Proceedings of the Sixth Workshop on Automated Semantic Analysis of on an Italian case study, describing in detail the type of data available, the technologies used, and the models for automatic classification.
Our contribution focuses on two fundamental parts: the search for a criterion for transferring the labels of a non-hierarchical structure within the labels of an existing hierarchical structure, and the enrichment of the data contained in the labels of this hierarchical structure. In the following of the paper, Section 2 introduces the background with related works, the definitions and the data used to perform the classification and the alignment tasks. Section 3 describes the method, while early results are detailed in Section 4. Section 5 concludes the paper.
Related work. The task of taxonomy alignment is typically concerned with aligning multiple taxonomies that share similar or related concepts. Although our context difers from this scenario, as we have a flat set of labels on one side and a taxonomy on the other, we present relevant research on taxonomy alignment as it represents the most analogous context in the literature.
This task has gained significant attention in recent years due to its applications in knowledge integration, data integration, and semantic interoperability. Several approaches have been proposed to tackle this problem, including ontology matching, hierarchical clustering, and rule-based methods. Ontology matching is a popular approach that leverages semantic similarity measures to align taxonomies [4]. Hierarchical clustering methods group similar nodes from diferent taxonomies [ 5], while rule-based methods use expert knowledge to map concepts across taxonomies [6]. Recent studies have explored the use of machine learning techniques, such as deep learning, to improve the accuracy of taxonomy alignment [7].
As we previously mentioned, our proposed method is novel in that it addresses a slightly diferent scenario. Specifically, we consider two collections of documents that are labeled with distinct sets of labels, where only one of the sets is organized in a taxonomy. This scenario is particularly noteworthy for several reasons, such as providing more structure to documents with flat labels or augmenting a coherent text collection with additional documents that lack extensive labeling. to a case, and can be identified by several different parameters. One identifier is obtained by combining the judgement code and the year of publication. The former indicates a sequential code given by the Court when it is published, the latter indicates the year of publication. • Subject: (i.e. Materia) indicates the macrocategory to which a given judgement belongs, as well as the section of the court that issued it. The judgements that are dealt with in this paper are related to the subject area of Labour Law. Other examples of subject areas are Civil Law (i.e.
Civil Law), Tax (i.e. Tributaria), etc. • Label: (i.e. Voce) indicates a categorisation label of a judgement. These labels respond to the individual court’s way of conceiving and categorising judgements. Specifically, labels can be presented in a taxonomic form, i.e. organised into labels and sub-labels, or they can be unstructured. Examples of labels are “risarcimento danni”, “invalidità civile”, “retribuzione”, etc. • NGR: an acronym standing for ’Numero Ruolo Generale’, it is an identifier corresponding to a numerical sequence specific to a particular court and assigned by that court to a specific case. It is used to link all the acts and documents relating to a specific case in a single folder.
Following the recovery of judgements from two diferent sources (Turin Court, and a public online archive), we applied diferent preprocessing techniques to model the data, rendering them useful to the study. This section describes the results of a classification test used to compare the non-aligned labels results with the ones obtained in this study, to demonstrate the presence of improvements following the alignment between labels. Finally, we define the pipeline of actions executed to obtain the alignment of two taxonomies of labels coming from two diferent sources. Figures 1 and 2 show two of the main steps of the alignment pipeline with the corresponding benefit in term of enrichment the Leggi d’Italia taxonomy with the judgments of Turin corpus.
• Judgement: (i.e. Sentenza, in Italian) is the judi
cial decision given by a judge or court, in relation Definitions. This section introduces some terms re- Sources. The judgements used in this study derive lated to the legal domain, as well as keywords that require from two diferent sources: a set of 27,477 judgements careful definition and disambiguation to access the mean- issued by the Court of Turin, relating to the labour secing of the technical parts of this article. tion and a set of 21,562 judgements extracted from Leggi d’Italia [8], an online archive which is a point of reference in legal matters in Italy. These led to a comparison of documents from diferent courts on the same semantic level in order to identify similar patterns concerning the way judgements are drafted. The data have been standardized, as the judgements obtained have diferent digital formats, such as ‘pdf’, ‘docx’, ‘doc’, ‘docm’ and ‘html’. Of all the Turin judgments, only a subset of 4,804 are labelled. This finding is very significant, as subsequent work on classification will be influenced by the reduced volume and will form the basis for a first attempt at transferring taxonomies, as will be discussed below.
Figures 3 and 4 show clearly that the judgements’ labels from the two sources are structured diferently. In fact, while Turin’s labels are organized in a linear structure without a precise hierarchy, Leggi d’Italia’s labels have a taxonomic relationship, structured in concepts and sub-concepts. The “/” character shows the end of a sub-tree and the start of a new sub-tree in the hierarchy.
Secondly, looking at the distributions, it is immediately apparent that the two sets of judgments are highly unbalanced, with inevitable consequences in terms of automatic classification. A small portion of the Leggi d’Italia’s labels tree is shown in figure 5. It can be observed how the labels are layered in sub-trees.
Data. An important phase preceding the work of alignment and automatic classification of the judgements concerns the retrieval of the data, as well as the segmentation and organization of the textual content of the processed documents. As anticipated in the previous paragraph, the set of judgements of Leggi d’Italia, henceforth called “corpus-LI”, was retrieved as a result of a scraping work conducted on the Leggi d’Italia web platform [8] using the python library scrapse [9].
The suite allows both retrievals of digital documents of judgements and extraction of the content in JSON format. A similar work, for uniformity, was conducted • sent code: identification code of the judgment • sent year: year of publication • nrg code: general role code • nrg year: year associated with the general role code. The nrg code and nrg year pair identifies a specific case within a court.
JSON corpus:. This second representation includes all the content information of a certain judgment. The most relevant ones are: • oggetto: (i.e object) in the form of a short sentence, it represents the topic addressed in the case from which the judgment is issued. Typically it is very informative about whether a judgment belongs to a certain category, but it is not suficient. • conclusioni: (i.e conclusions) Some indication of the conclusions of the trial for the parties in the case. • fatti: (i.e facts) represents the central body of the judgment in which the facts of the case are discussed. • decisione: (i.e decision) the decision made by the judge. In some cases, fact and decision are merged together. • P.Q.M: (i.e. Per Questi Motivi) the final verdict.
A third representation, for convenience, in unified format was derived by merging the previous two. 3.2. Preprocessing on the set of judgements of the court of Turin, which for simplicity we will call “corpus Turin”. The textual content was extracted and segmented tracing the same representation obtained on the corpus-LI. Finally, we obtain the following two JSON representations: JSON metadata and JSON corpus.
JSON metadata: This first representation in JSON format collects all the metadata found among the textual content of a judgment. Such information can be useful not only for data visualization purposes, such as knowing how many judgements were issued by a certain court rather than another but also for automatic classification and alignment purposes. Among the most significant pieces of information, the following metadata was collected: dings generation: “Continuous Bag of Words” (CBOW) and “Skip-Gram’(SG)” [16]. For the learning process, we considered the first one, CBOW, which implementation is visible in the python library: gensim.models.Doc2Vec [17].
The model, after a preprocessing step, specifically required for this implementation of the algorithm, was trained for 30 epochs with the following hyperparameters: vector_size = 300, negative=5, hs=0,min_count=2,sample=0, alpha=0.025, min_alpha=0.001. • Italian-Legal_bert: Italian-Legal_bert [18] is a version of a pretrained BERT-BASED [19] model (ITALIAN XXL BERT [20]) trained on italian legal texts. The embeddings of this model are obtained running an additional round of training for 4 epochs on a 3,7GB preprocessed text from the National Jurisprudential Archive using the Huggingface PyTorch-Transformers library [21].
• lemmatization Sensitive data such as first and last names were removed at the preprocessing stage to ensure the least possible dirty data to be given as input to machine learning models. To identify these entities, we retrieved a dataset of proper names found on the Agenzia per l’Italia Digitale web portal [10]. All preprocessing was done using the python Spacy library [11], however, the list of stopwords for the Italian language was enlarged using external resources [12].
work focused on a preliminary classification task. This preliminary task was exploratory in order to better understand the most appropriate vector space modeling patterns and representations on the data at our disposal.
Considering the imbalance of the data, classification tests were conducted on a limited number of judgments.
Specifically, two corpora were created which in the fol- Models. Our classification work focused more on data lowing we will call “corpus_11_labels_torino”, constructed representation than on the use of neural models and fineusing some the 15 labels in figure 3 for a total of 318 judg- tuning of networks. A first experiment has seen the use ments and a “corpus_10_labels_LI” for a total of 7,308 of a multiclass SVM [22] as a baseline model. Assuming judgments from Leggi d’Italia, whose details will be dis- nonlinearly separable data, we trained the SVM model cussed in the section 4. The reason why these 2 datasets using an ”rbf” kernel-trick [23]. In the second order, have a diferent sizes is to be found both from the un- considering the dimensions of the datasets, we conducted balancy of the distribution of the labels, we can see in some tests using a Logistic Regression [24] model with a ifgures 4 and 3, and from the results of the alignment ”lbfgs” solver. In presence of sparse and poor data, these process. Various vector space modeling techniques were models tend to show the same behaviour. Furthermore, used to create the datasets. Starting from these repre- we considered a Random Forest classifier [ 25] with max sentations, several classification tests were conducted 2,000 trees, which, instead, results more eficiently on employing some machine learning models. From the datasets with a limited number of features. Finally, the extracted judgments in JSON format, textual contents same tests were repeated running an Ensemble Learning related to the following fields (references in 2) were re- task with a simple Voting classifier [ 26] using all the trieved for the creation of the datasets: “subject”, “fact”, previous models. “decision”, “conclusion”. The information contained in the “P.Q.M” was discarded, as these are very recurring 3.4. Pipeline phrases, frequently used formulas in all subjects, as such, negatively afect the classification. Similar considerations Considering the structure of the data and the small volwill be taken up in section 5. Starting from these fields, ume of judgments, we initially attempted taxonomic 4 diferent datasets were created for the “corpus_11_la- alignment between Leggi d’Italia’s labels and those of the bels_torino”. At the end of the preprocessing pipeline Turin court. To proceed critically, we defined a pipeline on the “corpus_11_labels_torino”, the use of TF [13] and that considers the transfer process in steps, articulated TF-IDF [14] led us to define two sparse matrices of shape in: label comparison, semantic similarity and validation 10,955 x 318. To have a recent comparison regarding the with classification . state of the art on the embeddings representation, the remaining 4 datasets were created using the following 3.4.1. Labels comparison resources:
In this first stage we considered the labels’ alignment • Doc2Vec: Doc2Vec [15] is an unsupervised neu- exclusively from a lexical point of view. From a first sural network model that learns fixed-length fea- perficial reading, it is easy to find some similarities, by ture vectors for representing textual data. The looking at the 2 lists of labels in figure 3 and 4. Taking network architecture, like for word2vec [16], pro- the following labels as examples: “Labour and social sevides two diferent algorithms for the embed- curity (Disputes relating to) Legal fees” and “Legal fees”, respectively from Leggi d’Italia and from Turin. Without looking at the content of the judgments, it would seem that the two voices speak about very close topics, however, only through a deeper analysis can this observation be confirmed or refuted. The label comparison phase consisted of searching for criteria of approximation between the labels of the two sets of judgments, hence leading us to define the Reinforced Edit Similarity.
Reinforced Edit Similarity. The alignment criterion Figure 6: Comparison of the similarity scores between the used is a combination of edit distance [27] and cosine counter vectorizer’s cosine similarity and the reinforced edit similarity [28]. Since, Leggi d’Italia presents a taxonomy similarity. We perform a matching between two labels: ”social articulated in a tree structure, we decided to distribute iszeactuiorintyp/rcoivcielsdsiwsaeblielimtym”aantidze”cthiveilwinovrdalsi,dsso”.tBheefworoerdth”einvveacltidoir”it in N levels of labels, with N=3 the maximum depth. (i.e invalids) is trasformed into ”invalidare” (i.e invalidate) and Starting from the leaves, and going up to the root, we cal- ”civili” (i.e civils) into ”civile” (i.e civil). If we apply the reiforced culated the score for each entry pair of Turin and Leggi edit similarity before the lemmatization phase, we obtain a d’Italia. It should be considered that the labels were pre- cosine similarity score of 70,71%, because the words ”invalidi” processed, not only to facilitate better approximation and ”invalidità” (i.e disability) are trasformed into the same but also because they had punctuation symbols, special word, i.e ”invalidità”. characters, and many. In a first step, approximation was performed by tokenizing the labels, then applying co- Label Label similarity similarity similarity sine similarity on the vectorized representation created Torino Leggi d’Italia level 1 level 2 level 3 with counter vectorizer [29]. Later, we abandoned this Individual dismissal EDMismPLisOsYalMENT (RELATIONSHIP)/ 0.000 0.707 0.000 criterion as it did not take into account the diferences damage compensation CNIoVnI-LpeAcNuDniaCrRyIdMaImNaAgLeDAMAGES/ 0.353 0.499 0.000 itnagl.emFmigautrieza6tiosnhoowfsthheowworadssiwmiitlharrietyspseccotrteo othfe3ir5.P3O5%S jurisdiction jJaAuudDrrmiiMssiddnIiiNcicsttItiiSrooaTnntR/ibvdAeeiTvtcwIisVoeiuoEernnJtUsoofSrdTiInCaEr/y and 0.000 1.00 0.353 was derived from the two labels ”social security/civil laigmeintactyion RAGETEINRCEMYE(CNOTN/LTiRmAitCaTti)on 00..070007 01..000000 00..000000 disability” and ”civil invalids” (i.e., social security/Legal disability, legal invalids), against reinforced edit similar- Table 1 ity score of 70.71%.To facilitate a better approximation, lRaebienlfso. rTcehde eladsitt tshimreielacroitlyu mscnorsehsoowntahesuscboseretsoofnthtehetuthrirne’es once we switched to the vector representation using the levels of labels of the Leggi d’Italia’s hierarchy. CountVectorizer module, we calculated the edit distance between each pair of words, with a threshold ≤ 2. Pairs that do not have a distance greater than the threshold were transformed in such a way as to unify them (make Semantic similarity. In this phase we recovered an them identical). In this way, a subsequent application of even number of judgements from both the labels sources. cosine similarity will present a higher score, rewarding Some of the labels we used at this point of the pipeline, in fact, those labels that are lexicographically close. At resulting from the labels comparison phase, are visible the end of the alignment process, for each pair of Turin in table 1. We then proceeded to transform them in a and Leggi d’Italia labels, we considered the one with the Doc2vec form, obtaining then two clusters composed by highest cosine similarity score on the various levels. Ta- judgements of the same cardinality. To value the closeble 1 shows some results of the lexical similarity scores of ness between these clusters, we applied metrics such the Turin labels, evaluated on the various levels of labels as cosine similarity between the centroids vectors. The in the Leggi d’Italia taxonomy. semantic similarity score in combination with the Reinforced Edit Similarity score contributed to an overall score that allowed us to evaluate the alignment of the 3.4.2. Semantic comparison labels. Specifically, all matching of labels that returned a Embeddings representation. In this step of the semantic similarity score ≥ 70% were retrieved. Overall, pipeline we focused on the semantic aspect. Our first goal considering that some matches concerns labels that in has been to choose how to converge to a single vector both sources have limitations due to the fewer number representation. Looking at the table 3, in section results 4, of judgements, all of those that have a cardinality of less we chose to transform the judgements into embeddings than 10 judgements have been discarded, as we deemed Doc2vec. This choice has been motivated by the facts them of less importance. Table 2 shows the results of sethat on the available data we had, Doc2vec represented mantic similarity on a subset of the Turin labels, chosen the model that had the better results. from those most populated on both sources. 3.4.3. Validation with classification The last step of this pipeline implements an a posteriori validation of the quality of our alignment by performing a classification of the judgments on the labels in the table 1. In this phase we demonstrate, as shown in section 4, how the alignment did not have significantly negative impacts on the classification of judgments. From an initial set of 309 labels on the Turin corpus, only a subset of 11 labels returned a centroid similarity score ≥ 70%.
Here, the list of the candidates labels: “agency”, “social allowance”, “subordinate work”, “dismissal”, “individual dismissal”, “injunction”, “notification”, “proof”, “severance pay”, “sickness allowance” and “assistance”. At last we can confidently say that the results we obtained validate the alignment process. In particular, ”individual dismissal” and ”dismissal” are associated to the same label of the Leggi d’Italia hierarchy: “Subordinate work (Relationship of)/dismissal”, so during the classification process, these labels are considered as the same label. At the end of this final step of the pipeline we train some machine learning models using the “corpus_10_labels_LI” as training set and the “corpus_11_labels_torino” as testing set, for all these 11 labels.
sults of our experiments, after and before the alignment pipeline. All data visualized in the following tables are derived by applying a 10-fold cross-validation method on the datasets and models defined in the section 3.3.
Table 7 shows the accuracy scores evaluated on the “corpus_11_labels_torino” testing set, using all four models introduced in the section 3. As we noted, the performances of the models decreases significantly, as the number of items increases. Looking at the SVM curve, for the first 6 labels, the accuracy has a score of 95%, which decreases to a value of 80%, for a total of 11 labels. If we compare these results with the previous ones on the pre-alignment classification tests, we note that, for the ifrst 8 labels, the performance of the SVM model does not sufer a significant decrease, as instead it happens for the Logistic Regression and the Random Forest classifiers.
preprocess the judgements’ content to obtain more useful information. The latter can be added to the computation of a semantic similarity between judgements.
In this paper we explored a first approach of a transposition and alignment of a not-hierarchic structure in a well defined taxonomy, using a pipeline of diferent ap- 5.1. Improve preprocessing proaches. With the judgements that we obtained from two diferent sources and their labels, we realized a first Given a preliminary analysis of the dataset, integrate step in which we defined lexical similarity on the labels, to the processing pipeline a possible correction and/or while testing a new metric of lexical proximity result- elimination of words that contain spelling errors. Given ing from the combination of existing techniques. Hence, an initial analysis of word occurrences, it was found that going down to the semantic level, we applied cosine simi- those with minimal frequency contained spelling errors. larity by calculating the similarity of the centroids in the By extracting ten words at random, at least half have groups of judgments we identified as similar in the first spelling errors. step. After these two steps, as a check on the validity of our new found method, we trained some machine learn- 5.2. Keywords extraction ing models, then evaluated the performance on the data before and after the alignment. As the final check on Further improve the preprocessing with an expectation performance did not change negatively for some models, of increase the accuracy of the transferring taxonomy we were assured that the alignment did not lead to a loss pipeline described in section 3.4, by extracting the most of information in the newly constructed groups of judg- significant keywords of each labels. The goal is to remove ments. Indeed, the processing of the data and the various the most frequent words that have an even distribution phases of the pipeline we therefore described can be in across all labels, thus having a low significant impact, the future further analyzed with new metrics and calcu- and identify those that best identify each label. As can lus approaches or with a more targeted study on how to be see in figure 9, the word “operator” has a higher frequency under the label “agency” than under other labels, [5] P. Clerkin, P. Cunningham, C. Hayes, Ontology disnominating itself as a potential keyword. covery for the semantic web using hierarchical clustering, Technical Report, Trinity College Dublin,
Department of Computer Science, 2002. [6] S. Fernández, J. R. Velasco, M. A. López-Carmona,
A fuzzy rule-based system for ontology mapping, in: Principles of Practice in Multi-Agent Systems: 12th International Conference, PRIMA 2009, Nagoya, Japan, December 14-16, 2009. Proceedings 12, Springer, 2009, pp. 500–507. [7] J. Chen, E. Jiménez-Ruiz, I. Horrocks, D. Antonyrajah, A. Hadian, J. Lee, Augmenting ontology alignment by semantic embedding and distant supervision, in: The Semantic Web: 18th International Conference, ESWC 2021, Virtual Event, June 6–10, 2021, Proceedings 18, Springer, 2021, pp. 392–408. [8] Leggi d’Italia p.a., accessed 27.04.2023. URL: https:
//pa.leggiditalia.it/#mode=home,__m=site. [9] scrapse, accessed 27.04.2023. URL: https://pypi.org/ Figure 9: Average of ten common words in the six labels in project/scrapse/. the graph. From this graph, which ofers an initial analysis, it [10] Agenzia per l’Italia digitale, accessed 27.04.2023. can be seen that the word “duty” is evenly distributed across URL: https://www.dati.gov.it/view-dataset. all labels (to be eliminated), compared to the word ”operator” [11] Industrial-strength natural language processing, ac(potential keyword). cessed 27.04.2023. URL: https://spacy.io/. [12] Stopwords italian, accessed 27.04.2023. URL: https:
//github.com/stopwords-iso/stopwords-it. [13] H. P. Luhn, The automatic creation of literature 5.3. Other similarity approaches abstracts, IBM J. Res. Dev. 2 (1958) 159–165. Use of diferent labels similarity approaches like Poly- [14] K. S. Jones, A statistical interpretation of term specifuzz [30]. This package is used to compare similarity ifcity and its application in retrieval, J. Documentabetween strings using diferent type of models to cre- tion 60 (2021) 493–502. ate n-grams on a character level. After generating the [15] Q. V. Le, T. Mikolov, Distributed representations of n-grams and applying the models on the strings’ words, sentences and documents, in: International Conit use cosine similarity to compare the generated vectors. ference on Machine Learning, 2014. [16] T. Mikolov, K. Chen, G. S. Corrado, J. Dean, Eficient estimation of word representations in vector References space, in: International Conference on Learning Representations, 2013. [1] D. M. Katz, R. Dolin, M. J. Bommarito, Legal infor- [17] Gensim topic modelling for humans, accessed matics, Cambridge University Press, 2021. 27.04.2023. URL: https://radimrehurek.com/gensim/ [2] E. Sulis, L. Humphreys, F. Vernero, I. A. Aman- models/doc2vec.html.
tea, D. Audrito, L. D. Caro, Exploiting co- [18] D. Licari, G. Comandè, ITALIAN-LEGAL-BERT: A occurrence networks for classification of implicit Pre-trained Transformer Language Model for Italinter-relationships in legal texts, Inf. Syst. 106 (2022) ian Law, in: Symeonidou et al. (Ed.), EKAW, vol101821. doi:10.1016/j.is.2021.101821. ume 3256 of CEUR Workshop Proceedings, CEUR, [3] D. Audrito, E. Sulis, L. Humphreys, L. Di Caro, Ana- Bozen-Bolzano, Italy, 2022. URL: https://ceur-ws. logical lightweight ontology of eu criminal proce- org/Vol-3256/#km4law3. dural rights in judicial cooperation, Artificial Intel- [19] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: ligence and Law (2022) 1–24. Pre-training of deep bidirectional transformers for [4] A. Giabelli, L. Malandri, F. Mercorio, M. Mezzan- language understanding, in: ACL: HLT, Vol. 1, ACL, zanica, Weta: Automatic taxonomy alignment Minnesota, 2019, pp. 4171–4186. doi:10.18653/v1/ via word embeddings, Computers in Industry 138 N19- 1423. (2022) 103626. doi:https://doi.org/10.1016/j. [20] Italian bert, accessed 27.04.2023. URL: https:// compind.2022.103626. huggingface.co/dbmdz/bert-base-italian-xxl-cased.
[21] hugging face transformers, accessed 27.04.2023.
URL: https://huggingface.co/docs/transformers/ index. [22] B. E. Boser, I. M. Guyon, V. N. Vapnik, A training algorithm for optimal margin classifiers, in: Proceedings of the fith annual workshop on Computational learning theory, 1992, pp. 144–152. [23] Support vector classification, accessed 27.04.2023.
URL: https://scikit-learn.org/stable/modules/ generated/sklearn.svm.SVC.html. [24] Logistic regression classifier, accessed 27.04.2023. URL: https://scikit-learn.org/ stable/modules/generated/sklearn.linear_model.
LogisticRegression.html. [25] L. Breiman, Random forests, Machine Learning 45
(2001) 5–32. [26] Soft voting/majority rule classifier for uniftted estimators, accessed 27.04.2023. URL: https://scikit-learn.org/stable/modules/generated/ sklearn.ensemble.VotingClassifier.html. [27] Levenshtein edit-distance, accessed 27.04.2023.
URL: https://www.nltk.org/api/nltk.metrics.
distance.html. [28] Compute cosine similarity between samples in x and y., accessed 27.04.2023. URL: https://scikit-learn.org/stable/modules/generated/ sklearn.metrics.pairwise.cosine_similarity.html. [29] Convert a collection of text documents to a matrix of token counts., accessed 27.04.2023. URL: https:// scikit-learn.org/stable/modules/generated/sklearn.
feature_extraction.text.CountVectorizer.html. [30] Polyfuzz, accessed 27.04.2023. URL: https: //maartengr.github.io/PolyFuzz/.