The Italian Senate faces the problem of clustering amendments to optimize the scheduling of parliamentary sessions. Currently, this task is carried out by Similis, an application that tackles this problem by using a traditional term-frequency technique, which leads to clustering based on wording rather than semantics. Recent advances in natural language processing have led Italian institutions to investigate the adoption of pre-trained language models (PTLMs) for text analysis. Along this line, in this paper, we propose CLAMSE, an alternative system to Similis that uses Sentence-BERT pre-trained models to generate embeddings and then groups similar amendments through hierarchical agglomerative clustering. Our preliminary evaluation shows that CLAMSE achieves comparable performance to Similis using embeddings generated by pre-trained models without fine-tuning, paving the way for applying a clustering method with advanced contextual understanding. This study contributes to enhancing the efectiveness of institutional decision-making processes through the adoption of PTLMs.
CEUR ceur-ws.org
Pre-Trained Language Models (PTLMs) are emerging as valuable allies in addressing various problems across diverse domains and represent a great opportunity for enhancing parliamentary eficiency and efectiveness. In this context, and within the framework of a collaboration between Roma Tre University and the Italian Senate, an interest in systems based on PTLMs to support parliamentary activities has taken shape. This paper investigates the problem of clustering similar amendments.
Amendments represent proposed changes to legislative texts and are a fundamental element of the legislative process. They may vary widely in terms of wording but often share similar intentions and objectives. Similar amendment proposals should be discussed simultaneously, if possible. Therefore, clustering amendments according to their similarity is an essential activity to facilitate the work of oficials and efectively organize voting sessions, while ensuring the coherence and completeness of legislative proposals. Indeed, amendments that difer by only (R. Torlone) a few words are usually proposed in large numbers by parliamentary groups that want to iflibuster, with the aim of slowing down the legislative process. Combining debates on similar amendment proposals therefore allows for the greatest possible eficiency in terms of time.
In this paper, we aim to explore the potential of PTLMs in such a crucial context. The Senate already has a tool to support this activity, called Similis, which adopts a traditional term-frequency technique. Although Similis is efective in grouping short amendments sharing many tokens, it loses efectiveness with longer amendments that adopt diferent lexicons yet preserve the same semantics.
To overcome this issue we have investigated an alternative solution that leverages a PTLM.
Our approach has been implemented in CLAMSE (Clustering Amendments with Semantic
Embeddings), an alternative system to Similis, which relies on Sentence-BERT [
The preliminary experiments we conducted yielded promising results showing the efectiveness of adopting solutions based on PTLMs in demanding processes of public administration, with the advantage of simplifying the development process by eliminating the need for building from-scratch implementations or task-specific models.
The rest of the paper is organized as follows. In Section 2 we discuss related works and the pre-existent approach to amendments clustering. In Section 3 we illustrate our PTLM solution and in Section 4 we report on its evaluation. Finally, in Section 5 we draw some conclusions.
In recent years, there has been a growing emphasis on leveraging advanced text processing
techniques to enhance institutional work globally. In the context of a collaboration between
Roma Tre University and the Senate of the Italian Republic, and more generally in the application
of artificial intelligence systems to support legislative activities, several important studies have
been carried out. A machine learning system for the classification of documents has been
developed [
A pivotal aspect of this evolution lies in the transition from traditional vectorization methods
towards more semantic-based approaches. In the domain of text clustering, Term
FrequencyInverse Document Frequency (TF-IDF) stands out as one of the most commonly employed
methods for representing textual data. However, TF-IDF fails to incorporate the positional and
contextual aspects of words within sentences. A study conducted simulations to demonstrate
that BERT consistently outperforms the TF-IDF method [
This transition towards semantic representations marks a paradigm shift from shallow representations incorporating syntactic meaning to deep context understanding, providing a more sophisticated text comprehension.
Similis. The IT department of the Italian Republic Senate has developed a solution for
clustering similar amendments, called Similis, in close collaboration with the Institute of Legal
Informatics and Judicial Systems (CNR-IGSG) [
Our semantic embeddings-based solution consists of a pipeline, which involves three phases: () preprocessing, () encoding, and () clustering.
The system takes a set of amendments that refer to a single Senate act as input. The preprocessing phase aims at cleaning the text of each amendment from special characters, thereby increasing the quality of embeddings and, consequently, improving clustering performance. The pre-processed text is then sent to the encoding block, which computes an embedding for each amendment. It is worth saying that Sentence-BERT models are implemented in a Python framework known as SentenceTransformers. This provides several PTLMs, each with specific characteristics. The encodings produced by each model serve as input for the clustering phase that implements a traditional HAC. The final solution for the problem is determined by selecting the most optimal clustering among those generated by applying HAC to each corpus of embeddings obtained in the encoding phase.
In the following, we provide a more detailed description of the three phases.
The input dataset, which is structured as a JSON Line (JSONL) file, is transformed into the standard JSON format for readability. The JSON file reports a record containing the text and the amendment number for each amendment. Also, each record is associated with a cluster identifier, which represents the ground truth for a clustering assignment. 1
The text of the amendments includes numerous tags and annotations designed for display on web browsers but lacking any semantic significance. Therefore, it was decided to remove them to enhance clustering performance. In particular, we remove all the occurrences of the HTML 1In the original JSON file, these elements correspond to the ’num_em’, ’text_emend’, and ’id_cluster’ attributes macro for the blank space (” ”), and we replace all the occurrences of HTML tags with whitespace characters.
The cleaned corpus of amendments is passed to the encoding module. Here, a range of SentenceBERT PTLMs is loaded, and each model generates an embedding for the text of each amendment. Not all of the pre-trained models produce normalized embeddings. However, since normalizing embeddings leads to improved performance, a normalization step is added by dividing each component of the vector by its norm and ensuring that the norm of the resulting vector is equal to 1 (L2 normalization). Table 1 reports the PTLMs that we have considered in our experimental activities. Since the dataset used for the experiments contains amendments with an average of 137 tokens, models with a maximum sequence length of less than 512 tokens were excluded from the study. 2
In the last phase, a clustering activity is carried out on each corpus of embeddings generated in the previous phase. In particular, we adopt a HAC approach. Roughly speaking, a HAC algorithm operates as follows. Initially, each sample in the input dataset is treated as an individual cluster. Then, the algorithm iterates, merging in each step the most suitable pair of clusters until only one cluster remains. The decision on which pair of clusters to merge is based on the cosine similarity between the embeddings representing the amendments, with complete linkage, i.e., the similarity between clusters equals the similarity between the two most dissimilar samples, one in each cluster.
The cut-of threshold for the dendrogram generated by the HAC algorithm is set to a value that maximizes the quality of the clusters. In CLAMSE, we use the silhouette score to measure clustering quality: it measures how similar an element is to members of the cluster it belongs to, compared to other members of other clusters. It ranges from −1 to +1, where a high value indicates that there is cohesion between elements of the same cluster and separation between elements of diferent clusters. 2For a complete list of the models see https://www.sbert.net/docs/pretrained_models.html.
Since we have a clustering solution for each PTLM used to generate the embeddings, we choose the clustering with the highest silhouette score as the final solution.
The evaluation of CLAMSE is carried out in terms of both its absolute performance and in comparison to the pre-existing Similis solution. In particular, we first analyze the performance of the diferent PTLMs to identify the best solution that CLAMSE could achieve. Then, we compare CLAMSE to Similis. Finally, we present the results of an experiment that shows how the preprocessing phase can influence the final clustering results.
We evaluated CLAMSE on a set of amendments, named “Act 1248”. It contains 1261 amendments that have been clustered manually (each amendment, in the original JSONL file, is annotated with a label representing the target cluster). Table 2 reports the main characteristic of this dataset. Note there is a large variation in both the size of amendments occurring in such dataset (ranging from a few to thousands of tokens) and the number of elements occurring in a cluster (ranging from a singleton to tens of amendments). This makes the task of automatic amendments clustering challenging.
The performance of Similis is reported by means of the Adjusted Rand-Index (ARI) [
ARI assesses the agreement between the clusters produced by a clustering algorithm and
the ground truth, correcting for chance agreement. AMI is another measure used to evaluate
the similarity between two clusterings of data, but it is based on mutual information. Both
ARI and AMI are used to measure the similarity between the true labels in the ground truth
and the clustering labels produced by CLAMSE, taking into account the possible presence of
random agreements. Both metrics range from 0 to 1, where a score of 1 indicates that true labels
and clustering labels are identical, and a score of 0 indicates that they are completely diferent.
ARI is more suitable when the ground truth clustering has large equal-sized clusters. AMI is
preferable when the ground truth clustering is unbalanced and there exist small clusters [
As above mentioned, silhouette is another metric for evaluating the quality of clustering. However, in our solution, it is used internally to the HAC algorithm to find the optimal number of clusters.
The performances of CLAMSE compared to Similis are reported in Table 3, which shows that CLAMSE achieves better results both for ARI and AMI.
Table 3 reports also the performance of CLAMSE without the preprocessing phase. It is worth observing how cleaning text contributes to improved performance as the semantics of the amendments can thus be better understood by the model.
model
As we discussed in Section 3.2, CLAMSE utilizes several PTLMs from the Sentence-BERT library, known as SentenceTransformers, and chooses the solution that produces the best clustering based on the silhouette score. In order to evaluate the robustness of the approach, for each Sentence-BERT PTLM we have computed the best clustering that could be obtained from the HAC in terms of ARI and AMI. Essentially, at every iteration of the HAC algorithm, we compute ARI and AMI, which need the ground truth. The highest AMI and AMI scores represent the best performance that can be achieved by the CLAMSE approach, which chooses the best clustering of each PTLM based on the silhouette scores.
Table 4 shows, sorted by descending AMI, the best performances achievable by CLAMSE with each of the models in Tables 1. Observing the results of this experiment, two important observations occur. First, notice that in many cases, the CLAMSE algorithm achieves the best results it could obtain. This demonstrates the efectiveness of the silhouette score for choosing the best clustering of each model. Secondly, it’s important to note that while several models outperform Similis, there are also many models that exhibit inferior performance. This underscores the efectiveness of our approach in model selection.
We presented CLAMSE, a system that addresses the problem of amendments clustering. CLAMSE applies a HAC algorithm to the embeddings of the amendments built by several model MQMBC1 PMB2 MQMBD1 AMB1 GTXL silhouette Sentence-BERT PTLMs. This is a preliminary study whose main objective was to explore and evaluate the application of embeddings generated by PTLMs, in particular Sentence-BERT, in comparison to the traditional approach based on TF implemented by the existing system Similis. The preliminary results are encouraging and show that CLAMSE, without fine-tuning, achieved interesting performance.
While these results are promising, several limitations should be noted. First, due to the constraints of using models with a maximum input length of 512 tokens, amendments longer than this threshold are truncated, accounting for approximately 3.33% of the dataset. Second, the encoding is based solely on the textual content of the amendments, ignoring the specific articles of the law to which they refer. As a result, amendments with identical text but referencing diferent articles may be grouped together in cases where they should not be, potentially compromising the accuracy of the clustering results. In addition, the computational eficiency of the method is a concern, as it requires encoding the entire corpus with multiple models and running HAC each time to determine the optimal clustering result.
This evaluation suggests a promising potential for the future development of CLAMSE. In particular, the possibility of training specific models for amendment clustering could be explored, taking advantage of the pre-trained models that already show good eficiency in terms of semantic embedding. This approach may represent an interesting research direction to further optimize the performance of the system and refine its adaptability to new datasets of amendments to be clustered.
It is worth observing that Similis can leverage Linkoln [