This challenge consists of three classification tasks, in the context of argument mining in the legal domain. The tasks are based on a dataset of 225 Italian decisions on Value Added Tax, annotated to identify and categorize argumentative text. The objective of the first task is to classify each argumentative component as premise or conclusion, while the second and third tasks aim at classifying the type of premise: legal vs factual, and its corresponding argumentation scheme. The classes are highly unbalanced, hence evaluation is based on the macro F1 score.
To what extent are Large Language Models (LLMs) capable of reasoning, as opposed to simply recognizing patterns from vast amounts of data is an open research question and the subject of a lively ongoing debate [1]. A way to describe human reasoning is through its ability to understand, evaluate, and invent arguments composed by claims, evidence, and conclusions meaningfully connected with one another [2]. For this reason, the ability to recognize arguments could be considered as a first step in a sequence of reasoning tasks of increasing complexity, that goes from the detection and classification of argumentative discourse units or argument components, through argument structure prediction, reconstruction, evaluation, down to argument generation. Automatizing these tasks is the object of argument mining [3,4,5]. We believe that gauging the ability of LLMs to address even basic argument mining tasks would provide meaningful cues as to these models' ability to process and understand logical relations expressed in natural language. While several datasets for argument mining in English have been developed over the last decade [6,7,8,9,10], resources for other languages remain scarce. To the best of our knowledge, only a few works exist for Italian. In [11], the authors use the CorEA corpus of user comments to online newspaper articles, to assign the correct relation (support or attack), to pairs of arguments. In [12], the authors propose a new model for stance detection, trained and evaluated on a corpus of Italian tweets where users were discussing on a highly polarized political debate.
Among the many domains of interest for argument mining, our focus is on the legal domain, where argumentation is fundamental for the decision-making process. Legal reasoning relies heavily on well-structured arguments, as legal professionals must construct and deconstruct arguments within formal documents, providing a challenging setting for assessing an LLMs' ability to engage in complex reasoning tasks. Despite its relevance, little attention has been given to argument mining in the legal domain in Italian. Most existing work in legal NLP for Italian has focused on tasks such as law article retrieval [13,14], outcome prediction [15], analysis of contracts [16,17], and summarization [18,19].
Our challenge for CALAMITA [20] consists of three classification tasks over argumentative texts. We mostly follow the setting used in Demosthenes [21,22], a corpus for argument mining on legal documents in English. Since we leverage real legal documents, not synthetic or artificially constructed case studies, our dataset reflects the real complexity and nuances of legal argumentation. It is therefore particularly relevant for a robust assessment of LLMs' abilities in real-world applications. To the best of our knowledge, we are the first to propose a challenge of argument mining over legal documents in Italian.
The challenge requires understanding not only the Italian language but domain-specific technical language. Such a language uses complex syntactic structures, and a specialized terminology. Besides language, the challenge tests LLMs' ability to recognize and interpret legal arguments by recognizing typical argumentation schemes [23], e.g., patterns of reasoning used in human discourse, offering a principled approach to argument analysis and evaluation. Identifying schemes is challenging as there are many possible schemes, and arguments are often only partially laid out in the text, leaving many important parts implicit for brevity or because they are considered common knowledge. Nonetheless, this task lends itself to generalization beyond the legal domain, making the insights transferable to other fields where structured reasoning plays a critical role.
We consider an argument as a set of interconnected portions of texts called argument components. The connections between components form a specific pattern of relationships that represents a reasoning paradigm.
The following tasks presume that argument components have already been identified from the source documents. Argument components can therefore be classified according to their role in the connections (such as Premises or Conclusions), according to their content (such as Legal or Factual), and according to the relationship pattern they contribute to (the Argumentative Scheme).
This challenge proposes three classification tasks, in the context of argument mining in the legal domain: The following paragraphs contain a definition of each class, along with an example extracted from the dataset. The translated version of the examples is available in Appendix A.
• Argument premise: a proposition that provides a reason or support for the argument.
Si osserva poi che ritenere che la mancata possibilità di detrazione a favore di soggetti come il ricorrente comporti un aiuto di Stato in favore degli ospedali pubblici, in quanto le perdite degli stessi vengono ripianate dalle USL e dalla Regioni trascura di considerare l'accessibilità, indiscriminata, ai servizi dei nosocomi pubblici da parte dei soggetti iscritti al SSN, rispetto a quella ad un libero professionista sanitario che, in quanto tale, ben potrebbe rifiutarsi di prestare i propri servigi al pare di un normale contraente.
• Argument conclusion: the statement that follows logically from the premise(s) and represents the final point being argued for.
Dunque, l'ufficio ha riconosciuto la non imponibilità IVA delle cessioni all'esportazione, così cessando sul punto la materia del contendere.
Argument components can be involved in more than one relationship, therefore a component may be the conclusions of other premises, as well as a premise of other arguments. In that case, the component is to be classified as a premise.
Premise Type classification. Multi-label classification: classify an argumentative premise as factual or legal (or both).
• Factual premise: a premise that describes factual situations and events, pertaining to the substance or the procedure of the case.
Indubbiamente, la contribuente ha impugnato la sentenza di prime cure, rappresentando nuovamente di non aver potuto proporre appello avverso la pronuncia di condanna di primo grado, per causa di forza maggiore.
• Legal premise: a premise that specifies the legal content (legal rules, precedents, interpretation of applicable laws and principles).
La giurisprudenza citata, alla motivazione della quale si fa rinvio, ha tra l'altro preso posizione espressamente e positivamente sulla conformità della normativa italiana rispetto a quella dell'Unione Europea, risultando così confutata anche la doglianza della difesa sul punto che ha chiesto la sospensione del procedimento, con investitura della Corte di Giustizia Europea della questione.
Since a premise could be both factual and legal, this task is framed as multi-label binary classification.
Legal premises determine the nature of the legal reasoning they support, hence they are labeled with the corresponding reasoning pattern, called argument scheme. We define five schemes relevant for tax law. Each legal premise may be assigned multiple schemes, therefore we frame this task as multilabel multi-class classification. Given a legal premise, classify it as belonging to one or more of the following schemes: (established) rule, precedent, classification, interpretative, or principle.
• Rule (or established rule) scheme: it is used whenever an explicit reference to codified law is present. This reference can be the reference to a certain article or the quotation of the text of a certain article.
Infatti, è ben vero che, ai sensi del combinato disposto dagli articoli 54 e 23 D.Lgs. n. 546/1992, il convenuto in appello deve costituirsi entro 60 giorni dal giorno in cui ricorso è stato notificato.
• Precedent scheme: it is used whenever there is an explicit reference to a previous decision. In the dataset we considered only the references to a decision of both the Court of Cassation or the European Court of Justice.
L' Amministrazione "ha l'onere di provare ed allegare gli elementi probatori su cui si fondi la contestazione, tra i quali possono rilevare, in via indiziaria, quali elementi sintomatici della mancata esecuzione della prestazione dal fatturante, l'assenza della minima dotazione personale e strumentale, l'immediatezza dei rapporti (cedente/prestatore fatturante interposto e cessionario/committente), una conclamata inidoneità allo svolgimento dell'attività economica e la non corrispondenza tra i cedenti e la società coinvolta nell'operazione".
• Classification scheme: it is used whenever a legal concept is defined, its properties are listed, and a certain fact or legal deed must be qualified as having those properties.
Trattano esclusivamente prodotti finiti e non possono, congiuntamente, acquistare oro da gioielleria usato, fonderlo (per proprio conto o con incarico a terzi) e cedere il prodotto finito ottenuto.
• Interpretative scheme: it is used whenever the Court expresses new interpretative assertions (that may depend on previous case law) thereby creating new precedents.
Si vuole dire, in sostanza, che la finalità del contraddittorio anticipato è quella di mettere il contribuente nella condizione di potere fare valere le proprie osservazioni prima che la decisione sia adottata e, quindi, di far sì che l'Amministrazione possa tener conto di tutti gli elementi del caso nell'adottare (o non adottare) il provvedimento ovvero nel dare a questo un contenuto piuttosto che un altro.
• Principle scheme: it is used whenever the Court explicitly refers to a principle of law (e.g. the Principle of proportionality).
Nell'ordinamento unionale, pertanto, il principio del contraddittorio in ambito tributario prescinde dalla natura del tributo e deve trovare applicazione ogni qualvolta l'amministrazione sulla base della documentazione esibita ritenga dovere dare alla stessa documentazione interpretazione diversa da quella data dal contribuente invitandolo, come detto, a fornire nel corso del contraddittorio le ragioni della propria scelta.
The data consists of argumentative portions of text extracted from 225 Italian decisions on Value Added Tax (VAT) by the Regional Tax Commissions from various judicial districts. The decisions were downloaded partially from the open Giustizia Tributaria database 1 and from other judicial databases accessed through university licensing agreements. The decisions range from 2010 to 2022 and concern taxable transactions, exemptions, outof-scope transactions, and the right to obtain a deduction. The argumentative components were extracted from the sections "Motivi della decisione", "Diritto" or "Fatto e diritto", depending on the format of each decision. The collected data were anonymised modifying any identification data of natural or legal persons involved in the proceedings. In particular, the names of the parties in the proceeding and, to provide the highest privacy standards, also the names of the companies have been replaced with initials (e.g., Mario Rossi in "MR", Company s.r.l in "C s.r.l. "). The names of the judges composing the judicial panel have been replaced by "giu1, giu2, [...] giuN". Also, addresses and places were replaced with 'XXX', and dates were changed to show only the year in the following format: DD/MM/2015.
The dataset was annotated by four tax law experts. Annotation guidelines are significantly based on our previous work on the Demosthenes corpus [21], a dataset with English documents from the Court of Justice of the European Union. The guidelines were adapted to the Italian decisions, and refined through an iterative process of validation and discussion, to solve conflicts between annotators. In particular, the annotation is based on the same classes used in Demosthenes. However, the structure of the decisions is different: while in the English corpus the annotation is done at the sentence level, it is not always possible to meet this criterion in Italian decisions. Therefore, the constraint has been relaxed, allowing a single annotation to cover multiple sentences and a single sentence to contain multiple annotations. The tagged decisions are available in our GitHub repository.2
Data are available as a Hugging Face Dataset, 3
The composition of the dataset is summarized in Table 1. The splitting between train, validation, and test data was done at the document level so that components of the same document belong to the same split. It was performed manually, with a ratio of approximately 60:20:20, and the aim of balancing the Scheme classes as much as possible. We adopt the train/val/test format to make the results comparable with as many methods as possible, such as fine-tuned transformer-based models.
Premise
Due to the heavy unbalance between the classes, we evaluate the results using the macro F1 score. Additionally, we evaluate the F1 score of each class to provide further insights.
As a reference, in Demosthenes [21] the best macro F1 results for the three tasks are 0.88 for Argument Component classification, 0.85 for Premise Type classification, and 0.75 for Argument Scheme classification. It is important to specify that these scores are not directly comparable and we provide them only as a reference of the difficulty of the tasks.
The original documents, along with the argument mining annotation, are already available as part of the Adele tool. 4 The original documents, annotated according to the task of outcome prediction instead of argument mining, are also published in [15].
The dataset is limited in size, consisting of only 225 legal decisions on Value Added Tax (VAT). While this provides a valuable resource for testing argument mining models in the Italian tax legal domain, the relatively small dataset may not capture the full diversity of argumentative structures present in the broader Italian tax legal system or other legal domains. This could limit the scalability of models trained on this dataset. Also, given that the legal decisions are from a specific time frame (2010-2022), the dataset may not reflect more recent developments or changes in legal reasoning or tax law.
Secondly, the dataset has been anonymised to protect the privacy of individuals and legal entities. While this is necessary to comply with data protection regulations, the anonymisation process may have removed certain contextual details (e.g., names of places or entities) that could be relevant for understanding the nuances of certain legal arguments. As a result, models may not fully 4 https://adele-tool.eu/ capture or leverage such contextual details that would otherwise aid in more accurate argument classification.
Another limitation is the manual annotation process, which, despite efforts to ensure consistency through expert annotators and conflict resolution, may still be subject to human bias or interpretation inconsistencies. These subjective elements could affect the quality and reproducibility of the tasks.
The dataset comprises legal decisions that have been anonymised to protect the privacy of the individuals. However, it is important to acknowledge the potential risks related to re-identification, even with anonymisation efforts, especially in legal contexts where case details could be cross-referenced with external sources. Care was taken to remove any personal identifiers, such as names, addresses, and dates, but residual risks may remain.
Additionally, the use of this dataset raises questions regarding the deployment of AI systems in legal contexts. AI used by a judicial authority in researching and interpreting facts and the law are considered high-risk by the AI Act. 5 Those systems must conform to the essential requirements (e.g. data governance, user transparency, human oversight, etc.) and the conformity must be documented.
Finally, a critical aspect is the transparency and accountability of AI systems when applied in sensitive domains like law. Users of the models should understand their limitations, especially in tasks involving nuanced reasoning like legal argumentation. Furthermore, ensuring that legal professionals and stakeholders have the ability to audit and interpret the decisions made by AI models is crucial to avoid undermining trust in legal institutions.
The dataset used in this challenge consists of legal decisions on Value Added Tax (VAT) made by the Regional Tax Commissions in Italy, available and downloaded from the Giustizia Tributaria and other judicial databases accessed through university licensing agreements. These legal texts, being official public documents, are generally not subject to copyright restrictions. The dataset consists of a non-substantial part of the respective databases.
Moreover, the use of data is compliant with the text and data mining exception under the EU Copyright Directive and implementing national law. 6 Since the data has been processed and annotated, the annotations and derived data are subject to copyright by the authors of this challenge. To promote transparency and further research, the dataset is released under the Creative Commons Attribution 4.0 International (CC BY 4.0) license. This license allows others to share, use, and adapt the data, as long as appropriate credit is given to the creators, and any modifications are explicitly indicated.
It should be noted that viewing the inability to deduct expenses for individuals such as the plaintiff as state aid to public hospitals overlooks the indiscriminate accessibility of public hospital services for individuals registered with the National Health Service (SSN). In contrast, a self-employed healthcare professional may refuse to provide services as an ordinary contractor.
Thus, the office recognized the VAT non-taxable nature of the exportation, thus considering there is no longer any grounds to proceed on the matter.
Undoubtedly, the taxpayer appealed the first instance ruling, again representing that she could not appeal against the first instance decision due to force majeure.
The cited case law, to which reference is made for its reasoning, has explicitly and positively addressed the conformity of Italian legislation with that of the European Union. This effectively refutes the defense's objection on this point, which requested the suspension of the proceedings and the referral of the issue to the European Court of Justice.
In fact, it is true that under Articles 54 and 23 of Legislative Decree No. 546/1992, the defendant on appeal must come up for trial within 60 days from the day on which appeal was served.
The Administration "has the burden of proving and attaching the evidence on which the dispute is based, among which the absence of the minimum personal and instrumental equipment, the immediacy of the relationships (transferor/interposed invoicing provider and transferee/buyer), an overt unsuitability to carry out the economic activity and the mismatch between the transferors and the company involved in the transaction may be circumstantial. "
In conclusion, given what has been said so far, "gold shop" can be defined as "business establishments that buy, trade or resell used objects of gold, precious metals or bearing precious stones and dispose of them in the form of material, scrap gold or precious metals to foundries or other companies specialising in the recovery of precious materials". They deal only in finished products and may not purchase used jewelery gold, melt it down (for their account or by commissioning a third party) and dispose of the resulting finished product.
It means that, in essence, the purpose of the right to be heard is to put the taxpayer in the position of being able to make his or her observations before the decision is made and, therefore, to ensure that the administration can take into account all the elements of the case in adopting (or not adopting) the measure or in giving this one content rather than another.
In the European Union system, therefore, the right to be heard in tax matters is independent of the nature of the tax and must be applied whenever the administration on the basis of the documentation exhibited deems it necessary to give the same documentation an interpretation that differs from that given by the taxpayer, inviting him, as mentioned, to provide in the exercise of the right to be heard the reasons for his choice.
"Classify the following argumentative text as premise 'prem' or conclusion 'conc'. A premise (prem) is a proposition that provides a reason or support for the argument. A conclusion (conc) is the statement that follows logically from the premise(s) and represents the final point being argued for. Text:"
"Classify the following premise as factual 'F', legal 'L' or both. Factual premises (F) describe factual situations and events, pertaining to the substance or the procedure of the case. Legal premises (L) specify the legal content (legal rules, precedents, interpretation of applicable laws and principles). The expected output is a list with all applicable labels. For example: ['F', 'L']. Text:"
"Classify the following legal premise as one or more of the following argumentative schemes: Rule, Prec, Class, Itpr, Princ. Rule: whether there is an explicit or implicit reference to an article of law or citation of the text of a certain article. Prec: whether there is a reference to a previous ruling of the Supreme Court or the Court of Justice of the European Union. Class: if there is a definition of a legal concept or its constituent elements. Itpr: if there is reference to one of the interpretative criteria contained in Article 12 of the prelegislations (literal, teleological, psychological, systematic) to the Civil Code. Princ: if there is a reference to a general principle of law (e.g. principle of proportionality). The expected output is a list with all applicable labels. For example:
Testo: Si osserva poi che ritenere che la mancata possibilità di detrazione a favore di soggetti come il ricorrente comporti un aiuto di Stato in favore degli ospedali pubblici, in quanto le perdite degli stessi vengono ripianate dalle USL e dalla Regioni trascura di considerare l'accessibilità, indiscriminata, ai servizi dei nosocomi pubblici da parte dei soggetti iscritti al SSN, rispetto a quella ad un libero professionista sanitario che, in quanto tale, ben potrebbe rifiutarsi di prestare i propri servigi al pare di un normale contraente Risposta: prem Testo: L'appello è infondato e va respinto Risposta: conc Testo: Va osservato che la motivazione dell'atto di accertamento non può esaurirsi nel rilievo dello scostamento, ma deve essere integrata con la dimostrazione dell'applicabilità in concreto dello 'standard' prescelto e con le ragioni per le quali sono state disattese le contestazioni sollevate dal contribuente. "Classifica la seguente premessa come di fatto 'F', legale 'L' o entrambe. Le premesse di fatto (F) descrivono situazioni ed eventi fattuali relativi al caso di specie. Le premesse legali (L) specificano il contenuto giuridico (norme giuridiche, precedenti, interpretazione delle leggi e dei principi applicabili). L'output atteso è una lista con tutte le label applicabili. Ad esempio: ['F', 'L'].
Testo: Per i primi giudici nel caso di specie questa esenzione non poteva essere applicata perché la complessiva attività di 'A' srl era un'attività commerciale svolta in concorrenza con altre imprese operanti nel settore Risposta: ['F'] Testo: In assenza di siffatti elementi, che in via presuntiva avrebbero potuto fare giungere questo giudice a conclusioni diverse in via logica, si deve confermare l'esito cui è giunta la commissione provinciale Risposta: ['F'] Testo: Su questo si osserva che si deve condividere la circostanza dedotta dal giudice di prime cure per cui deve essere il contribuente, ove sia contestata la inerenza e verità della rappresentazione ricavabile dal documento contabile, a dare la dimostrazione della fondatezza e della correttezza del comportamento tenuto Risposta: ['L'] Testo: L'Ufficio non potrà impedire ad un imprenditore, per esempio, di cedere immobili con prezzi bassi onulli per ricavare liquidità a fronte di nuovi impegni, ma dovrà rilevare la condotta antieconomica dello stesso sulla base dell'utile di esercizio Risposta: ['L'] Testo: Invero l'avviso di accertamento è fondato sul mancato rispetto, da parte del contribuente, nel calcolo del ROL, delle disposizioni dell'articolo 96, secondo comma, del TUIR, che ne definisce le modalità Risposta: ['F', 'L'] Testo: La società 'A', per quanto previsto dall'art. Itpr: se c'è il riferimento a uno dei criteri interpretativi contenuti all'art. 12 delle preleggi (letterale, teleologica, psicologica, sistematica) al codice civile. Princ: se c'è un riferimento espresso a un prinicpio generale del diritto (es. principio di proporzionalità). L'output atteso è una lista con tutte le label applicabili. Ad esempio: ['Prec', 'Princ', 'Rule'].
Testo: Infatti, è ben vero che, ai sensi del combinato disposto dagli articoli 54 e 23 D.Lgs. n. 546/1992, il convenuto in appello deve costituirsi entro 60 giorni dal giorno in cui ricorso è stato notificato.
Testo: L'Amministrazione "ha l'onere di provare ed allegare gli elementi probatori su cui si fondi la contestazione, tra i quali possono rilevare, in via indiziaria, quali elementi sintomatici della mancata esecuzione della prestazione dal fatturante, l'assenza della minima dotazione personale e strumentale, l'immediatezza dei rapporti (cedente/prestatore fatturante interposto e cessionario/committente), una conclamata inidoneità allo svolgimento dell'attività economica e la non corrispondenza tra i cedenti e la società coinvolta nell'operazione" Risposta: ['Prec'] Testo: In conclusione, per quanto fin qui esposto, i "compro oro" possono essere definiti come "esercizi commerciali che acquistano, commerciano o rivendono oggetti d'oro, di metalli preziosi o recanti pietre preziose usati e li cedono nella forma di materiale, di rottami d'oro o di metalli preziosi alle fonderie o ad altre aziende specializzate nel recupero di materiali preziosi". Trattano esclusivamente prodotti finiti e non possono, congiuntamente, acquistare oro da gioielleria usato, fonderlo (per proprio conto o con incarico a terzi) e cedere il prodotto finito ottenuto Risposta: ['Class'] Testo: Si vuole dire, in sostanza, che la finalità del contraddittorio anticipato è quella di mettere il contribuente nella condizione di potere fare valere le proprie osservazioni prima che la decisione sia adottata e, quindi, di far sì che l'Amministrazione possa tener conto di tutti gli elementi del caso nell'adottare (0 non adottare) il provvedimento ovvero nel dare a questo un contenuto piuttosto che un altro.
Testo: Nell'ordinamento unionale, pertanto, il principio del contraddittorio in ambito tributario prescinde dalla natura del tributo e deve trovare applicazione ogni qualvolta l'amministrazione sulla base della documentazione esibita ritenga dovere dare alla stessa documentazione interpretazione diversa da quella data dal contribuente invitandolo, come detto fornire nel corso del contraddittorio le ragioni della propria scelta Risposta: ['Princ'] Testo: In sintesi per esterovestizione si intende la fittizia localizzazione della residenza fiscale di un soggetto all'estero, in particolare in un Paese con un trattamento fiscale più vantaggioso di quello nazionale,che la giurisprudenza configura in termini di abuso del diritto riconosciuto, in via tendenziale, come principio generale anche nel diritto dei singoli Stati membri (v. Cass., Sez. Un., n. 30055 del 2008, secondo la quale il divieto di abuso del diritto si traduce in un principio generale antielusivo che trova fondamento, in tema di tributi non armonizzati, nei principi costituzionali di capacità contributiva e di progressività dell'imposizione).
Risposta: ['Prec', 'Class', 'Princ'] Testo: La denuncia, infatti, non codificata nel codice di procedura penale (a differenza della notizia di reato di cui all'articolo 347 c.p.p.), può definirsi come qualunque atto con il quale chiunque abbia notizia di un reato perseguibile d'ufficio ne informa il pubblico ministero o un ufficiale di polizia giudiziaria.
Risposta: ['Rule', 'Itpr', 'Class'] Testo: "
This work was partially supported by the following projects: "ADELE -Analytics for DEcision of LEgal cases" (Justice Programme, GA. No. 101007420); PRIN2022 PRIMA -PRivacy Infringements Machine-Advice (Ref. Prot. n.: 20224TPEYC -CUP J53D23005130001); "FAIR -Future Artificial Intelligence Research" -Spoke 8 "Pervasive AI'', under the European Commission's NextGeneration EU programme, PNRR -M4C2 -Investimento 1.3, Partenariato Esteso (PE00000013).