Events in the legal domain: rst impressions

Abstract

Dealing with events is a challenging task in Arti cial Intelligence; just trying to de ne what is an event and deciding what information should be considered relevant to it is a di cult and domain dependent endeavour. Despite this task has been already tackled in the legal domain, no consensual de nition nor standard representation for legal events has been established yet. In this paper, we analyze current approaches to event representation and extraction in the legal domain, and we review generic approaches as well. We expose our rst impressions derived therefrom, and we o er a rst round of event annotations of judgments of the European Court of Justice. Aspects such as the concept of relevance are discussed, along with choices for evaluation. Possible strategies for extraction of legal events are described. Finally, a roadmap for a formal, complete de nition and delimitation of events in the legal domain is presented.

Keywords

• legal events
• court decisions
• relevance
• European Court of Justice

-

Extracting legal relevant events from a sheer scale of legal corpora is challenging because of the ne grained information contained therein. For example, caselaw3 can be long (and rambling) and information related to important legal events of the case can be scattered. Moreover, there is no settled way a factual element can be depicted, and the information is not complete as presented, in the sense that inferences must be made to determine a de nite value. By locating 3 Research on legal information-seeking behavior [ 1 ] and legal information retrieval [ 2 ] posit that lawyers search for useful legal documents that are relevant for their task at hand, to enable decision-making or problem solving (\situational relevance"). Haruna et al. [ 3 ] assert that the most common information needs that lawyers have refer to the latest higher court decisions and the most recent legislation. Therefore, in this work we refer to the judgments of the European Court of Justice (ECJ), identifying therein the legal events. textual regions likely to compact relevant legal events will avoid incurring the expense of reading entire texts. Also, common event-based constituents [ 4 ] eases the detection of events: the legal history of a claim (what give raise to the dispute and occurrence of events time-line, among others); parties (who are the people involved in the dispute); nal decision, etc. Perusal and a de ned consensual de nition of legal events will enable further harvesting thereof.

Events can also be found in other kind of legal documents, such as contracts and regulations, so they could be used, for instance, to check compliance among di erent jurisdictions based on the information surrounding a same event.

In this paper, we describe the preliminary steps we have taken towards extraction of legal events. We elaborate on the quali cation and scope of legal events, having as a use-case the ECJ judgments. The research question of this paper consists in identifying which linguistic and legal information is required to reliably identify legal events.

The remainder of the paper is organized as follows. Section 2 reviews di erent proposals in the literature both for extraction and representation, from generic to legal speci c ones. Section 3 analyses how the previous approaches addressed the evaluation of event extraction. Section 4 further elaborates on the varying criteria to which legal events are bounded to, and introduces our rst impressions on the task. Finally, Section 5 presents the conclusions and the future work. 2

Concept of event and previous approaches

For the tasks of event extraction and representation, di erent approaches have been proposed in the literature. Most of them provide for their own de nition and formalization of the concept of event. In this section we will review di erent proposals, both for extraction and representation, from generic to legal speci c ones. 2.1

General approaches

Regarding representation, there are several ontologies available dealing with events. The W3C Time Ontology4, focusing on temporal expressions, is usually combined with the PROV Ontology5 to represent events stressing their temporal information. The Event Ontology of Yves Raymond6 allows representing a basic concept of event, considering just as surrounding information where it happens, when it happens, and who intervenes in that event, as well as the factor and the product of it. Also sub events can be declared. Similarly, the Simple Event Model (SEM)7 also deals with that information, but adding concepts such as Role, Authority or EventType; SEM has also been later expanded8 to its use in EventKG 4 https://www.w3.org/TR/owl-time/ 5 https://www.w3.org/TR/prov-o/ 6 http://motools.sourceforge.net/event/event.html 7 https://semanticweb.cs.vu.nl/2009/11/sem/ 8 http://eventkg.l3s.uni-hannover.de/schema model.html [ 5 ], a event-centric knowledge graph. Besides ontologies, frames, scripts or case frames [ 6 ] are also used to represent events; Ontology Design Patterns9 suggest how to represent events for di erent kind of situations.

Di erent annotation schemas are also often used for event extraction. One of the most used mark up language in the temporal domain is TimeML10. TimeML allows to annotate temporal expressions (such as dates or times), events and temporal relations among them. TimeML events can be of several types (like reporting, aspectual or state); by using the tag MAKEINSTANCE, several instances can be linked to a given event and also attributes like polarity or aspect can be expressed. Also the ACE model [ 7 ] has been widely-used in previous literature, and has its own de nition of event11. The ACE project guidelines focus on di erent types of events, among which we nd JUSTICE, with its own subtypes (namely, arrest-jail, release-parole, trial-hearing, charge-indict, sue, convict, sentence, ne, execute, extradite, acquit, pardon and appeal ). Several studies were conducted to compare these approaches [ 8, 9 ]. Finally, Chambers et al. [ 6, 10 ] consider that verbs sharing coreferring arguments are semantically connected, what they call narrative coherence; they use this information and Semantic Role Labeling to learn new events in an unsupervised way.

Nevertheless, regarding extraction, most approaches develop their own de nition and strategies. Hagege et al. [ 11 ] claim the di culty of de ning an event from a concept perspective, so they decide to consider an event any verb (state or action), any deverbal noun, any noun argument of the preposition during, or any time span noun. Carpet et al. [ 12 ] developed their own ad hoc way to represent events, consisting on some templates with the core of the event and some coordinates (agents, other participants, places and time). The work of Hogenboom et al. [ 13 ] gives an overview on di erent event extraction methods.

Existing proposals lack therefore of a common de nition of an event, usually using ad hoc representations for very speci c type of event extraction. Annotation formats are on the other hand too generic, and uneasy to adapt to the legal domain needs. The ontologies available for representation are also very similar, having as a basis the same information (what, who and where). 2.2

Approaches in the legal domain

Regarding the legal approaches to event extraction and representation, we have divided previous literature in three di erent communities: Legal Information Retrieval (LIR) proposals, Events in Legal Requirements Engineering and Events in Legal Knowledge Representation. An overview of the di erent approaches analyzed can be found below in Table 1. Therein we mention, in relation to each paper or contribution, their objective, if whether and which there is a de nition of legal event, the analyzed corpus (if any), the methodology and the level of implementation readiness. 9 http://ontologydesignpatterns.org/wiki/Community:Event Processing 10 http://www.timeml.org/ 11 \An event is a speci c occurrence involving participants. An Event is something that happens. An Event can frequently be described as a change of state."

Name LKIF [ 20 ]

Name of LegalRuleML. It was neither used for formalizing or annotating the content of a legal corpora, either automatically or manually, to understand how events in the legal domain are ascertained. In general, event in the legal domain is used as a ne grained structure, and also combined with others (entity, relation, event, temporal expression, value). 3

Evaluation

Since there are several interpretations and formalizations on what is an event, evaluation is also a controverse issue. Di erent concepts imply di erent representations and attributes, needing therefore of di erent metrics to measure relevant aspects. An example of this di erence becomes evident when comparing TimeML 12 13 14 15

Some of these concepts have correspondences with RE literature on Legal Requirements https://www.oasis-open.org/committees/tc home.php?wg abbrev=legalruleml http://www.akomantoso.org http://www.akomantoso.org/?page id=47 concept of event and the one by Ji et al [ 22 ]. While the rst tackles events from a pure temporal point of view, marking them up as temporal entities with attributes and linking them just to other temporal expressions, the second focuses in argument detection, centering its evaluation on Named Entities and temporal ordering. It must be noted that the latest deals with cross-document event extraction, where common entities are a useful lead to connect related events and ordering tends to be more tricky than in a single (and usually chronological) document. In this section we will analyze the di erent evaluation methods proposed in previous literature, trying to derive the one that ts the most to our aim in the legal domain.

Time-related tasks in SemEval have been for many years the reference venue where di erent temporal tools demonstrated their capabilities. While rst editions of TempEval [ 23, 24 ] made use of the TimeML standard for dataset annotation, evaluation consisted in using the gures Precision, Recall and F-measure on the extent and the type of event (among the nine classes de ned in TimeML guidelines), and optionally other attributes (tense, aspect, polarity, and modality). Additionally, Temporal relations were also extracted in another task, and could be considered additional information of an event, but not a part of its formalization. Nevertheless, in SemEval-2015 [ 25 ] the evaluation changed to temporal question-answering, prioritizing therefore temporal understanding over simple annotation in the text, where just di ering in the span of the annotated event led to an error. On the other side, ACE 2005 evaluation plan [ 26 ] proposes the ACE value VDR (Event Detection and Recognition), a metric developed considering the extent of the annotation, the arguments of the event and the attributes value and modality.

Other proposals include several levels of evaluation. It is the case of BioNLP'09 shared task on event extraction16, where di erent tasks tackled di erent levels of detail of event extraction, from just identifying the so called core of the event (namely, its trigger, type and main argument) to surrounding entities (such as the location of the action) and its factuality (e.g., if it actually happened or is negated or introduced as a mere possibility)

One of our main concerns was relevance. A judgment can contain thousands of events in the broadest sense of the word, but not all of them are relevant from the legal point of view. How could relevance be measured? Is it related to the agents intervening in the event? By the appearance of this event in other judgments from the same court? From the same legal eld? Is it possible to extract a common structure of most legal procedures in a court? A jurisdiction, a country? Regarding relevance, some e orts have been previously made in literature to distinguish important events [ 27 ], although no speci c evaluation metrics on this aspect were found. Just when these events were further used for an alternative task (e.g., summarization), some evaluation was performed using metrics of the nal format [ 28 ]. 16 http://www.nactem.ac.uk/tsujii/GENIA/SharedTask/

Our

rst analysis In this section we introduce the results of a rst study of events on the legal domain. First, with some general observations on the concept of legal events; then, with more speci c considerations derived from the rst annotation of a corpus of judgments from the ECJ. 4.1

General Observations

Legal events (and the annotations thereof) may vary according to di erent criteria contingent to the legal realm. We present possible di erential criteria in a general perspective: Multi-jurisdictionality and multilinguism : legal events uctuate according to common law or civil law jurisdictions, and to the languages in which they are expressed in or translated to.

Document-dependency : quali cation of legal events can vary according to the heterogeneity of the embedding document: legal/corporate documents, e.g. contracts, parliamentary documents, doctrine books, court decisions (landmark cases vs commonplace cases), legislation (primary, secondary, domestic, European, international). Moreover, both the jurisdiction and the underlying domain of the document (civil, criminal environmental, taxes, business, etc.) are pertinent when determining the relevance of an event. For instance, comprising the domestic judicial hierarchy and their procedural rules, and the procedural rules of the higher Courts, e.g. request for a preliminary ruling, hearings, submitting observations, opinions, citing case-law, etc.

Level of abstraction: legal events can be considered from a casuistic analysis (a speci c argument of a case), to a general consideration of the facts of a case. For example, the same event (e.g., someone declares something) can be seen as a declaration (a reporting event), a procedural event (speci cally, part of the timeline of the proceedings) or as an abstract legal event (an event in the document level); also, what is being reported could suppose events on their own. Agents and role: the consideration of legal events can vary according to an agent, which is a participant in some juridical relationship, e.g., the applicant (a victim vs perpetrator), or a reputed judge, a notary, a legal scholar. The role of the court is also in uenced by their level of authority ( rst, second national instances). Temporal, contextual and spatial features : this quadrant (time, context and space) can be illustrated by chronological events within court proceedings, such as the submission of an application in a certain date in the applicant's national Court; pleadings; ulterior appeals to a di erent located court; judgment delivery by the ECJ (located in a di erent State) {as the ultimate decision that ends all proceedings, etc.

Scenario or application-based : annotation of legal events might vary according to the sought application, purpose or scenario. If one considers predicting judicial decisions, events referring to case-facts will be mostly regarded [ 29 ]. If however the devised application aims to detect arguments for legal argumentation, then the party's claims will be the target annotation [ 30 ]. The same holds for considering the most cited case-law for the purposes of considering the authoritative and relevant ones [ 31 ].

These criteria gives a hint on the di erent dimensions to take into account when sketching a rst de nition of an event. They are also a good starting point for building a event extractor in the legal domain. 4.2

Lessons learned from our case study

In this section we refer to the composition and speci cities of the corpus, and the adopted methodology on the annotations; then we expound on the lessons learned and nally we provide an initial scope of legal events.

Corpus, data collection and annotations The current sourced corpus consists of 10 European Court of Justice (ECJ) decisions, dated of 2014, extracted from the EUR-LEX database. These sources were collected because the ECJ decisions contain di erent types of events related to di erent actors and they contain a standard (but not xed) structure in which di erent legal events are embedded. The ECJ decisions generally include the following structural sections (according to the Rules of Procedure of the ECJ): i. Preamble, stating information on the parties and the main object of the judgment; ii. Legal context: listing all the legal instruments used in the judgment; iii. Background of the case. Two settings are observed therein: it is depicted the factual background of the case; and an account of the procedural history of the case (typically) followed at national level, before domestic courts or the state's authorities, before the application was lodged by the Court. The ECJ contains also the questions referred therein to the Court, normally titled as \the dispute in the main proceedings"; iv. Considerations of the questions referred: the observations submitted to the Court by the parties and other actors, such as the Governments of Member States, and the responses of the Court; v. Cost attribution; and nally vi. The ruling, i.e., the nal decision and the orders to the parties.

The judgments were manually annotated following a procedure herein described. While analyzing the decisions, we considered the subsequent constituent parts: i. preamble; ii. background of the case; iii. consideration on the questions referred iv. nal ruling, as they contain legal relevant events.

However, we discarded ii. the legal context and v. the costs attribution, as no legal events are put forth.

As the text of the decisions is divided into numbered paragraphs, in selecting the events, we proceeded as follows. For each paragraph, if events were present, we labeled (i) the contextual information, (ii) the event, and (iii) related synonyms, along with the natural-language content of the judgment. Observations From our text analysis, we present the ensuing event-aware observations: { Events can cover a great deal of linguistic information, consisting of verbs (declare), nouns (appeal) and nominal phrases (the facts of the case). Additionally, it gets more complex to denote its boundaries in terms of the extension to actors and also their roles, within the time/space/context axis. { As usual while processing texts in the legal domain, legal terminology, characterized by synonymy, ambiguity, vagueness, polysemy { suppose an extra challenge. We captured the following variations: a. Conventional terms change their meaning in the judicial decision-making; this is the case, for instance, of the expressions \lodging an application before the Court ", or \criminal proceedings were instituted against the applicant ", where the terms before and against do not have the conventional meaning; b. Terms can have several interpretations. For instance, two verbs (submit, argue) are indicative expressions of an argument. However, the verb \submit" can also refer to submitting written observations or pleadings, which consist in procedural documents lodged before the Court. { In the preamble we can nd di erent types of event-aware information regarding to: a. Identity-related event on identi able information related to all the participants involved in the main proceedings: i. The referring court, and ECJ (and its internal composition); ii. The litigant parties: applicant/defendant iii. Agents (States, European Commission, etc); b. Location-event and date-related event : it is possible to identify both the date of the request and the place of the national referring court, e.g., \10 September 2014, Request for a preliminary ruling from the Krajsky Sud v Presove (Slovakia)". c. Domain-related event of the judgment: the initial summary of the judgment indicates for the domain at stance, e.g., consumer protection (or others, illegal migration, genetic modi ed food, etc.) { The background of the case is the most interesting part to annotate, as it includes the relevant events, arguments and facts of the parties, e.g. \On 26 February 2009 , Mrs Kusionova concluded a consumer credit agreement with SMART Capital for an amount of EUR 10 000", Case 34/13, paragraph 25. { Events can be subsumed to decisions of the national courts ( rst-instance and second-instance courts that refer to the ECJ). Expressions on the text mentioning \Regional Court", \District Court", \national court" illustrate what are the juristic positions of the former courts according to a legal problem. { Interpretative issues were considered when analysing the di erent versions alleged by each of the parties in the dispute with regard to the same event, e.g. the claimant alleges there was illegal use of goods and no smuggling. Each of the involved parties claim that the other one is at fault, which consists in an interpretative indicator of a same event. { Negation and Factuality: it must be noted that some events can be negated or not actual facts, but \possibilities, intentions or preferences" [ 32 ]. Examples of paragraphs 51 and 82, respectively: \The Court does not have before it any evidence which might raise doubts as to the compliance of the legislation at issue in the main proceedings with that principle"; (...) \In the light of the answer to the rst three questions, it is not necessary to respond to that request by the Slovak Government." { As for event-related relations, we observed that two-way (bi-directional) relationships can be found in the same judgment engaging both parties, e.g. actions \submitted by", \brought by"; or \the facts of the case, as submitted by the parties; or \observations submitted by the Government and the observations in reply submitted by the applicant". { Legal related events can be identi ed at di erent (internal) structures of the documents, e.g. at a paragraph level of a court decision, or in the summary, or in the conclusion thereof. { Within the ECJ judgments, we considered as legal relevant events the following descriptors/indicative expressions, among: a. Concerning identi cation elements: petition type, e.g. preliminary ruling; composition of the court, date of the judgment; parties involved; topical content of the case; facts; b. Quoted case-law; The Courts attempt to ground the decision by reference to established past case-law; c. Judgment delivery, where the merit of the case is assigned. 5

Conclusions and future work

This paper presents a theoretical re exive work on events in the legal domain, wherein an account of their varied de nitions, representations and applicationbase uses is o ered.

We performed a literature review of events in the legal domain in three di erent communities: Legal Information Retrieval (LIR), Legal Requirements Engineering and Legal Knowledge Representation. Even if events in the legal domain are slightly considered within known frameworks { as it is the case of LegalRuleML, Akoma Ntoso, LKIF { no consensual de nition nor standard representation for legal events has been established.

Our initial analysis, based on a small corpus of ECJ annotated judgments, permitted to convey an initial scope of legal events. The quali cation of event in the legal domain seeks to be wide-ranging in scope and facilitating the detection and extraction thereof, regardless of their applicable domain (criminal, civil), but customizable/modular for instantiation.

The envisaged usefulness and wide prospective applicability of event extraction/representation as a supportive tool can be evidenced in legal information retrieval (LIR), in legal knowledge engineering, in Legal Requirements Engineering, and legal argumentation communities, where events are already accounted. Other applicable scenarios can be contemplated, among the following: e-discovery; case-based reasoning (CBR); legal argumentation; determining contractual positions and concept de nition; contract review; prediction technology; electronic billing.

As future work, the authors will further prune the de nition of legal events by atomizing their properties, instances, and attributes and set some competency questions relevant for extracting necessary information from court decisions. Forth bringing annotations of a corpus of ECHR judgments will proceed for reviewing, canvassing and consolidating the annotation frames acquired in this paper. A semantic model will be constructed from these annotations to provide for structured legal data on the web. Regarding evaluation, assessment criteria for validating the relevance of an event within a legal text will be developed.

Acknowledgments

This work was supported by a Predoctoral grant from the I+D+i program of the Universidad Politecnica de Madrid.

References

1. Stephann Makri , Ann Blandford, and Anna L Cox. Investigating the informationseeking behaviour of academic lawyers: From elliss model to design . Information Processing & Management , 44 ( 2 ): 613 { 634 , 2008 .

2. Marc Van Opijnen and Cristiana Santos . On the concept of relevance in legal information retrieval . Arti cial Intelligence and Law , 25 ( 1 ): 65 { 87 , 2017 .

3. Ibrahim Haruna and Iyabo Mabawonku . Information needs and seeking behaviour of legal practitioners and the challenges to law libraries in lagos, nigeria . The International Information & Library Review , 33 ( 1 ): 69 { 87 , 2001 .

4. Jody J. Daniels and Edwina L. Rissland . Finding legally relevant passages in case opinions . In Proceedings of the 6th International Conference on Arti cial Intelligence and Law , ICAIL '97 , pages 39 { 46 , New York, NY, USA, 1997 . ACM.

5. Simon Gottschalk and Elena Demidova . Eventkg: A multilingual event-centric temporal knowledge graph . In Aldo Gangemi , Roberto Navigli, Maria-Esther Vidal , Pascal Hitzler, Raphael Troncy, Laura Hollink, Anna Tordai, and Mehwish Alam, editors, The Semantic Web , pages 272 { 287 , Cham , 2018 . Springer International Publishing.

6. Nathanael Chambers and Dan Jurafsky . Unsupervised learning of narrative event chains . Proceedings of ACL-08: HLT , pages 789 { 797 , 2008 .

7. Ace (automatic content extraction) english annotation guidelines for events . https://www.ldc.upenn.edu/sites/www.ldc.upenn.edu/ les/english-eventsguidelines -v5.4 .3.pdf.

8. David Ahn . The stages of event extraction . In Proceedings of the Workshop on Annotating and Reasoning about Time and Events , pages 1 { 8. Association for Computational Linguistics, 2006 .

9. Jacqueline Aguilar , Charley Beller, Paul McNamee , Benjamin Van Durme, Stephanie Strassel , Zhiyi Song, and Joe Ellis . A comparison of the events and relations across ace, ere, tac-kbp, and framenet annotation standards . In Proceedings of the Second Workshop on EVENTS: De nition, Detection, Coreference, and Representation, pages 45 { 53 , 2014 .

10. Nathanael Chambers and Dan Jurafsky . Unsupervised learning of narrative schemas and their participants . In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - Volume 2, ACL ' 09 , pages 602 { 610 , Stroudsburg , PA, USA, 2009 . Association for Computational Linguistics .

11. Caroline Hagege and Xavier Tannier . Xtm: A robust temporal text processor . In International Conference on Intelligent Text Processing and Computational Linguistics , pages 231 { 240 . Springer, 2008 .

12. Philippe Capet , Thomas Delavallade, Takuya Nakamura, Agnes Sandor, Cedric Tarsitano, and Stavroula Voyatzi . A risk assessment system with automatic extraction of event types . In International Conference on Intelligent Information Processing , pages 220 { 229 . Springer, 2008 .

13. Frederik Hogenboom , Flavius Frasincar, Uzay Kaymak, and Franciska De Jong. An overview of event extraction from text . In Workshop on Detection, Representation, and Exploitation of Events in the Semantic Web (DeRiVE 2011 ) at Tenth International Semantic Web Conference (ISWC 2011 ), volume 779 , pages 48 { 57 . Koblenz, Germany: CEURWS. org., 2011 .

14. K Tamsin Maxwell , Jon Oberlander, and Victor Lavrenko . Evaluation of semantic events for legal case retrieval . In Proceedings of the WSDM'09 Workshop on Exploiting Semantic Annotations in Information Retrieval , pages 39 { 41 . ACM, 2009 .

15. Nikolaos Lagos , Frederique Segond, Stefania Castellani, and Jacki ONeill. Event extraction for legal case building and reasoning . In International Conference on Intelligent Information Processing , pages 92 { 101 . Springer, 2010 .

16. Gerardo Sierra , Gemma Bel-Enguix, Guillermo Lopez-Velarde, Ricardo Saucedo, and Luc a Rivera. Event extraction from legal documents in spanish . In Workshop on Language Resources and Technologies for the Legal Knowledge Graph , LREC 2018 . Miyazaki, Japan, 2018 .

17. Anderson Bertoldi , Rove Chishman, Sandro Jose Rigo, and Tha s Domnica Minghelli . Cognitive linguistic representation of legal events. towards a semanticbased legal information retrieval . In COGNITIVE 2014 : The Sixth International Conference on Advanced Cognitive Technologies and Applications , 2014 .

18. Nadzeya Kiyavitskaya , Nicola Zeni, Travis D Breaux , Annie I Anton , James R Cordy, Luisa Mich, and John Mylopoulos. Automating the extraction of rights and obligations for regulatory compliance . In International Conference on Conceptual Modeling , pages 154 { 168 . Springer, 2008 .

19. Silvia Ingolfo , Ivan Jureta, Alberto Siena, Anna Perini, and Angelo Susi . Nomos 3: Legal compliance of roles and requirements . In Eric Yu, Gillian Dobbie, Matthias Jarke, and Sandeep Purao, editors, Conceptual Modeling , pages 275 { 288 , Cham , 2014 . Springer International Publishing.

20. Thomas F Gordon. The legal knowledge interchange format (lkif) . Estrella deliverable d4, 1 , 2008 .

21. Fabien Gandon , Guido Governatori, and Serena Villata . Normative requirements as linked data . In The 30th international conference on Legal Knowledge and Information Systems (JURIX 2017 ), 2017 .

22. Heng Ji , Ralph Grishman, Zheng Chen, and Prashant Gupta . Cross-document event extraction and tracking: Task, evaluation, techniques and challenges . In Proceedings of the International Conference RANLP-2009 , pages 166 { 172 , 2009 .

23. James Pustejovsky , Marc Verhagen, Xue Nianwen, Robert Gaizauskas, Mark Hepple, Frank Schilder, Graham Katz, Estela Saquete, Tommaso Caselli , et al. Tempeval2: Evaluating events, time expressions and temporal relations (in semeval tasks proposal).

24. Naushad UzZaman , Hector Llorens, Leon Derczynski, James Allen, Marc Verhagen , and James Pustejovsky . Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations . In Second Joint Conference on Lexical and Computational Semantics (* SEM) , Volume 2 : Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013 ), volume 2 , pages 1{9 , 2013 .

25. Hector Llorens , Nathanael Chambers, Naushad UzZaman , Nasrin Mostafazadeh, James Allen, and James Pustejovsky . Semeval-2015 task 5: Qa tempeval-evaluating temporal information understanding with question answering . In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015 ), pages 792 { 800 , 2015 .

26. The ace 2005 ( ace 05 ) evaluation plan. evaluation of the detection and recognition of ace entities , values , temporal expressions , relations , and events . https://www.semanticscholar.org/paper/The-ACE-2005 -(-ACE- 05 -)- EvaluationPlan - Evaluation-NTRODUCTION/ 3a9b136ca1ab91592df36f148ef16095f74d009e .

27. Rachel Chasin . Event and temporal information extraction towards timelines of wikipedia articles . Simile , pages 1 {9 , 2010 .

28. Radityo Eko Prasojo, Mouna Kacimi, and Werner Nutt . Modeling and summarizing news events using semantic triples . In European Semantic Web Conference , pages 512 { 527 . Springer, 2018 .

29. Nikolaos Aletras , Dimitrios Tsarapatsanis, Daniel Preotiuc-Pietro, and Vasileios Lampos . Predicting judicial decisions of the european court of human rights: A natural language processing perspective . PeerJ Computer Science , 2 : e93 , 2016 .

30. Marco Lippi , Francesca Lagioia, Giuseppe Contissa, Giovanni Sartor, and Paolo Torroni . Claim detection in judgments of the eu court of justice . In Ugo Pagallo, Monica Palmirani, Pompeu Casanovas, Giovanni Sartor, and Serena Villata, editors, AI Approaches to the Complexity of Legal Systems , pages 513 { 527 , Cham , 2018 . Springer International Publishing.

31. Marc van Opijnen. Towards a global importance indicator for court decisions . In Legal Knowledge and Information Systems - JURIX 2016: The Twenty-Ninth Annual Conference , pages 155 { 160 , 2016 .

32. Mar a Navas-Loro, Ken Satoh, and V ctor Rodr guez-Doncel. Contractframes: Bridging the gap between natural language and logics in contract law . In Proceedings of theTwelfth International Workshop on Juris-informatics (JURISIN) (November 2018 ), 2018 .