English. In this paper, we illustrate the creation of a NER for the Public Administration (PA) domain. We discuss the creation of an annotated corpus with documents from the Italian Albo Pretorio Nazionale and provide results of the system evaluation.
In the Public Administration (PA) domain, the rapid adoption of the new legislation about the governance transparency has been forcing Italian municipalities to produce their acts in a digital form and to make them available for both citizens and authorities. However, the acts delivered by PAs are typically in a free-text electronic format, which is not convenient for searching, decisionsupport, and data analysis. Therefore, the development of NLP tools to extract high-quality structured information, including Named Entities (NEs) such as Persons and Organizations, represents a key factor to enable the access to the wealth of information produced by PAs, and a crucial step in turning the keyword of “transparency” into reality. The potentialities of NLP tools can be exploited to mine the large document repositories produced by PA daily, with the aim of identifying trends in their activity, suggesting possible synergies to increase their efficiency, and raising “red flags” about suspicious behaviors, especially for their relationships with private companies.
In this paper, we focus on Named Entity Recognition (NER) for PA. Several approaches have been proposed in literature including Rule-based, Machine Learning-based and Hybrid methods.
Hand-made Rule-based NERs focus on
extracting names using lots of human-made rules. In
general, these systems consist of a set of patterns
based on grammatical (e.g., part of speech),
syntactic (e.g., word precedence) and orthographic
features (e.g., capitalization) in combination with
dictionaries
Machine learning techniques, on the contrary,
use a collection of annotated documents for
training the classifiers. Therefore the development time
moves from the definition of rules to the
preparation of annotated corpora
Finally, Hybrid NER systems, combine
rulebased and machine learning-based methods, and
make new methods using strongest points from
each method
Existing general purpose Italian corpora
annotated with NEs such as I-CAB
For the creation of a NER for PA, we decided to
exploit the existing architecture employed for the
project SEMPLICE1 and in particular we adopted
a statistical method based on the Stanford NER
This paper is structured as follows: In section 2, we present the CoLingLab NER and we show its performance on a sample of PA documents; in 1The SEMantic instruments for PubLIc administrators and CitizEns (SEMPLICE; www.semplicepa.it) is a 2year project funded by Regione Toscana in collaboration with IT companies to develop NLP-based tools for knowledge management, information extraction and opinion mining for local public administrations. section 3 we describe the adaptation of the system to PA texts and its performances (section 4.1). In section 5, we report on the annotation of relations that we performed on a sample of the corpus and finally discuss the results and ongoing work. 2
The standard Italian CoLingLab NER was trained
on the Italian Content Annotation Treebank
(ICAB
As we said before, this model is unsatisfactory for the domain of Public Administration in two main respects. First, its classes are insufficient to deal with the type of information in the PA documents, that are full of references to other “linked” acts and legislative reference; second, the language used in these documents is a peculiar and highly complex variant of standard Italian (cf. above). In addition, the performance of the model, attested at 0.66 of F1-score on a portion of I-CAB decreases dramatically on the PA documents, reaching a F1-score of 0.35. To measure such performances, in the test set we mapped ORG PA (cf. below) with ORG, and in the training set we mapped GPE with LOC. 3
The adaptation of the CoLingLab NER to the PA
domain included the extension of the standard
NE classes
For the creation of the corpus, we used documents
taken from the Albo Pretorio Nazionale with the
aim of capturing the variability of the texts
produced by PA. Overall, the corpus includes 460
documents, for a total of 724,623 tokens,
annotated with the following NEs: (i) ACT: documents
belonging to the Albo Pretorio Nazionale, with
their type (optional), number and date: Determina
n. 4 del 12/02/2011; (ii) LAW: legislative
references: art. 183 comma 7 del D.Lgs. n. 267/2000;
(iii) LOC: locations and geo-political entities:
Comune di Pisa; (iv) ORG PA: organizations related
to the Public Administration such as municipal
Departments: Sezione Anagrafe; (v) ORG:
organizations: Consip Spa; (vi) PER: physical persons.
The corpus has been linguistically annotated by
means of a pipeline of general purpose NLP tools
and in particular, it has been POS-tagged with the
Part-Of-Speech tagger described in Dell’Orletta
(2009), dependency parsed with the DeSR parser
NE annotation has been performed by means of an incremental process: first 100 documents have been annotated by 2 annotators (one of them was a domain expert). In a second phase we trained a CRF model on these documents and we used it to automatically annotate new documents. Finally, we identified the most common errors of the classifier and two new annotators manually revised the output. This process has been repeated for each group of 100 documents up to covering the whole corpus that includes 460 distinct documents. The average length of the documents is 1,575.26 tokens and the total number of the tokens is 724,623. Figure 1 shows the distribution of the different NE classes in the corpus.
NEs have been annotated on the CONLL
The inter-annotator agreement between two annotators (attested at 0.8) has been calculated using the Cohen’s K index on a sample of 25 documents of 25 different municipalities, for a total of 26,190 tokens. 4
To train the NER, no information from gazetteers was used. The model includes the following groups of features: SEQUENCES: Next and previous words and a window of 6 words (3 preceding and 3 following the target word) and their classes; N-GRAMS: Character-level features, i.e., substrings of the word with a maximum length of 6 letters; ORTHOGRAPHY: “word shape” features such as spelling, capital letters, presence of non–alphabetical characters etc.;
the sentence (numeric attribute), the lemma,
and the PoStag (nominal attribute);
TERMS: We employed complex terms as features
to train the model. Terms have been extracted
with EXTra
We trained the CRF model based on the CoLingLab NER on the annotated PA corpus, and we tested its performances first with cross-validation and then on a sample of new 25 documents of 25 different municipalities. This choice stems from the fact that very often different municipalities tend to use different templates and different ways to refer to particular entities. This is particularly common in some NE classes such as ACTS and ORG PA, that vary a lot across municipalities. For example, some of the analyzed texts contain strings of the form YYYY/G/NNNNN to refer to the acts, where the number is actually a string encoding both the date (year: YYYY), a code for the type (G) and the number of the act (NNNNN). Other municipalities instead adopt a less strictly codified pattern to indicate the act such as Type of act, number N* of DD/MM/YYYY. Likewise, depending on the writing style (and conventions) of the municipalities, the various departments (i.e., ORG PA) can include both strings like Corpo dei Vigili Urbani and codes like Tec-01/ICT. To evaluate the system performance with respect to the variation of the naming conventions adopted by different municipalities, we randomly selected 25 municipalities and one document for each of them balanced for length.
Table 1 reports on the results obtained in cross validation and Table 2 shows the performance on the sample of 25 documents. Figure 2 shows also the confusion matrix for that sample.
In order to investigate the contribution of nonlinguistic features, we performed ablation experiments and we tested the results on the sample of 25 documents. The F1-Score for such groups is as follows: SEQUENCES: 3%; N-GRAMS: 1%; ORTHOGRAPHY: 4%. In addition, we performed an additional experiment by training the NER on a combination of I-CAB and the PA documents. In this case, we noticed a F1-Score of 2% by respect to the original model.
ACT LAW LOC ORG ORG PA PER
For a subset of the corpus, we also annotated the semantic relations occurring between two entities in the domain of the PA, using the following scheme: ACT LAW LOC ORG ORG PA PER
PART OF: the relation of hyponymy, which can occur between: (i) two locations (e.g. a Municipality in Province); (ii) two organizations (e.g. a participated into a holding company); (iii) a person and an organization (e.g. a member of an organization). Implicit attribute for this reation is “work in”.
LOCATION: an entity placed into a particular location, occurring between: (i) an organization and a location (e.g. an organization located in a certain region). Possible attributes for this relation are “work in” and “placed in”; (ii) a person and and a location (e.g. a person living in a particular area). Possible attributes are “work in”, “born in” and “placed in”.
IS RELATED TO: an underspecified relation between any entity pair.
Preliminary experiments have been performed to examine the characteristics of an automatic classifier for extracting relations from administrative acts, and the performance seem to be very promising, despite the size of the training set, which includes in total 100 documents so far. The extension of the annotated corpus and the training of the relational classifier are currently ongoing. 6
The results show that the NER reaches
satisfactory results for most of the classes, although
legging behind in the recognition of PA
Organizations, which, among others, tend to have a higher
formal variability, including for example both
entities like Corpo dei Vigili Urbani and Tec-01/ICT.
Moreover, in the recognition of Location names in
the domain of the PA, the system is expected to
detect entities with a non-standard detail level going
from the name of the municipalities (e.g. Comune
di Pisa) to very detailed addresses (e.g., via S.
Maria n. 36, 56126 Pisa (PI) interno 15). A
similar problem occurs in the recognition of very small
organizations, whose name contains the name of
its founder (i.e., Mario Rossi snc). In these cases,
especially when snc is omitted, the system predicts
the class PER instead of the correct class ORG. We
are confident that adding lexicons and gazetteers
will improve the identification of entities of this
kind, but it could be interesting to investigate
automatic normalization, disambiguation and entity
linking approaches
Named entities play an important role in administrative acts, especially in those - like the documents in the Albo Pretorio - describing the main actions taken by Municipalities. This kind of information is very useful to fullfil the obligations related to supervisory monitoring, disclosure, periodic self-assessment, and review of the government decisions.
In this paper, we presented a NER for PA that shows a significant ability to identify the relevant entities, and in particular legislative reference and connected acts. It is important to stress the lexical and syntactic complexity of bureaucratic language represents a big challenge for NLP tools and methods. Such a complexity derives from the technical lexis of other domain-specific languages with which PA deals daily, such as education, environment, ICT technologies, public health and so on. In near feature we plan to explore the possibility of re-engineering our system to take advantage of new algorithms for entity extraction such as neural networks and in particular from character level word embeddings. Moreover, we will focus on the development of classifiers for Relation Extraction and Entity Linking.
This research has been supported from the Project SEMantic instruments for PubLIc administrators and CitizEns (SEMPLICE), funded by Regione Toscana, and the Company ETI3j Evolution, Technology & Innovation. Special acknowledgements go to Roberto Battistelli and Francesco Sandrelli (ETI3) for support, and to the students Roswita Candusso, Carmela Cinquesanti, Federica Semplici and Ludovica Vasile for manual annotation.