English. New regulations on transparency and the recent policy for privacy force the public administration (PA) to make their documents available, but also to limit the diffusion of personal data. The present work displays a first approach to the extraction of sensitive data from PA documents in terms of named entities and semantic relations among them, speeding up the process of extraction of these personal data in order to easily select those which need to be hidden. We also present the process of collection and annotation of the dataset.
Italiano. Le nuove regolamentazioni sulla trasparenza e la recente legislazione sulla privacy hanno spinto la pubblica amministrazione a rendere i loro documenti pubblicamente consultabili limitando pero` la diffusione di dati personali. Presentiamo qui un primo approccio all’estrazione di questi dati da documenti amministrativi in termini di named entities e relazioni semantiche tra di esse, in modo da facilitare la selezione dei dati che devono rimanere privati. Presentiamo inoltre il processo di collezione e annotazione del dataset.
In recent years, public administrations (PA) in the Italian government have been forced to publish a huge amount of documents, to make them available to citizens, organisations, and authorities. This is the result of the recent legislation
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). about the transparency. For instance, municipalities have to share their documents in a virtual place called Albo Pretorio. In most cases, the online publication of these acts is a necessary condition for their purposes to become effective.1
On the other side, the General Data Protection Regulation (GDPR), approved in 2016 by the European Union, enhances individuals’ control and rights over their personal data, limiting its diffusion over any medium (especially including online platforms such as websites and social networks).
In this context, it is important for the public servants within the PA to amend some documents by hiding the data that cannot be publicly published. Nowadays, most of this work is done manually, hiding the sensitive information document by document. This procedure is clearly time-consuming, non-scalable, and error-prone.
Natural Language Processing (NLP) techniques can be seen as a watershed between a manual management of the PA documents and a new generation of instruments that will finally speed up the process, leaving manual effort as the sole final check just before the publication of the data.
This is not the first time this problem is tackled
using NLP, but past works are mainly focused on
English and limited to the entity extraction task
Our approach to the extraction of personal data
from documents focuses on a combination of three
NLP instruments:
• Named-entity Recognition (NER). This
task consists in seeking texts in natural
language to locate and classify named entities
(NE) mentioned in them. This search is
usually limited to a few needed categories: the
most common are persons, locations, and
organisations. Several approaches have been
1In the Italian legislation, this is called “referto di
pubblicazione”. See also: http://qualitapa.gov.it/
used in literature, between completely
rulebased
In this paper we present REDIT (Relation and
Entities Dataset for Italian with Tint), a complete
framework that aims to solve the personal data
identification in textual documents. The software
is mainly based on Tint
The content is structured as follows. Section 2 presents in detail how we collected the documents that are annotated and how we used fictitious data to make the resource available for download. In Section 3, we describe the process used to annotate the data. Section 4 illustrates the dataset, giving some statistics on the entities and relations included in it. In Section 5 we give some results on the performance of the resulting entity extraction and relation extraction system. The downloadable package (that contains the dataset, the model and the Java code) is finally described in Section 6. 2
The corpus is composed of documents taken from different institutions of the public administration. The documents with which we have worked are different types of forms, varying from license for parking to adoption forms, school enrollments, marriage licenses and so on.
Starting from this set, we create two datasets. One is composed of documents compiled with real data and one with documents compiled by us with ifctitious data, using lists of all the Italian streets and surnames in order to guarantee the diversification of the data in the compiled forms, and to not exclusively rely on the annotators’ fantasy. The ifctitious compilation aims to avoid using sensitive data in terms of privacy issues, leading to the possibility of publicly releasing the dataset. The documents which contain real data are indeed not included in the public dataset. For instance, a sentence such as Il sottoscritto Gianluca Freschi, nato a Pesaro il 12/12/1990 e residente in Pesaro, Via Virgilio n.76 presents data whose association was invented by the annotator. It could be possible that a person called Gianluca Freschi exists in real world but it is almost impossible that he would fit with the rest of the data since they all derive by annotator’s fantasy. However, as we can see from the example, while the data are fictitious the structure of the document is identical to that of real ones. 3
Each document in the set is annotated both with entities and relations between them.
For the annotation of entities we adopt the
guidelines already used for KIND
In addition to the NE annotation, we are interested in annotating the relations among them. In particular, we need to develop a system of relations which links the person with its personal data or with its role in terms of responsibility of the company/public administration or in terms of relative/family relationships.
Since for the annotation task a relation must connect two entities, some additional entity types are annotated only when involved in a relation (see below). The list of additional entities includes ROLE for personal and organisation roles (for example, words such as “responsabile”, “titolare”, “genitore”, and so on, representing the role of a person in a company, in the PA domain, or in a family), DOCTYPE for document types (such as “passaporto”, “patente”), EMAIL for e-mail addresses, DATE for dates, NUMBER for generic numbers (such as VAT), CF for the Italian “codice ifscale” sequence of chars.
Regarding relations, address is used for instance to link a LOC entity representing an address to the person or company to which the address belongs, while birthDate, birthLoc link respectively the date and location of birth.
Table 1 shows the complete list of the relations included in the dataset.
The annotation is performed by a domain
expert using INCEpTION
This is an example of NER annotation: Al [Comune di Alessandria]ENTE. [Casale Monferrato]LOC, 20 settembre 2021.
Il sottoscritto [Davide Aiello]PER, nato a [Milano]LOC il [31/07/1985]DATE, [titolare]ROLE della ditta [Aiello Ceramiche S.r.l.]ORG, ai sensi dell’ [art. 76 del D.P.R. n. 445/2000]LEX, dichiara di voler partecipare all’evento “Il mercante in Fiera”.
These are the corresponding relations: • birthLoc (Davide Aiello, Milano) • birthDate (Davide Aiello, 31/07/1985) • companyRole (Davide Aiello, titolare) • personInOrg (Davide Aiello, Aiello Ceramiche S.r.l.)
In the example, “31/07/1985” is tagged as DATE, since it is involved in the birthDate relation. On the contrary, since no relations include “20 settembre 2021”, it’s not mandatory, for the annotator, to mark it as DATE.
The system uses two different approaches to identify entities. Entities such as DATE or ROLE are annotated only when involved in a relation because they are labels identified through a rulebased approach which can be easily recognised without the need of training data. For what concerns instead PER, LOC, ORG, ENTE and LEX the identification occurs using a machine-learning technique and they need to be always annotated. 4
As we have seen in Section 2, the complete dataset consists of two parts: the first one presents the documents fictitiously compiled and it is publicly released; the latter, on the contrary, comprehends instances compiled with real data and is not released. Nevertheless, we consider also the unreleased dataset in training the model, so that the amount of annotated relations in the final dataset is 7,821, while that of annotated entities is 21,307. The released one presents 1,439 annotated entities and 1,476 annotated relations.
Looking at the data in Table 1, it is possible to notice that the amount of annotations referring to some relations (marked with *) are considerably fewer than others. Despite the small amount, we have already annotated them in the view of future works on these relations but we do not consider them in the experiments.
Tint processing
Tokenization Named-entities recognition
Dependency parsing REDIT processing Relation extraction REDIT dataset Part-of-speech tagging Rule-based extractors Lemmatization ENTE classifier To work properly, REDIT relies on a complex pipeline that includes various steps, very different in structure and management (see Figure 1). Most of the steps are performed using well-known tools and algorithms (sometimes not reaching state-ofthe-art accuracy), so that the whole program does not need particular hardware (such as the GPUs needed in environments using deep learning and transformers) and is easy to run on almost every common software environment.
1. First, the input text is parsed with Tint
Sometimes, the entities annotated as ORG in KIND (such as “Unione Europea”) should have been annotated as ENTE in REDIT. We then decided, in the training phase, to merge all ENTE entities into ORG. We then trained a classifier dedicated to the ENTE tag (see Subsection 5.3), trained on REDIT dataset only, that performs the sole disambiguation between ORG and ENTE.
To enhance the classification, Stanford NER
also accepts gazettes of names labelled with the
corresponding tag. We collect a list of
persons, organizations and locations from the Italian
Wikipedia using some classes in DBpedia
The evaluation is performed by randomly splitting the dataset into train/dev/test using 80/10/10 ratio. During training phase, we tried some sets of features choosing among the ones available in Stanford NER. We obtained the best results (considering also a good balance between training/testing time and performances) with word shapes, n-grams with length 6, previous, current, and next token/lemma/class. Table 4 displays the results of the NER module. 5.2
As said in Section 3, there is the need for more entity types, because in the training phase we need to have both arguments of a relation annotates as an entity (of any type). For this reason, we use Relation LEX LOC ORG PER Total (micro) Total (macro) a rule-based approach to annotate DATE, ROLE, DOCTYPE, EMAIL, NUMBER, and CF. • Tint TIMEX annotator is used to tag DATE entities. • ROLE and DOCTYPE entities are extracted given a list of roles taken from the annotated training set. • Numbers, e-mail addresses and Italian codice ifscale are tagged using regular expressions. 5.3
After the previous steps, the entities that should be marked with ENTE now falls into the ORG or LOC entity sets. We then use a simple SVM classifiers (using shallow features, such as words, bigrams, previous and following content words, etc.) that, given an entity tagged as LOC or ORG, return whether it should be annotated as ENTE. The training set used by the classifier consists in entities taken from REDIT and annotated as ORG, LOC, and ENTE. The first two categories represent the zero class, while entities tagged with ENTE represent the other class. It is therefore a binary classifier. In a 10-fold cross-validation environment, results shows a F-score equals to 0.978 (precision 0.981, recall 0.974). 5.4
The last module in REDIT is Stanford Relation
Extractor
Similarly to the NER training, we test approaches with different sets of features, obtaining the best results with unigrams/bigrams, adjacent words, argument words, argument class, dependency path between the arguments, entities and concatenation of POS tags between arguments.
Table 5 shows the results on the relation
extractor (the evaluation is performed using gold-labeled
entities).
All parts of REDIT (except part of the annotated
dataset, see Section 4) are released for free under
the CC BY 4.0 license,2 and can be downloaded
on Github.3 These include the annotations, in
WebAnno format
A working demo of the tool is available online (See Figure 2).4 Its web interface is written with VueJS/Boostrap and it is available for download in the Github project page. 7
In this paper we present a completely automatic approach to extract personal data (view as entities) and relations between them from documents of the public administration written in Italian texts. The pipeline relies on a mix of rule-based and machine learning-base modules. The latter are trained using a manually annotated dataset, which is in part available for download. All the source code, instead, is released and available for download.
In the future, we plan to enhance the coverage of our system by adding more examples on relations that are less represented (see Table 1).
The research leading to this paper was partially supported by Wemapp Srl, Potenza, Italy.5 2https://bit.ly/cc-by-40-intl 3https://github.com/dhfbk/redit 4https://bit.ly/relation-extraction 5https://wemapp.eu/