Nowadays, the convergence of Artificial Intelligence and data privacy is of crucial importance. This paper introduces NERPII, a Python library utilizing Named Entity Recognition (NER) and synthetic data generation to identify and protect Personal Identifiable Information (PII). We discuss the architecture of NERPII and provide a concise tutorial on its application, demonstrating how to extract entity information from datasets containing personal data and synthesize new PII while preserving data characteristics. Additionally, the study discusses the library's potential contributions and implications for future research. In conclusion, NERPII emerges as a practical tool for addressing ethical concerns related to data privacy in the AI domain.
In the era of data-driven insights and technological progress, the intersection of Artificial
Intelligence (AI) and data privacy has assumed an increasingly important role. AI has
profoundly reshaped societal dynamics, influencing everything from finance and healthcare to
transportation and entertainment. However, this progress has not been devoid of ethical
concerns, particularly those related to the exposure of Personal Identifiable Information (PII) [
The protection of PII is a critical aspect of privacy and data security, as it involves safeguarding
individuals’ personal information from unauthorized access, use, or disclosure. Various laws
and regulations, such as the European Union’s General Data Protection Regulation (GDPR) [
In this perspective, some Natural Language Processing (NLP) techniques, such as Named
Entity Recognition (NER), a challenging learning problem that involves processing a text and
identifying certain occurrences of words or expressions as belonging to particular categories
of Named Entities (NE) [
So, NER, as mentioned before, is a technique often used on unstructured data, namely texts. However, companies or institutions often possess entire structured datasets, such as Excel or CSV files, containing numerous PII records. The challenge, therefore, was to create a tool capable of performing NER on tabular data in order to recognize PII and replacing it with synthetic data.
In literature, numerous tools have been created to associate semantic types with table columns.
For instance, Hulsebos et al. [
Although these tools have achieved high performance on state-of-the-art benchmarks, their architecture makes them computationally expensive. Furthermore, these tools only enable the recognition of entities and relationships within a table, without the capability to regenerate a potential PII once identified in order to ensure privacy.
Considering this, we introduce NERPII1, a Python library that leverages NER methods to efectively identify PII within structured data formats, such as CSV files, and subsequently regenerate them in a privacy-preserving manner. To perform NER, we used Presidio [14], a Microsoft SDK that provides a fast identification for private entities such as credit card numbers, names, locations, phone numbers, financial data and more, and a BERT model ( dslim/bert-baseNER) [15] to identify additional entities. The use of the BERT model thus makes the tool more robust, allowing for the expansion of the entities recognized by the library. Moreover, in order to generate synthetic PII, we use Faker [16], a Python library created to generated fake PII. The
idea is to assign an entity to each column which contains PII in a dataset, and then replace each values in that column with a coherent fake PII.
In this paper, we describe the architecture of NERPII and we show a short tutorial on how to use the library to extract entity information from a dataset containing personal information and to synthesize new PII that maintains data characteristics while safeguarding privacy. Finally, we discuss conclusion and future directions.
The NERPII library consists of two distinct classes: NamedEntityRecognizer and FakerGenerator.
NamedEntityRecognizer is used to perform Named Entity Recognition on structured data, typically in the form of a CSV file. An instance of NamedEntityRecognizer requires several parameters: a Pandas DataFrame containing the data to be subjected to NER, an optional count indicating the desired number of samples to be processed (with a default value of 500), and a NaN filler—a string used to fill NaN values in the dataset (set to ’?’ by default). Within the NamedEntityRecognizer, a Presidio analyzer is initialized, which attempts to assign a named entity from those supported by the library to each column of the dataset, and the BERT model that tries to assign the organization entity to the columns. To assign an entity to each column, the analyzer and the model first assign an entity to each value within the column. The entity ultimately assigned to the column is the one that has been most frequently assigned to the values contained within that column. When an instance of NamedEntityRecognizer is created, the NER is performed on the dataset, resulting in a dictionary accessible through the instance’s attribute dict_global_entities. Within this dictionary, for each column, the assigned entity and the confidence score with which that entity has been assigned are recorded.
FakerGenerator is used to generate synthetic PII. The parameters of a FakerGenerator object include the dataset previously analyzed by the NamedEntityRecognizer and the associated dict_global_entities dictionary. The generator divides columns for which the recognizer was able to assign an entity from those without an associated entity. For those columns with an associated entity, the generator replaces each value within the column with synthetic data. The FakerGenerator can regenerate the following Named Entities: address, phone number, email address, first and last name, city, state, URL, zipcode, credit card number, Social Security Number (SSN), and country.
Consequently, the output of the generator is a new dataset in which the originally present PII has been substituted with synthetic PII.
Suppose you have a dataset containing personal information of several people, such as first name, last name, phone number, address, e-mail, etc., like the one shown in Table 1. You need to anonymize the PII contained in the dataset, so firstly, you have to install the library by cloning the github repository or by simply installing it via pip. pip install nerpii Once you have installed the library, you can import the class NamedEntityRecognition by using the following line of code. from nerpii.named_entity_recognizer import NamedEntityRecognizer Then, you can create a recognizer passing as parameter the path to the CSV file that contained your dataset or directly your dataset in Pandas DataFrame format. recognizer = NamedEntityRecognizer(’./csv_path.csv’) Once you have created your recognizer, you can performed NER using the following functions. recognizer.assign_entities_with_presidio() recognizer.assign_entities_manually() recognizer.assign_organization_entity_with_model() These functions assign an entity to most of the columns. The final output is a dictionary, like the one shown below, accessible with recognizer.dict_global_entities in which column names are given as keys and assigned entities and a confidence score as values. {’first name’: {’entity’: ’PERSON’, ’confidence_score’: 0.9127725856697819}, ’last name’: {’entity’: ’PERSON’, ’confidence_score’: 0.8625}, ’address’: {’entity’: ’ADDRESS’, ’confidence_score’: 0.8926174496644296}, ’city’: {’entity’: ’LOCATION’, ’confidence_score’: 0.8731343283582089}, ’state’: {’entity’: ’LOCATION’, ’confidence_score’: 0.976}, ’zip’: {’entity’: ’ZIPCODE’, ’confidence_score’: 1.0}, ’phone’: {’entity’: ’PHONE_NUMBER’, ’confidence_score’: 0.888}, ’email’: {’entity’: ’EMAIL_ADDRESS’, ’confidence_score’: 1.0}} After performing NER on your dataset, you can generate new PII using Faker. You can import the class FakerGenerator by using the following command. from nerpii.faker_generator import FakerGenerator
generator = FakerGenerator(dataset, recognizer.dict_global_entities)
generator.get_faker_generation() At the end of the whole process you will have obtained a dataset, identical to the original (see Table 2), where the values in the various columns will have been replaced with synthetic PII.
Overall, the library has been tested on two openly available and two proprietary data sets. The Classic Models data set represents a Customer Relationship Management database 2, while the AWS Honeypot data set 3 comes from the cybersecurity domain. The proprietary data sets were a synthetic table depicting a financial fraud detection use case and a table containing user data collected by an IT department. The following table contains an overview of the metrics achieved on the aforementioned tests.
first name last name
The evolution of AI has undeniably transformed numerous sectors, contributing to the creation of several new technologies and showing rapid progress. However, this transformation has not been without its ethical implications, particularly regarding the exposure of sensitive personal data. The synthesis of data coupled with NLP techniques, ofers a promising solution that ensure the protection of PII.
NERPII, as detailed in this paper, tries to combine the remarkable potential of AI with the importance of data privacy. By employing NER methods and harnessing the power of synthetic data generation, the library adeptly identifies and categorizes PII within structured data formats. The integration of Presidio and a BERT model in the identification process, along with the usage of the Faker library for synthetic data generation, demonstrates a comprehensive and innovative approach.
The strength of NERPII lies in its ability to work with structured data, which is often overlooked by other solutions that focus more on unstructured data such as texts. Additionally, with NERPII, the privacy of personal data is guaranteed, without sacrificing the semantic component of the data. Indeed, there are other solutions, like Presidio itself, that allow for the identification of PII in texts and their anonymization using predefined or customizable tags. However, in our view, this approach results in a significant loss of informative content in the data. On the other hand, the use of synthetic data enables complete anonymization without any loss of meaning.
A promising direction for the future development of this library is to adapt it to other languages in addition to English, such as Italian, using Faker providers for the Italian language. Currently, the library is designed to recognize Named Entities in structured data in English and to regenerate PII in the American format (for example, if it needs to regenerate a Social Security Number, it will do so according to the American nine-digit format). Moreover, it could be useful to expand the number of Named Entities recognized by the NamedEntityRecognizer and those regenerated by the FakerGenerator. Finally, a future development could examine the aspect of coherence among diferent columns: in the current version of the library, data synthesis occurs independently for each column, despite it being evident that some columns are semantically correlated (for example, the column containing the city with the one containing the state).
In conclusion, therefore, NERPII can represent a practical solution for addressing some of the ethical issues related to data privacy in the field of artificial intelligence. [11] D. Zhang, Y. Suhara, J. Li, M. Hulsebos, Ç. Demiralp, W.-C. Tan, Sato: Contextual semantic type detection in tables, arXiv preprint arXiv:1911.06311 (2019). [12] X. Deng, H. Sun, A. Lees, Y. Wu, C. Yu, Turl: Table understanding through representation learning, ACM SIGMOD Record 51 (2022) 33–40. [13] Y. Suhara, J. Li, Y. Li, D. Zhang, Ç. Demiralp, C. Chen, W.-C. Tan, Annotating columns with pre-trained language models, in: Proceedings of the 2022 International Conference on Management of Data, 2022, pp. 1493–1503. [14] O. Mendels, C. Peled, N. Vaisman Levy, T. Rosenthal, L. Lahiani, et al., Microsoft Presidio: Context aware, pluggable and customizable pii anonymization service for text and images, 2018. URL: https://microsoft.github.io/presidio. [15] J. Devlin, M. Chang, K. Lee, K. Toutanova, BERT: pre-training of deep bidirectional transformers for language understanding, CoRR abs/1810.04805 (2018). URL: http://arxiv. org/abs/1810.04805. arXiv:1810.04805. [16] D. Faraglia, Other Contributors, Faker, 2014. URL: https://github.com/joke2k/faker. [17] P. M. Schwartz, D. J. Solove, The pii problem: Privacy and a new concept of personally identifiable information, NYUL rev. 86 (2011) 1814.