We systematically assess the performance of three leading API-based de-identi!cation systems-Azure Health Data Services, AWS Comprehend Medical, and OpenAI GPT-4o-against our de-identi!cation systems on a ground truth dataset of 48 clinical documents annotated by medical experts. Our analysis, conducted at both entity-level and token-level, demonstrates that our solution, Healthcare NLP, achieves the highest accuracy, with a 96% F1-score in protected health information (PHI) detection, signi!cantly outperforming Azure (91%), AWS (83%), and GPT-4o (79%). Beyond accuracy, Healthcare NLP is also the most cost-e"ective solution, reducing processing costs by over 80% compared to Azure and GPT-4o. Its !xed-cost local deployment model avoids the escalating per-request fees of cloud-based services, making it a scalable and economical choice. Our results underscore a critical limitation: zero-shot commercial APIs fail to meet the accuracy, adaptability, and cost-e#ciency required for regulatory-grade clinical de-identi!cation. Healthcare NLP's superior performance, customization capabilities, and economic advantages position it as the more viable solution for healthcare organizations seeking compliance and scalability in clinical NLP work$ows.
Electronic Health Records (EHRs) are now widespread across the United States healthcare
system, with adoption rates surpassing 96% in acute care hospitals and 86% among o#ce-based
physicians [
Given the highly sensitive nature of this data, it must undergo a de-identi!cation process
before use. De-identi!cation involves removing or obscuring personal health information (PHI)
from medical records to protect patient privacy. De-identi!ed data refers to health information
that has been stripped of all “direct identi!ers”—elements that could uniquely identify an
individual. The Health Insurance Portability and Accountability Act (HIPAA) Safe Harbor
guidelines de!ne 18 such direct identi!ers (U.S. Department of Health & Human Services, 2023
) [
Recent studies suggest that deep learning-based automated de-identi!cation models can
surpass human annotators in identifying PHI, with hybrid approaches demonstrating the
greatest potential [
This study examines the performance and compares de-identi!cation services, developed by us and named as Healthcare NLP library, AWS Comprehend Medical, and Azure Health Data Services, with a focus on their accuracy when applied to a dataset annotated by healthcare experts. The comparison of these services provides valuable insights into their respective strengths and limitations, enabling informed decision-making for researchers, developers, and organizations seeking appropriate de-identi!cation tools. Additionally, this comprehensive analysis equips stakeholders with the necessary information to select the most suitable tool based on accuracy, compliance, cost-e"ectiveness, and scalability for processing sensitive healthcare data.
For researchers, this analysis helps identify the most accurate, reliable, and cost-e"ective
service for processing sensitive data, which is crucial for maintaining data integrity in clinical
studies. Developers bene!t from understanding the ease of integration and API $exibility of
each service, essential factors for building scalable solutions that can handle large volumes of
clinical data [
The comparison highlights variations in performance among the evaluated services. Our Healthcare NLP library achieved the highest accuracy, with macro and weighted average F1scores of 96% and 99%, respectively, followed by Azure Health Data Services with 85% macro and 99% weighted average F1-scores, and AWS Comprehend Medical with 80% macro and 98% weighted average F1-score. However, performance may vary based on speci!c use cases and dataset characteristics. Additionally, a cost analysis for processing one million clinical notes (each containing 5,250 characters) revealed that the Healthcare NLP library is the most cost-e"ective option, followed by Azure Health Data Services and AWS Comprehend Medical.
The de-identi!cation of unstructured data has been extensively studied, with various Natural
Language Processing (NLP) approaches proposed over the years [
Early de-identi!cation systems in the clinical domain were predominantly rule-based, as seen
in the work of Sweeney [
The !eld of automated PHI detection and de-identi!cation has seen signi!cant advancements
in recent years, with several major cloud providers and specialized services o"ering solutions
to address the growing need for secure handling of sensitive healthcare data. The concept of
automatic de-identi!cation gained prominence in 2014 through the Informatics for Integrating
Biology and the Bedside (i2b2) project, which introduced a pioneering academic NLP challenge
focused on automatically detecting PHI identi!ers from medical records [
Recent research suggests that deep learning-based automated de-identi!cation models can
surpass human annotators in PHI identi!cation, with hybrid approaches demonstrating the
greatest potential [
A notable study by Steinkamp et al. [
Recent advancements in Large Language Models (LLMs) have prompted researchers to
investigate their potential for de-identifying clinical notes. A study by Altalla et al. [
Despite these promising results, the application of LLMs for de-identi!cation presents several
challenges. The nascent stage of LLM utilization in this !eld raises concerns regarding the
privacy and security of health data, particularly when employing API-based models [
This study aims to contribute to previous performance comparisons in PHI entity recognition and assist researchers and decision-makers in selecting the most suitable tool for processing large-scale datasets with high accuracy and cost-e"ectiveness. To achieve this, we compare three widely used and advanced de-identi!cation tools that incorporate state-of-the-art models while ensuring consistency: Our Healthcare NLP library, Azure Health Data Services, AWS Comprehend Medical and GPT4o, a state-of-the-art commercial multi-modal LLM.
In this section, we will provide brief information for each de-identi!cation solution that supports di"erent set of PHI entities. The list of PHI entities supported by each model is shared in Table A2.
The Healthcare NLP library is a powerful component of Spark NLP platform [
Additionally, the library features custom large language models (LLMs) in various sizes and quantization levels for tasks like medical note summarization, question answering, retrievalaugmented generation (RAG), and healthcare-related conversational interactions. It also provides a robust solution for de-identifying medical records using advanced NER models to automatically detect and remove PHI from clinical notes. This ensures compliance with privacy regulations while preserving data utility for research, enabling secure data sharing, enhancing patient privacy, and promoting innovation in medical research.
The Healthcare NLP library allows users to create custom de-identi!cation pipelines
targeting speci!c labels or to utilize pre-trained pipelines with two lines of code to de-identify a
broad range of entities. These entities include AGE, CONTACT, DATE, ID, LOCATION, NAME,
PROFESSION, CITY, COUNTRY, DOCTOR, HOSPITAL, IDNUM, MEDICALRECORD,
ORGANIZATION, PATIENT, PHONE, STREET, USERNAME, ZIP, ACCOUNT, LICENSE, VIN, SSN,
DLN, PLATE, IPADDR, EMAIL, and more. In Beyond Accuracy: Automated De-Identi!cation of
Large Real-World Clinical Text Datasets [
In this study, a pre-trained de-identi!cation pipeline was utilized, speci!cally designed to extract and de-identify entities such as NAME, IDNUM, CONTACT, LOCATION, AGE, and DATE. Notably, this pipeline operates independently of any large language model (LLM) components.
Azure Health Data Services’ de-identi!cation service is designed to safeguard sensitive health information while maintaining data utility. This API employs advanced natural language processing techniques to identify, label, redact, or surrogate PHI in unstructured medical texts. The service provides three essential operations: Tag, Redact, and Surrogate, which allow healthcare organizations to process various types of clinical documents securely and e#ciently. By utilizing machine learning algorithms, the service can detect HIPAA’s 18 identi!ers and other PHI entities, ensuring compliance with various regional privacy regulations such as GDPR and CCPA.
Amazon Comprehend Medical is a HIPAA-eligible natural language processing (NLP) service that leverages machine learning to extract valuable health data from unstructured medical text. This tool quickly and accurately identi!es medical entities such as conditions, medications, dosages, tests, treatments, and Protected Health Information (PHI) from various clinical documents including physician’s notes, discharge summaries, and test results. With its ability to understand context and relationships between extracted information, AWS Comprehend Medical o"ers a robust solution for healthcare professionals and researchers looking to automate data extraction, improve patient care, and streamline clinical work$ows.
GPT-4o is a multi-modal model that o"ers improvements in response times and classi!cation accuracy compared to GPT-4, which could theoretically enhance the precision of identifying and redacting sensitive information via prompting. While GPT-3.5 and GPT-4 have been extensively studied for their de-identi!cation capabilities, particularly in processing medical text, GPT-4o presents an intriguing option due to its enhanced performance over GPT-4 in various tasks. However, no formal study has yet evaluated GPT-4o’s de-identi!cation capabilities. Given the importance of PHI redaction in healthcare AI applications, understanding the model’s strengths and limitations in this area remains crucial. Despite these advantages, its e"ectiveness in deidenti!cation remains speculative without empirical studies directly assessing its performance. While there are cost-e"ective alternatives for de-identi!cation, we opted for GPT-4o due to its widespread adoption, strong presence in research, and its demonstrated advancements over previous models.
The annotation of patient identi!ers within clinical data is a critical process in healthcare
research and data management. This study employed a comprehensive annotation methodology
utilizing the John Snow Labs’ Annotation Lab software, which facilitated a multi-stage approach
to entity recognition and labeling. The process began with a pre-annotation step using deep
learning models to extract initial entities, followed by human re!nement guided by a dynamic
annotation guide. This iterative approach, involving multiple rounds of review and correction,
ensured high accuracy and adaptability throughout the !ne-tuning and evaluation phases [
The dataset employed in this study comprised 48 clinical notes meticulously annotated by our domain experts. The dataset was speci!cally curated to facilitate the evaluation of deidenti!cation systems in a healthcare context. Expert annotations focused on six key entity types: IDNUM, LOCATION, DATE, AGE, NAME, and CONTACT. These entities represent critical categories of Protected Health Information (PHI) that are commonly subject to de-identi!cation under regulatory frameworks such as the Health Insurance Portability and Accountability Act (HIPAA) and the General Data Protection Regulation (GDPR).
The selection of these entity types was motivated by their frequent occurrence in clinical narratives and their signi!cance in ensuring patient privacy. Identi!ers such as patient names, contact details, and unique ID numbers pose a high risk of re-identi!cation if not properly anonymized. Similarly, location information, age, and date-related details can contribute to indirect re-identi!cation, necessitating robust de-identi!cation strategies. By centering the benchmark on these entities, this study ensures that the performance evaluation remains directly aligned with real-world de-identi!cation challenges in healthcare settings.
To enhance reproducibility, the benchmark dataset utilized in this study has been made
publicly available in a dedicated repository[
The most signi!cant di"erence between these tools lies in their adaptability. Azure Health Data Services, Amazon Comprehend Medical and GPT-4o are API-based, black-box cloud solutions, making modifying or adapting results to speci!c needs impossible. On the other hand, the Healthcare NLP library’s de-identi!cation pipeline can be loaded and utilized with just two lines of code. The pipeline outputs can be customized by adjusting its stages to meet speci!c needs, and it can also be used locally with no internet connection.
In this benchmark study, we employed two distinct approaches to compare accuracy:
Since de-identifying PHI data is a critical task, we evaluated how well de-identi!cation tools detected entities present in the annotated dataset, regardless of their speci!c labels in the ground truth. The detection outcomes were categorized as: • full_match: The entire entity was correctly detected. • partial_match: Only a portion of the entity was detected.
• not_matched: The entity was not detected at all.
For example, for the text: “Patient John Doe was admitted to Boston General Hospital on 01/12/2023.”, the ground truth entity “John Doe (NAME)” could have the following predicted entities: • Predicted Entity: “John Doe (NAME)” ==> full_match • Predicted Entity: “John” ==> partial_match • Predicted Entity: “Patient” ==> not_matched
For evaluation results, please refer to Figure A1 in the Appendix section.
The text in the annotated dataset was tokenized, and the ground truth labels assigned to each token were compared with predictions made by the Healthcare NLP library, Amazon Comprehend Medical, Azure Health Data Services, and GPT-4o model. Classi!cation reports were generated for each tool, comparing their precision, recall, and F1 scores. Token-level evaluation results are presented in Figure A2 in the Appendix section.
In this study, di"erences were observed between the predictions generated by the deidenti!cation services and the ground truth annotations. The ground truth dataset utilized generic entity labels; for instance, all names were annotated as NAME, rather than distinguishing between PATIENT_NAME and DOCTOR_NAME. To ensure consistency in evaluation, the predicted labels from the de-identi!cation tools were mapped to their corresponding ground truth labels.
To maintain a fair comparison, entities that did not have a direct mapping to the ground truth labels—such as PROFESSION, ORGANIZATION, and other non-essential entity types—were excluded from the predictions before conducting the performance evaluation. This preprocessing step ensured that the assessment focused solely on the six critical entity types relevant to healthcare de-identi!cation. Entity mapping table showing entity mapping across di"erent providers can be seen at Table A3. After obtaining the model predictions and applying the preprocessing steps, the entity distribution was summarized in Table A5. While evaluating GPT4o, we used a one-shot prompt to provide the model some sample PHI entity extraction tasks (the prompt is shared in the Appendix).The model was con!gured with a temperature of 1 and executed as a single run, while all other parameters were maintained at their default settings to ensure consistency in evaluation.
The !nal results can be found at Table 1. The entity-level and token-level evaluations including comparative analyses and benchmark scores can be found in the Appendix.
The primary objective of de-identi!cation is to accurately detect PHI entities. In this regard, we also wanted to evaluate binary classi!cation performance in which entities were classi!ed as either PHI or non-PHI, disregarding speci!c subcategories. The PHI entity detection results are also summarized in Table 1 and Figure 2.
Cost is a critical factor when processing large-scale clinical datasets. To estimate expenses, we simulated the cost of de-identifying 1 million unstructured clinical notes, each averaging 5,250 characters.
The pricing estimates are as follows: • Amazon Comprehend Medical: Processing 1M documents costs approximately $14,525. • Azure Health Data Services: Processing 1M documents costs approximately $13,125. • Open AI GPT-4o: Processing 1M documents costs approximately $21,400. • Healthcare NLP: Using John Snow Labs’ Healthcare NLP Prepaid on an EC2 c6a.8xlarge instance ($1.2/hour), de-identifying PHI from 48 documents took 39.4 seconds. Extrapolating, processing 1M documents would take approximately 228 hours (9.5 days), but with proper scaling, it could be completed in a single day. The total estimated cost: – Infrastructure: $273 – License: $2,145 (if one-month license cost set to $7,000) – Total: $2,418
In this study, we conducted a comparative analysis of the performance of Healthcare NLP, Amazon Comprehend Medical, Azure Health Data Services, and Open AI GPT-4o model on a ground truth dataset annotated by medical experts. The evaluation was performed at two levels: entity-level and token-level.
The entity-level analysis demonstrated that Healthcare NLP outperformed its counterparts in accurately capturing entities while minimizing missed detections. Azure Health Data Services exhibited the second-best performance, followed by Amazon Comprehend Medical. The GPT-4o model ranked fourth in this comparative assessment.
The token-level evaluation further reinforced these !ndings, with Healthcare NLP achieving the highest precision, recall, and F1-score. Azure Health Data Services, Amazon Comprehend Medical and GPT-4o followed in that order, indicating a consistent pattern of superior performance for Healthcare NLP across both evaluation metrics.
A key di"erentiator among these tools is their adaptability. While Azure Health Data Services, Amazon Comprehend Medical and GPT-4o function as API-based, black-box cloud solutions with no customization capabilities, Healthcare NLP provides a $exible and transparent framework. Its de-identi!cation pipeline can be implemented with minimal coding e"ort, and users can modify pipeline stages to tailor the output to their speci!c requirements.
From a cost-e"ectiveness perspective, Healthcare NLP emerges as the most viable solution for large-scale clinical data processing. Unlike cloud-based services, which impose per-request pricing that escalates with increasing data volumes, Healthcare NLP allows for !xed-cost, local deployment. Even when processing substantial datasets, such as one billion clinical notes, its pricing remains stable over the same time period, providing a signi!cant economic advantage over API-based alternatives.
In summary, Healthcare NLP consistently outperformed Azure Health Data Services, Amazon Comprehend Medical, and GPT-4o across all evaluation metrics by 5-10%, achieving the highest accuracy while minimizing missed detections. Beyond its superior performance, its adaptability o"ers a crucial advantage over the black-box nature of cloud solutions, enabling users to customize de-identi!cation pipelines to meet speci!c needs. Furthermore, its cost-e"ective deployment model presents substantial savings, making it a compelling alternative to API-based solutions.
The original text with identifiable information
He is a 60-year-old male.
Original
Healthcare NLP Library Azure Health Data Services
AGE, CONTACT, DATE, ID, LOCA- Highly flexible; the de-identification TION, NAME, PROFESSION, CITY, pipeline can be easily loaded with two lines COUNTRY, DOCTOR, HOSPITAL, of code and customized to meet specific IDNUM, MEDICALRECORD, OR- requirements. Additionally, it can be used GANIZATION, PATIENT, PHONE, locally.
STREET, USERNAME, ZIP, ACCOUNT, LICENSE, VIN, SSN, DLN, PLATE, IPADDR, EMAIL DATE, DOCTOR, HOSPITAL, IDNUM, PATIENT, MEDICALRECORD, PHONE, AGE, STREET, STATE, CITY, HEALTHPLAN, PROFESSION, ZIP, EMAIL, ORGANIZATION, USERNAME, FAX, URL, LOCATIONOTHER, ACCOUNT, COUNTRYORREGION, SOCIALSECURITY
API-based, black-box solution; no direct control over results; suitable for integrated, cloud-based environments but lacks flexibility for task-specific adjustments.
AWS Compre- DATE, NAME, ADDRESS, ID, AGE, API-based, black-box solution; dehend Medical PHONE_OR_FAX, PROFESSION, identification is limited to specific URL, EMAIL pre-configured models; lacks customization and flexibility for adapting results to specific needs.
GPT-4o
No pre-built set of entities
API-based, black-box solution; identification is run via prompting.
de
The results obtained by comparing the predictions made by Healthcare NLP, AWS Comprehend Medical, and Azure Health Data Services with the ground truth entities are presented below. Figure A1: Entity Level Evaluation
To further analyze the performance of each de-identi!cation tool, a token-level evaluation was conducted. This involved tokenizing the ground truth text and associating each token with the corresponding predicted labels from Healthcare NLP, Amazon Comprehend Medical, Azure Health Data Services and GPT-4o.
Figure A2: Token Level Evaluation Table A4 Match Statistics for Healthcare NLP, Azure, AWS, and GPT-4o Predictions. The table shows the number of matches and their corresponding percentages for the di!erent prediction models.
Full Match Partial Match Not Matched
1342 (90.7%) 124 (8.4%) 13 (0.9%)
1258 (85.0%) 164 (11.1%) 57 (3.8%)
AWS 1108 (74.9%) 219 (14.8%) 152 (10.3%)
GPT-4o 983 (66.5%) 280 (18.9%) 216 (14.6%) You are an expert medical annotator with extensive experience in labeling medical entities within clinical texts. Your role is to accurately identify and annotate Protected Health Information (PHI) entities in the provided text, following the specified entity types.
– IDNUM, LOCATION, DATE, AGE, NAME, CONTACT [ {{’begin’: 24, ’end’: 30, ’entity_type’: ’DATE’, ’chunk’: ’2/16/69’}} {{’begin’: 42, ’end’: 45, ’entity_type’: ’NAME’, ’chunk’: ’Hale’}} {{’begin’: 50, ’end’: 67, ’entity_type’: ’LOCATION’, ’chunk’: ’Senior Care Clinic’}} {{’begin’: 69, ’end’: 79, ’entity_type’: ’LOCATION’, ’chunk’: ’Queen Creek’}} {{’begin’: 83, ’end’: 84, ’entity_type’: ’LOCATION’, ’chunk’: ’SD’}} {{’begin’: 96, ’end’: 105, ’entity_type’: ’NAME’, ’chunk’: ’Terri Bird’}} ] — Task: Extract all PHI entities from the text below. The entity types to identify are: IDNUM, LOCATION, DATE, AGE, NAME, CONTACT.
Expected Output Format: { entities:[ {’begin’: <start_index>, ’end’: <end_index>, ’entity_type’: ’<entity_type>’, ’chunk’: ’<extracted_text>’} ] } — Text to Annotate: {text} —
Your Response: