Guardians of the Data: NER and LLMs for Effective Medical Record Anonymization in Brazilian Portuguese

Mauricio Schiezaro¹Guilherme Rosa^1*Bruno Augusto Goulart Campos²Helio Pedrini^3*

• ¹Venturus Centro de Inovacao Tecnologica, Campinas, Brazil
• ²Universidade Estadual de Campinas Faculdade de Ciencias Medicas, Campinas, Brazil
• ³Universidade Estadual de Campinas Instituto de Computacao, Campinas, Brazil

The anonymization of medical records is essential to protect patient privacy while enabling the use of clinical data for research and Natural Language Processing (NLP) applications. However, for Brazilian Portuguese, the lack of publicly available and high-quality anonymized datasets limits progress in this area. In this study, we present AnonyMed-BR, a novel dataset of Brazilian medical records that includes both real and synthetic samples, manually annotated to identify personally identifiable information (PII) such as names, dates, locations, and healthcare identifiers. To benchmark our dataset and assess anonymization performance, we evaluate two anonymization strategies: (i) an extractive strategy based on Named Entity Recognition (NER) using BERT-based models, and (ii) a generative strategy using T5-based and GPT-based models to rewrite texts while masking sensitive entities. We conduct a comprehensive series of experiments to evaluate and compare anonymization strategies. Specifically, we assess the impact of incorporating synthetic generated records on model performance by contrasting models fine-tuned solely on real data with those fine-tuned on synthetic samples. We also investigate whether pretraining on biomedical corpora or task-specific fine-tuning more effectively improves performance in the anonymization task. Finally, to support robust evaluation, we introduce an LLM-as-a-Judge framework that leverages a reasoning Large Language Model (LLM) to score anonymization quality, estimate information loss, and assess reidentification risk. Model performance was primarily evaluated using the F1 score on a held-out test set. All evaluated models achieved good performance in the anonymization task, with the best models reaching F1 scores above 0.90. Both extractive and generative approaches were effective in identifying and masking sensitive entities while preserving the clinical meaning of the texts. Experiments also revealed that including synthetic data improved model generalization, and that task-specific fine-tuning yielded greater performance gains than pretraining the model on biomedical domain. To the best of our knowledge, AnonyMed-BR is the first manually annotated anonymization dataset for Brazilian Portuguese medical texts, enabling systematic evaluation of both extractive and generative models.