This paper is an extended abstract of a recent work, in which we introduce COSINER, a novel approach to enhancing Named Entity Recognition (NER) tasks through data augmentation. Unlike traditional methods that risk introducing noise, COSINER leverages context similarity to substitute entity mentions with more contextually appropriate ones, yielding superior performance in limited-data scenarios. Experimental results demonstrate COSINER's efectiveness over existing baselines, with computational times comparable to basic augmentation methods and superior to pre-trained model-based approaches.
like word replacement [1], random deletion [
To address this challenge, we present our method, COntext SImilarity-based data augmentation for NER (COSINER) [8], which utilizes similarity metrics to generate augmented examples that closely resemble real context. Our approach introduces a context-based mention replacement technique, substituting mentions in input data with entities from an Entity Lexicon that are contextually appropriate. In this paper, which is an extended abstract of our previous work [9], our contribution consists of the development of COSINER and an extensive evaluation across three prominent biomedical benchmark datasets that demonstrate COSINER’s superiority over existing methods, highlighting its general applicability beyond the biomedical domain. Notably, COSINER’s efectiveness is attributed to its ability to improve performance primarily through top-ranked samples, reducing reliance on large augmented datasets and enhancing computational eficiency.
COSINER utilizes mention replacement to expand the initial training set, a technique previously explored by Dai et al. [5]. While their method randomly substitutes entities within sentences using a binomial distribution, we introduce a systematic approach centered on similarity, where entity mentions are replaced with counterparts closely matching in syntax, semantics, and context. Despite the quadratic time complexity of our methodology, equal to (2) for computing cosine similarity between embeddings (with a size of ), the time spent generating new examples remains insignificant. Figure 1 provides an overview of our methodical flow, elaborated further in subsequent sections.
Lexicon generation In the training set, each entity, referred to as a , needs to be collected for replacement purposes. A can comprise one or a group of words, and we also record the frequency of each word’s appearance in the training set within the Lexicon . The size of the Lexicon varies depending on the number of mentions in the dataset. It is significant to emphasize that although the size of the Lexicon influences the speed of computing similarity values between entity pairs, this influence is not a constraint, particularly as we conduct experiments in few-shot scenarios.
Embeddings extraction In order to calculate entities similarities, it’s imperative to establish a comprehensive representation () for all Lexicon concepts, which serves as viable input for our predictive model. We employ a pre-trained language model as a feature extractor [10, 11] to process each phrase containing a given mention from the Lexicon, mapping each token to
Training set (augmentable sentences) Original Training set Dataset
Entity extraction
Lexicon
Embeddings extraction
Embeddings Similarity calculation
Similarity lists Training set (augmentable sentences)
Training set +
IR Validation and test sets
NER model
Results
Augmented set generation Augmented training set its word embedding (i.e. an array of numerical features representing the token in its context). In cases where mentions consist of multiple tokens, is obtained by averaging the word embeddings of all tokens.
Upon retrieving , the numerical representation of the concept is updated using the formula:
= + · (1 − sim) · , where denotes a regularization term determined by the inverse of the frequency of a mention across the entire dataset, and represents the cosine similarity between and . Initially, is set to the value of the first sentence where the mention appears.
Similarity computation We calculate the cosine similarity between the embeddings of every pair of entities in the Lexicon to derive a ranked list of similarity scores = sim(, ) associated with each Lexicon entry. We define two ranking criteria: 1)Maximum (descending order): Prioritizing concepts with the highest relatedness at the top of the list. This approach facilitates the generation of realistic augmented samples that uphold contextual consistency within sentences.
2)Minimum (ascending order): By initially considering the least similar entities, we encompass samples farthest from the knowledge boundary. This inclusion enables the recognition and accurate classification of extreme cases.
Augmented set generation The augmented set is constructed from all sentences featuring at least one mention. Each sentence is assigned a similarity value , , which is computed as the mean of entity similarity scores for the additional entities present within the sentence. We employ two strategies:
1) Local Augmentation: Each sentence results in the generation of new samples, ensuring the contribution of every training instance to the augmented set.
2) Global Augmentation: Similar to the previous strategy, new samples are generated for each sentence. Subsequently, we rank all newly generated sentences in a single list based on their similarity value and select the first ℎ elements.
In Figure 2 we emphasize the distinctions between the two strategies.
NER model training We adhere to the IOB2 scheme for the NER token-classification task [ 12]. The original training dataset and the augmented samples are fed into a Transformer network backbone [10, 11]. Model parameters undergo optimization via cross-entropy minimization.
We conduct training and evaluation on three renowned benchmark datasets sourced from biomedical articles: i) NCBI-Disease [13]: Comprising 793 PubMed abstracts, with 6,881 disease entities, ii) BC5CDR [14]: Comprising 1,500 PubMed articles, containing 15,935 chemical mentions, and BC2GM [15]: Comprising 20,000 sentences extracted from PubMed abstracts, involving 20,702 gene entities.
We delineate three distinct few-shot scenarios, each characterized by the percentage of samples drawn from the available corpora employed in implementing our methods: specifically, 2%, 5%, and 10%. Subsequently, we present all experimental findings within these aforementioned few-shot scenarios. Dataset statistics and few-shot scenarios details are summarized in Table 1. 3.1. Hyperparameter tuning Table 2 presents results achieved using various parameter configurations for similarity computation (Maximum vs Minimum) and augmented set generation (Local vs Global), as discussed in Section 2. As anticipated, employing Maximum similarity computation generally yields superior performance, as augmented samples are plausible and closer to the test distribution. Nevertheless, the notable performance achieved with the Minimum configuration suggests that at times, considering "distant" entities may prove beneficial in expanding the NER model’s scope. Regarding augmented set generation, the Local criterion typically outperforms, owing to its augmentation of all sentences in the original dataset. In summary, it’s noteworthy that Maximum local emerges as the most favorable overall strategy.
When creating an augmented dataset, the quantity of augmented samples is a crucial parameter to consider. Therefore, we conducted experiments using three distinct budgets for the augmented set: small (100 samples), medium (300 samples), and large (500 samples).
Figure 3 illustrates the results obtained across the three benchmark datasets. Due to the similarity-based approach, which prioritizes the most informative examples in the top-ranked positions, there is minimal discrepancy observed when using higher budgets.
We contrast our top-performing results with baselines drawn from current literature [5], as follows: • No Augmentation: Results obtained using the original training set assessed with a BERT 0.8
In this study, we have employed a context similarity-based approach to generate augmented data, aiming to enhance the performance of NER tasks while mitigating the adverse efects of noisy and mislabeled data commonly encountered with existing techniques.
Our experiments conducted in the medical domain, where data augmentation is particularly crucial, underscore the eficacy of our method. We have demonstrated its superiority over several state-of-the-art baselines, achieving comparable or improved execution times.
Looking ahead, our approach holds promise for integration with complementary techniques beyond Mention Replacement. Future investigations will explore its applicability across diverse contexts and with various entity types, fostering a deeper understanding of its potential and versatility.
E66G22000400009). tificial Intelligence Research (FAIR)” – CUP E63C22002150007 This work has been funded by the project NextGenerationEU via PNRR - DM We acknowledge financial support from the PNRR project “Future Ar352 (CUP: instance learning for neural dialogue generation via learning to augment and reweight, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, text classification tasks, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019, pp. 6382–6388. [3] J. Min, R. T. McCoy, D. Das, E. Pitler, T. Linzen, Syntactic data augmentation increases robustness to inference heuristics, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, [4] K. M. Yoo, Y. Shin, S.-g. Lee, Data augmentation for spoken language understanding via joint variational generation, in: Proceedings of the AAAI conference on artificial intelligence, volume 33, 2019, pp. 7402–7409. [5] X. Dai, H. Adel, An analysis of simple data augmentation for named entity recognition, in: Proceedings of the 28th International Conference on Computational Linguistics, International Committee on Computational Linguistics, Barcelona, Spain (Online), 2020, pp. 3861–3867. [6] S. Chen, G. Aguilar, L. Neves, T. Solorio, Data augmentation for cross-domain named entity recognition, in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 2021, pp. 5346–5356. [7] R. Zhou, X. Li, R. He, L. Bing, E. Cambria, L. Si, C. Miao, Melm: Data augmentation with masked entity language modeling for low-resource ner, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 2251–2262. [8] I. Bartolini, V. Moscato, M. Postiglione, G. Sperlì, A. Vignali, Cosiner: Context similarity data augmentation for named entity recognition, in: International Conference on Similarity Search and Applications, Springer, 2022, pp. 11–24. [9] I. Bartolini, V. Moscato, M. Postiglione, G. Sperlì, A. Vignali, Data augmentation via context similarity: An application to biomedical named entity recognition, Information Systems 119 (2023) 102291. [10] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1, Association for Computational Linguistics, Minneapolis, Minnesota, 2019, pp. 4171–4186. [11] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, D. Amodei, Language models are few-shot learners, in: H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, H. Lin (Eds.), Advances in Neural Information Processing Systems, volume 33, Curran Associates, Inc., 2020, pp. 1877–1901. [12] L. A. Ramshaw, M. P. Marcus, Text chunking using transformation-based learning, in:
Natural language processing using very large corpora, Springer, 1999, pp. 157–176. [13] R. Doğan, R. Leaman, Z. Lu, NCBI disease corpus: a resource for disease name recognition and concept normalization, 2014. [14] J. Li, Y. Sun, R. Johnson, D. Sciaky, C. Wei, R. Leaman, A. Davis, C. Mattingly, T. Wiegers, Z. Lu, BioCreative V CDR task corpus: a resource for chemical disease relation extraction, 2016. [15] L. Smith, L. Tanabe, R. Ando et al., The BioCreative II - Critical Assessment for Information
Extraction in Biology Challenge, 2008. [16] G. A. Miller, Wordnet: A lexical database for english, Commun. ACM 38 (1995) 39–41.