Acronym Disambiguation (AD) aims to find the correct expansions of an ambiguous acronym in a given sentence, which is essential for scientific document understanding tasks. In supervised AD, a significant challenge is to classify the meaning of most words under low resource conditions. For example, 82.64% of the annotated acronym examples in the legal AD training data are less than 15. This problem becomes more apparent when the distribution of words and senses is unbalanced. In this paper, we propose ANACONDA, an Adversarial training framework with iNtrust loss in ACrONym DisambiguAtion. Experiments on Legal English show the efectiveness of our proposed methods, and our score ranks 1st in SDU@AAAI-22 shared task 2: Acronym Disambiguation.
nents of a phrase or name, called the expansion [
Acronym disambiguation (AD) aims to determine the correct long form of an ambiguous acronym in a given text [3]. It is usually formulated as a sequence classification problem in general [5]. Figure 1 is an example of damental to the implementation of the NEPAD priorities of political, economic and corporate governance, a central element in strengthening Africa’s ownership of NEPAD and Input: -Sentence: GPS The Mechanism is fundamental to the implementation of the NEPAD priorities of political, economic and corporate governance, a central element in strengthening Africa's ownership of NEPAD and a means of attracting support from development partners. -Dictionary: GPS: 1. global positioning system 2. Governance, peace and security hand-designed rules [7], hand-made functions [8], word embedding [9] and pre-training techniques [10]. However, due to the lack of high-quality annotation data and the heavy expertise and workload required to expand these materials, the potential of these methods is severely limited. This problem has been afecting many tasks of NLP for a long time, primarily related to word sense disambiguation [11], because the granularity of word meaning is very fine, and it is often dificult to distinguish. If the distribution in the corpus is not balanced, it will further aggravate the dificulty of AD classification. data have problems such as lack of suficient labelled samples, complex samples (their meaning very close), and unbalanced data distribution. These problems make it dififcult for the model to predict the meaning of acronyms correctly. Therefore, we adopt a dynamic curriculum learning method to dynamically extract complex samples (model predicted error and low-confidence data) from the training data and add them to the training process to let the model learn several times. In addition, we also use adversarial training techniques to improve the robustness of the model. Finally, diferent from the general cross-entropy loss function, we use the enhanced In-trust loss [12] function to improve the model’s generalisation ability further.
The main contributions of this work are summarized as follows: • We analyze and found several problems that make AD tasks hard to improve, including complex samples, unbalanced data distribution, and provide solutions. • We propose ANACONDA, Adversarial training with iNtrust loss in ACrONym DisambiguAtion. This method helps the model learn dificult samples of acronyms and improve the robustness of the model. • Experiments conducted on the legal English dataset demonstrate that the proposed method has better performance and outperforms other competitive baselines.
1-5 6-10 11-15
>15
We have done some analysis on legal English data. As As the core task of natural language processing, word shown in Figure 2, 82.64% of acronyms’ samples appear sense disambiguation has been extensively studied. Some less than 15 times. There are many acronyms with multi- work has been used to deal with the lack of labelling data. ple possible extensions, which means that each extension Early work used WordNet’s lexical relationships, especan only have two or fewer examples to learn. It brings cially the singular and plural kinship relationships of dificulties to acronym disambiguation. At the same time, polysemous words, to calculate correlations [13]. Althe unbalanced sample distribution of acronyms is also though these methods prove their ability to generate a significant problem. As shown in Figure 3, acronyms new training examples, they still need to be improved with less than 15 samples have an average possible ex- in cross-language and domain expansion. Later work pansion of about 2.5. The acronyms with more than 15 designed features to build a classifier for specific words samples have more than four possible expansions on aver- [14]. There are also studies to solve these problems using age. If more expansions of acronyms need more samples, parallel corpora [15] or multilingual knowledge bases then the number of samples between 11-15 should be [16]. Recently, Stengel-Eskin proposed a neural discrimmore than 0-5, but that is not the case. The emergence inant architecture for word alignment and applied it to of this situation also brings dificulties to the acronym Chinese NER label propagation [17]. As far as we know, disambiguation. this work is the first to use neural word alignment to
In this paper, we propose ANACONDA, Adversarial project word meanings across languages, which is of reftraining with iNtrust loss in ACrONym DisambiguAtion. erence significance. Other work explored more lexical Our purpose is to help the model learn complex samples resources, such as knowledge graph structure [18]. of acronyms and improve the robustness of the model.
Specifically, after analyzing the data, we found that some
Acronym disambiguation aims to find the correct meaning of an ambiguous acronym in a given sentence. The input = 1, 2, … , is a sentence containing an ambiguous acronym, where is the total length of the sentence and the acronym is . The dictionary contains all possible extensions =
1, 2, … , corresponding to the acronym, represents the total length of the probable sentences. The systems are expected to find the correct expanded form of the acronym given the possible expansions for the acronym.
tively.
set is divided into training (2949), development (385) and testing (383). The training and validation sets of the legal
curriculum learning [21] method to get hard instances in the input data. Then we send the hard instances to the model for numerous training. At the same time, we also use adversarial training [22], including Fast Gradi(PGD) [24] methods, to increase the learning dificulty of Precision =
∑=1 precision Recall = ∑=1 recall
Macro F1 = 2 × Precision × Recall
Precision + Recall where is the number of total classes, the (1) (2) (3) and represent the precision and recall of class respec- ent Method (FGM) [23] and Projected Gradient Descent simple samples and improve the robustness of the model. The common method in adversarial training is the In addition, diferent from the traditional cross-entropy Fast Gradient Method (FGM) [23]. The idea of FGM is loss function, we use an enhanced In-trust [12] loss func- straightforward. Increasing the loss is to increase the tion in our task to further improve the model’s ability gradient so that we can take to identify acronyms and expand correctly. Finally, we merge the results obtained by the two models to achieve Δ = ∇ (, ; ) (4) the best disambiguation efect.
the characteristics of human learning. The learning materials of humans and animals are presented in the order of easy to dificult so that the learning efect will be better. Learning the curriculum from simple to complex (in this task are samples that are easy to understand and samples that are not easy to learn), so that it is easy for the model to find a better local optimum, and at the same time speed up the training. Specifically, we send sentences = { 1, 2, … , } into the models and and get the prediction result = (, ) , where is the size of a batch and represents whether the prediction is correct, and represents the prediction confidence. We collect and classify each model’s prediction error and low confidence instances , as a hard instance , and then add it to the training set again. Through repeated learning, the model will learn the features of complex cases and improve model prediction accuracy. The dynamics are embodied in that as the training deepens, the model will choose diferently for hard instances. Therefore, we will dynamically update the set of hard instances in each epoch.
In recent years, with the increasing development and implementation of deep learning, adversarial training [22] have also received more and more attention. In NLP, adversarial training is more used as a regularization method to improve the generalization ability1 of the model.
(5) (6) (7)
is the model parameter, (, ; ) is the loss of a single sample, Δ is the anti-disturbance.
Of course, to prevent Δ from being too large, it is usually necessary to standardize ∇ (, ; ) . The more common way is Δ = ∇ (, ; ) ‖∇ (, ; ) ‖
Gradient Descent (PGD) [24], which uses multiple iterations to achieve a larger Δ for ( + Δ, ; ) .
not all data contribute to models’ generalization. Crossentropy loss is not a good loss function when the data distribution is unbalanced or noisy, especially when the model is over-fitting. Incomplete-Trust (In-trust) [ 12] loss function, which boosts with a robust Distrust Cross-Entropy (DCE) term, can efectively alleviate the overfitting caused by previous loss function.
= − log( + (1 − )) In-trust = +
We made further improvements and changed the to a more appropriate for the previous task, which further improved the efect.
In this section we will introduce baseline models, experimental settings and results.
is far worse than the pre-trained model, and its generalization ability is poor. It can only deal with some pre• Rule-based method The baseline method pro- defined acronym ambiguity mechanically. Among the posed by Schwartz is a rule-based method [7]. three pre-training models, RoBERTa has the worst efect. In this baseline, the similarity of the candidate LegalBERT has been pre-trained on many texts in the long-forms with the sample text (in terms of sev- legal field, so it has the best performance and can better eral over- lapping words) is first computed. Then, identify the ambiguity of acronyms in legal English. Due the long-form with the highest similarity score is to the similarity between legal texts and scientific literchosen as the final prediction. The related codes ature, SciBERT, trained on a large amount of scientific can be found on the website 1. documents, performs well. Our experiments show that • LegalBERT model The LegalBERT [19] model dynamic curriculum learning, adversarial training and is a domain-specific pre-trained language model enhanced In-trust loss function methods are efective pre-trained on a large number of legal texts. The for this task. Dynamic curriculum learning can help the architecture of LegalBERT follows the same archi- model learn the features of hard instances. Adversarial tecture as BERT [25] to capture a well-formed rep- training improves the learning dificulty of simple samresentation of legal data. This model has achieved ples and makes the model more robust. The enhanced better performance than the original BERT-based In-Trust loss function enables the model to learn well method in some legal tasks and can be regarded even when the data is unbalanced distributed. as a good backbone for acronym disambiguation. • SciBERT model The SciBERT [20] is a pretrained language model for science. This archi- 5. Conclusion tecture of the SciBERT follows the same architecture as BERT [25] to capture the well-formed In this paper, we analyze the dificulties of acronym disrepresentation of the scientific data. This model ambiguation in legal English, including hard instances, has achieved better performance than the origi- unbalanced data distribution, and lack of labeled samples. nal BERT- based method in some scientific tasks. We propose ANACONDA, a framework that combines Law and science have many similarities, so this adversarial training and dynamic curriculum learning model is also suitable in the legal field, which can with enhanced In-trust loss function. The experimental be viewed as a good backbone for the acronym results respectively reflect the efectiveness of each stratdisambiguation. egy. Our method achieved the best performance in the • RoBERTa model The RoBERTa [26] is mainly acronym disambiguation of legal English, which shows trained on general domain corpora with Byte Pair the efectiveness and competitiveness of our methods. Encoding [27] based on the original structure of the BERT. This model can provide a good fine- 6. Acknowledgement grained representation of the sentence which can be used in distinguishing acronyms.
and Development Program of China (2020AAA0106400) 4.7. Experimental settings and the National Natural Science Foundation of China (61922085, 61976211). The work is also supported We conducted experiments on four baseline models, in- by the Beijing Academy of Artificial Intelligence cluding the rule-based model [7], LegalBERT [19], SciB- (BAAI2019QN0301), the Key Research Program of the ERT [20] and RoBERTa [26]. All models are implemented Chinese Academy of Sciences under Grant (ZDBS-SSWbased on Huggingface’s open-source converter library JSC006), the independent research project of the National [28]. We use mixed-precision training [29] based on the Laboratory of Pattern Recognition, China and the Youth Apex library. We use the initial learning rate of 5e-5 for Innovation Promotion Association CAS, China. ifne-tuning and the AdamW optimizer with a batch size of 32 for optimization. We use the enhanced In-trust loss [12] function to optimize the model. References
in the Table 2. We can find that the rule-based method
AD. Acronym Extraction and Disambiguation Shared free text, in: Proceedings of the ACL 2010 system Tasks at SDU 2022, in: Proceedings of SDU@AAAI- demonstrations, 2010, pp. 78–83.
22, 2022. [15] A. Raganato, C. D. Bovi, R. Navigli, Neural sequence [3] A. P. B. Veyseh, F. Dernoncourt, T. H. Nguyen, learning models for word sense disambiguation, in: W. Chang, L. A. Celi, Acronym identification and Proceedings of the 2017 Conference on Empirical disambiguation shared tasks for scientific document Methods in Natural Language Processing, 2017, pp. understanding, arXiv preprint arXiv:2012.11760 1156–1167.
(2020). [16] R. Navigli, S. P. Ponzetto, Babelnet: The automatic [4] N. M. Amir Pouran Ben Veyseh, S. Yoon, R. Jain, construction, evaluation and application of a wideF. Dernoncourt, T. H. Nguyen, MACRONYM: A coverage multilingual semantic network, Artificial Large-Scale Dataset for Multilingual and Multi- intelligence 193 (2012) 217–250.
Domain Acronym Extraction, in: arXiv, 2022. [17] E. Stengel-Eskin, T.-r. Su, M. Post, B. Van Durme, A [5] A. P. B. Veyseh, F. Dernoncourt, Q. H. Tran, T. H. discriminative neural model for cross-lingual word Nguyen, What does this acronym mean? introduc- alignment, arXiv preprint arXiv:1909.00444 (2019). ing a new dataset for acronym identification and [18] S. Kumar, S. Jat, K. Saxena, P. Talukdar, Zero-shot disambiguation, arXiv preprint arXiv:2010.14678 word sense disambiguation using sense definition (2020). embeddings, in: Proceedings of the 57th Annual [6] M. Bevilacqua, T. Pasini, A. Raganato, R. Navigli, Meeting of the Association for Computational Linet al., Recent trends in word sense disambigua- guistics, 2019, pp. 5670–5681. tion: A survey, in: Proceedings of the Thirtieth [19] I. Chalkidis, M. Fergadiotis, P. Malakasiotis, N. AleInternational Joint Conference on Artificial Intelli- tras, I. Androutsopoulos, Legal-bert: The mupgence, IJCAI-21, International Joint Conference on pets straight out of law school, arXiv preprint Artificial Intelligence, Inc, 2021. arXiv:2010.02559 (2020). [7] A. S. Schwartz, M. A. Hearst, A simple algorithm for [20] I. Beltagy, K. Lo, A. Cohan, Scibert: A pretrained identifying abbreviation definitions in biomedical language model for scientific text, arXiv preprint text, in: Biocomputing 2003, World Scientific, 2002, arXiv:1903.10676 (2019).
pp. 451–462. [21] Y. Bengio, J. Louradour, R. Collobert, J. Weston, [8] L. Luo, Z. Yang, P. Yang, Y. Zhang, L. Wang, H. Lin, Curriculum learning, in: Proceedings of the 26th J. Wang, An attention-based bilstm-crf approach to annual international conference on machine learndocument-level chemical named entity recognition, ing, 2009, pp. 41–48.
Bioinformatics 34 (2018) 1381–1388. [22] D. Lowd, C. Meek, Adversarial learning, in: Pro[9] A. Jaber, P. Martínez, Participation of uc3m in sdu@ ceedings of the eleventh ACM SIGKDD internaaaai-21: A hybrid approach to disambiguate scien- tional conference on Knowledge discovery in data tific acronyms., in: SDU@ AAAI, 2021. mining, 2005, pp. 641–647. [10] Q. Zhong, G. Zeng, D. Zhu, Y. Zhang, W. Lin, [23] T. Miyato, A. M. Dai, I. Goodfellow, Adversarial B. Chen, J. Tang, Leveraging domain agnostic and training methods for semi-supervised text classifispecific knowledge for acronym disambiguation., cation, arXiv preprint arXiv:1605.07725 (2016). in: SDU@ AAAI, 2021. [24] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, [11] B. Scarlini, T. Pasini, R. Navigli, Sensembert: A. Vladu, Towards deep learning models resistant to Context-enhanced sense embeddings for multilin- adversarial attacks, arXiv preprint arXiv:1706.06083 gual word sense disambiguation, in: Proceedings (2017). of the AAAI Conference on Artificial Intelligence, [25] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, volume 34, 2020, pp. 8758–8765. Bert: Pre-training of deep bidirectional transform[12] X. Huang, Y. Chen, S. Wu, J. Zhao, Y. Xie, W. Sun, ers for language understanding, arXiv preprint Named entity recognition via noise aware training arXiv:1810.04805 (2018). mechanism with data filter, in: Findings of the [26] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, Association for Computational Linguistics: ACL- O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, IJCNLP 2021, 2021, pp. 4791–4803. Roberta: A robustly optimized bert pretraining ap[13] M. Lesk, Automatic sense disambiguation using proach, arXiv preprint arXiv:1907.11692 (2019). machine readable dictionaries: how to tell a pine [27] R. Sennrich, B. Haddow, A. Birch, Neural machine cone from an ice cream cone, in: Proceedings of translation of rare words with subword units, arXiv the 5th annual international conference on Systems preprint arXiv:1508.07909 (2015).
documentation, 1986, pp. 24–26. [28] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. De[14] Z. Zhong, H. T. Ng, It makes sense: A wide- langue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funcoverage word sense disambiguation system for towicz, et al., Huggingface’s transformers: State-ofthe-art natural language processing, arXiv preprint arXiv:1910.03771 (2019). [29] P. Micikevicius, S. Narang, J. Alben, G. Diamos,
E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, et al., Mixed precision training, arXiv preprint arXiv:1710.03740 (2017).