Acronym extraction is committed to extracting acronyms (e.g., short-forms) and their meaning (e.g., long-forms) from the original document, this is one of the key and challenging tasks in scientific document understanding (SDU@AAAI-22) tasks. Previous work regarded them as a task of named entity recognition, ignoring the relationship between acronyms and their meaning, especially the importance of initials. In this paper, we propose a novel Initial Reminder Framework (IRF) for acronym extraction task. Specifically, the IRF recognize the span of acronym for the first time, combined with the initial information, and recognized their meaning again. At the same time, considering that acronyms are often close to their meaning, the IRF adopts Neighborhood Search Strategy. Experiments on two acronym extraction dataset show IRF outperforms the previous methods by 5.90/7.10 F1. Further analysis reveals IRF is efective in extracting short-forms and long-forms.
the acronym can correspond to the initial of the long- two acronym extraction data sets, including Spanish and
Acronym extraction is a task to identify acronyms and
their meanings, which is very important for scientific
previous method regards this task more as a sequence
annotation task[
The context of acronyms often have more obvious characteristics, for example, there are brackets around acronyms, or acronyms themselves have a specific format, which leads to a higher accuracy of identifying acronyms. However, the accuracy of identifying long-term is relatively low, and there are some problems, such as inaccurate identification and no identification.
As shown in Figure 1, in a document, we need to identify the acronyms and long-term. The context of acronyms often has some characteristics (e.g. brackets), nition is a challenge. It needs to have a certain understanding of the document content. The better solution is to know what the corresponding acronym is before extracting long term, which will help model recognition of long term.
Through Figure 1, we can find that each character of orange text represents long term, and red text represents initials. At the same time, red, blue and black lines indicate the correspondence between initials and acronyms, respectively. (Dataset: Spanish) In this paper, we propose a novel Initial Reminder Framework (IRF) for acronym extraction task. Through experiments, we find that the model has high accuracy in acronym recognition than long-term recognition. Specifof identifying acronyms, the F1 score is only 83%. At the same time, considering the correlation between acronyms and long-term, IRF first completes the task of identifying acronyms. On this basis, combined with the initial information contained in acronyms, IRF further identifies long-term. We verify the efectiveness of our method on We summarize our contributions as follows: • We introduce a fresh perspective to revisit the acronym extraction task with a principled problem formulation, which implies a general algorithmic framework that helps the identify long-term • we propose a novel Initial Reminder Framework (IRF) for acronym extraction task. Specifically, IRF makes use of the high accuracy of acronym recognition and helps the model recognize longterm by integrating the initial information. • We conduct experiments on two acronym extraction datasets. Experimental results demonstrate that our IRF model can achieves state-of-the-art performance compared with baselines.
training (5928), development (741), and testing (741) sets from the whole dataset. As shown in Table 2, the Danish dataset is divided into training (3082), development (385), and testing (160) sets according to the whole dataset.
Both datasets have been manually labeled, where the label is a list of position boundaries.
We regard the acronym extraction task as a sequence annotation task. Diferent from the previous methods, considering the high accuracy of acronym recognition, we will first recognize the acronym, and then use the 3. Methodology character information of the acronym to recognize the long-term. Given a document D = { 1, 2, … , }, the In this section, we will introduce our proposed IRF model. initials of each word in the document is I = { 1, 2, … , }. IRF utilizes the corresponding relationship between the Utilizing our IRF model, we will get each acronyms and characters of acronyms and the initials of long-term, this long-term : will efectively help the model improve the accuracy of long-term recognition. , =
IRF ( 1, 2, … , ; 1, 2, … , )
(1) where A refers to acronyms and L refers to long-term.
Given a document D = { 1, 2, … , }, and the initials of each word in the document is I = { 1, 2, … , }. We leverage the pre-trained language model as an encoder to obtain the embedding as follows: The online results will be evaluated with the macroaveraged precision, recall, and F1 scores. The final score is the prediction correctness of short-form (i.e., acronym) and long-form (i.e., phrase) boundaries in the given sen- = BERT Encode ( 1, 2, … , ; 1, 2, … , ) (2) tence. The short-form or long-form predictions are correct once the beginning and the end of the position of the where = [ℎ 1, ℎ2, … , ℎ ] is the embedding of each predicted short-form or long-form are equal to the label token, is the embedding of each initial. respectively. The oficial score is counted based on the macro average of short-form and long-form F1 scores.
This task contains various multi-lingual datasets composed of document sentences in science fields. Among them, the statistics of the Spanish and the Danish datasets are shown in Table 1. The Spanish dataset is divided into The low level tagging module is designed to recognize all possible acronyms in the input sentence by directly decoding the encoded vector produced by the N-layer BERT encoder. More precisely, it adopts two identical binary classifiers to detect the start and end position of acronyms respectively by assigning each token a binary tag (0/1) that indicates whether the current token corresponds to a start or end position of a acronym. The detailed operations of the acronyms tagger on each token are as follows: = (
) = ( ℎ + ) (3) (4) where and
represent the probability of identifying the i-th token in the input sequence as the start and end position of a acronym, respectively. The corresponding token will be assigned with a tag 1 if the probability exceeds a certain threshold or with a tag 0 otherwise. ℎ is the encoded representation of the i-th token in the input sequence, i.e., ℎ = [] , where and
represent the trainable weight, and are bias and is the sigmoid activation function.
Considering that each acronym and its meaning are always connected together, we utilize Neighborhood Search Strategy to select the context near the search acronym, so as to extract the correct long-term.
The high level tagging module simultaneously identiifes the long-term with respect to the acronyms obtained at lower level. As show in the Figure 2, for the acronyms , We search for its corresponding long term in a limited context. Diferent from acronyms tagger directly decoding the encoded vector , the Long-term Tagger takes the acronyms features and initial features into account as well. The detailed operations of the Long-term
Neighborhood Search Strategy and end position of a long-term respectively, and ℎ represents the encoded representation vector of the k-th subject detected in low level module, the represents the embedding of initials (i.e., C, M and P). For each acronym, we iteratively apply the same decoding process and on it. Meanwhile, for the Neighborhood Search Strategy, we set the search length to , where the is a hyperparameter which is the longest distance between acronyms and long in the statistical training set. 4. Experiments 4.1. Baseline models
on the website1.
• Rule-based method The rule-based baseline
method is proposed to adopt manual rules for
this task [
Besides, the BiLSTM has advantages in the LSTM
that avoids gradient vanishing compared with
the RNN. The output hidden state of BiLSTM will
be concatenated between the forward LSTM
and backward LSTM networks as final output
[ , ]. This feature is calculated with the
crossentropy loss with the target token-level labels.
• BERT-CRF model The BERT-CRF [
We evaluated our method on two acronym extraction datasets
mainly including Spanish dataset and Danish dataset. Specifically, the Spanish dataset has 7410 samples, and the Danish dataset has 3853 samples [10].
We used cased BERT-base, or RoBERTa-large as the
encoder on Spanish and Danish dataset. All models are
implemented based on the open-source transformers
library of huggingface [11]. we initialize the model with
mbert [12]. We use mixed-precision training [13] based
on the Apex library. Our model is optimized with AdamW
[14] using learning rates ∈ [2−5, 3−5, 5−5, 1−4]
, with
a linear warmup [15] for the first 6% steps followed by
a linear decay to 0. We report the mean and standard
deviation of F1 on the development set by conducting 5
runs of training using diferent random seeds. We utilize
the In-trust loss [
In the Spanish and Danish datasets, we compare IRF with In this paper, we propose a novel Initial Reminder Framebaselines, including Rule-based, BiLSTM-CRF, BERT- work (IRF) for acronym extraction task. Specifically, CRF and Roberta-CRF. Results in Table 3 and Table 4 2https://competitions.codalab.org/competitions/34925results 85.31 (+ 4.89)
84.23 (+ 5.12) 90.13 (+ 6.69)
89.07 (+ 6.88) WENGSYX2.
show that PAEE performs better than these methods. Specifically, in Spanish dataset, our best model, IRF Val/Test set than Roberta-CRF. In addition, in Danish dataset, IRF built upon , is +7.65 / +7.10 F1 better on Val/Test set than Roberta-CRF. They obtain new state-of-the-art(SOTA) results, we held the first position on the CodaLab scoreboard under the alias , is +5.66 / +5.90 F1 better on Considering the correlation between acronyms and the initials of long-term, our IRF establishes the relationship between acronyms and long-term, which improves the accuracy of extracting long and the overall performance of the model. In order to further explore the efectiveness of our method, we analyze the accuracy of identifying long-term in the acronym extraction task. As show in improve the accuracy of extracting long-term. Specifically, on the F1 score, we have a maximum performance improvement of 5%. The significant increase of the recognition accuracy of the model in long term will help to improve the overall performance of the model.
IRF utilizes Acronyms Tagger to recognize the span of Roberta: A robustly optimized bert pretraining apacronym for the first time. Then combining with the ini- proach, arXiv preprint arXiv:1907.11692 (2019). tial information, IRF utilizes Long-term Tagger to recog- [10] S. Y. R. J. F. D. T. H. N. Amir Pouran Ben Veynize the long-term. IRF captures the relationship between seh, Nicole Meister, MACRONYM: A Largeacronyms and long-term in the dataset. Meanwhile, uti- Scale Dataset for Multilingual and Multi-Domain lizing the character information in acronyms, the IRF Acronym Extraction, in: arXiv, 2022. improves the accuracy of long-term recognition. We con- [11] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Deduct experiments on two acronym extraction datasets. langue, A. Moi, P. Cistac, T. Rault, R. Louf, M. FunExperimental results demonstrate that our IRF model towicz, et al., Huggingface’s transformers: State-ofcan achieves state-of-the-art performance compared with the-art natural language processing, arXiv preprint baselines. arXiv:1910.03771 (2019). [12] J. Libovickỳ , R. Rosa, A. Fraser, How languageneutral is multilingual bert?, arXiv preprint References arXiv:1911.03310 (2019). [13] P. Micikevicius, S. Narang, J. Alben, G. Diamos,
E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, et al., Mixed precision training, in: International Conference on Learning
Representations, 2018. [14] I. Loshchilov, F. Hutter, Decoupled weight decay regularization, in: International Conference on
Learning Representations, 2018. [15] P. Goyal, P. Dollár, R. Girshick, P. Noordhuis,
L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, K. He, Accurate, large minibatch sgd: Training imagenet in 1 hour (2018).