This paper describes the system proposed for addressing the research problem posed in Task1 of scientific document understanding (SDU@AAAI-2021): Acronym Identification. We proposed an end-to-end model that takes the text as input and corresponding to each word gives the label of word to be acronyms (short-forms) or their meanings (long-forms). We take experiment on several totally different ideas, including features engineering, transformer model, multi-task learning, Span and CRF. Our result shows that feature-based method can handle this task well, and transformer-based models are particularly effective in this task. Moreover, different model frameworks complement each other. We achieved the best f1 score of 0.931 on test dataset and were ranked second.
An obstacle of scientific document understanding (SDU) is
the extensive use of acronyms which are shortened forms
of long technical phrases
We tried four different approaches for the task. Our
feature-based approach use BiLSTM-CRF
Our XLNet-Softmax approach is inspired by the
Emphasis Selection paper
We take this task as sentence level sequence labeling problem. Given a sequence of words or tokens in a text, the task is to compute a label Li for each xi which indicates the boundaries of short-form (i.e., acronym) and long-form (i.e., phrase).
The approaches are evaluated based on their macro-averaged precision, recall, and F1 scores on the test set computed for correct predictions of short-form (i.e., acronym) and longform (i.e., phrase) boundaries in the sentences. A short-form or long-form boundary prediction is counted as correct if the beginning and the end of the predicted short-form or long-from boundaries equal to the ground-truth beginning and end of the short-form or long-form boundary, respectively. The official score is the macro average of short-form and long-form prediction F1 score. The dataset consisting of 20000+ sentences extracted from English scientific papers published at arXiv. The dataset is divided into training (17506), development (1715) and test (1750) sets by organizers. The training and development datasets are manually labeled. Each sample has three attributes: tokens, labels and id. The short-form and long-form labels of words are in BIO format. The labels B-short and Blong identifies the beginning of a short-form and long-form phrase, respectively. The labels I-short and I-long indicates the words inside the short-form or long-form phrases. Finally, the label O shows the word is not part of any shortform or long-form phrase.
The baseline method proposed by Schwartz is a rule-based
method
BiLSTM-CRF (Liu et al. 2019) and BERT-large model
In this section, we introduce the four types of approaches in detail, and the overall architecture of our approaches is shown in Figure 1.
Since the data is relatively regular, i.e., a word with more than 60% of the characters in uppercase are most likely to be a acronym, and its long-form may probably appear before or after it up to a certain window size2, we consider to extract some task-specific features to improve our models. After exploring the whole dataset, we extract the following features and concatenate them as additional inputs.
Token Length Token length is a commonly used feature in NLP. In our experiments, we statistic the length of every tokens and decide to group them by their lengths, except for the tokens whose lengths are greater than 15, in which we set them a separate group. In this way, we have a 16-dimensional feature.
POS Tagging Part-of-speech Tagging is another commonly used feature in NLP. We first count the number of categories of POS tags on the entire dataset, then use Ndimensional one-hot features to represent. In our experiments, N is 43.
2https://github.com/amirveyseh/AAAI-21-SDU-shared-task-1Letter Case According to our data analysis, a word in uppercase has an 85.4% probability of being acronym. So we design a 4-dimensional one-hot feature to represent that if the words are uppercase, lowercase, capitalization or others.
Token Type Generally speaking, a token composed entirely of punctuation cannot be an acronym or a longform, but a token that is alphanumeric strings or a mixture of English letters and punctuation is most likely to be an acronym or long-form. So we design another 4dimensional one-hot feature to represent that if the tokens are just all punctuation, all English strings, mixture strings, or strings with non-English.
prior knowledge is that a token matching this pattern is probably a acronym, such as 2D, 3D and 5G. A 2dimensional one-hot feature about this is extracted. Character Match Features We notice that the official baseline, i.e. Character Match, can achieve a very high precision scores, 93.22, which may be used to design some strong features. In our experiments, we use the predictions of Character Match and group the tokens into their predicted labels (’O’, ’B-long’, ’I-long’, ’B-short’, ’I-short’) to present a 5-dimensional one-hot feature. Previous Tokens During our data exploring, we find that acronyms or long-forms are often following some specific tokens, such as ’the’, ’a’, ’called’, ’mean’, ’of ’, ’or’, ’and’, ’(’, ’)’, ’-’, ’,’, ’:’, ’”’, . So we design a 14dimensional one-hot feature to represent it.
Next Tokens The acronyms and long-forms are also often followed by the same specific tokens as Previous Tokens. So another 14-dimensional one-hot feature is extracted. Pseudo Map Short to Long We design two types of features according to the facts that acronyms are the short forms of longer phrases. Firstly we design a pattern that can match most of acronyms. This pattern must consist of letters and digits and contains at least two letters in uppercase, except for composing of all digits. Tokens which match the pattern are collected as a set, named shorted set, while the letters in these tokens are collected as another set, named shorted letter set. The first type of features is to determine if each token matches the pattern, or the first letter is in the shorted letter set, or other cases. So a 3-dimensional one-hot feature can be extracted. The second type of features is to traverse the sentence to see if tokens matches the pattern or the first letter is matching part of acronym with label ’B’ or ’I’. For example, token ’mean’ can be the beginning of the acronym MSE, which means ’mean square error’, and the inside of the acronym RMSE, which means ’root mean square error’. There can be five cases in this way, that a token may in the shorted set meaning that it is acronym itself, or be the beginning or the inside (appearing in only one acronym), or both the beginning and the inside (appearing in multiple acronyms), or neither the beginning nor the middle (not appearing in any acronym). Then a 5-dimensional feature is extracted.
We concatenate the features mentioned above all. Additionally, since the GloVe vectors3 are usually used as supplementary input for the bert-based models, we also concatenate the GloVe vector with our handful features to generate a 410-dimensional feature.
In this approach, words are tokenized into sub-words
using corresponding tokenizer of transformer model. For
the Transformer-base models, we used pre-trained XLNET
sists of short-start, short-end. Simultaneously, the cross entropy loss for Long-Phrase-Segmentation is calculated. Finally, we sum the three parts of loss. Addition to the multitask work frame, we tried adversarial Learning, which is inspired by MRC framework (Li et al. 2019).
We design a BERT-CRF model
In addition to treat the task as a sequence labeling problem, we also model it as determining the boundary of phrase spans, including short-forms and long-forms.
We employ two binary classifiers to output multiple start indexes and multiple end indexes, one to predict whether each token is the start index or not, the other to predict whether each token is the end index or not. Given the representation matrix output from BERT encoder, the model predicts the probabilities of each token being the start or end position of need forms. At training time, each sentence is Model Character Match BiLSTM-CRF BERT-large 1. Fixed-RoBERTa+Feature+BiLSTM-CRF (K-fold) 2. XLNet-SoftMax 3. XLNet-SoftMax+Multi-Task+Adversarial-Learning 4. BERT-CRF 5. BERT-Span 6. Model vote (Model-2,4,5) 7. BERT-Span+BERT-CRF paired with two label sequences Ystart and Yend of length n representing the ground-truth label of each token xi being the start index or end index of any form. The loss of two weighted cross-entry losses are jointly trained in an end-toend fashion with parameters shared at the BERT layer. At test time, we select span based the same tag from the start indexes predictions I^start and end indexes predictions I^end.
In order to take advantage of the above two models, we also employ multi-task model based on BERT-CRF model and BERT-Span model with the same BERT layers. And we train the multi-task model using BERT-CRF loss and BERT-Span loss weighted sum, and in prediction process, we decode the tags with the BERT-Span model. And the multi-task learning model touchs the highest f-score with a single model on the development dataset, but we have no enough time to verify the effect on the test dataset.
Our implementation uses PyTorch 4 library for deep learning models and the Transformers 5 library by Hugging face for the pre-trained transformer models and their tokenizers.
In XLNet Softmax model, the pre-trained XLNet-bigcased model was used without freezing the layers, and the outputs of all the layers were concatenated. Additionally, letters embedding, first letter embedding and tag indicating whether capitalize the first letter were concatenated. We use hidden dimension of 128 and pre-trained large model of XLNET (large-xlnet-cased). The tokens size is padding to 345 and letter size is padding to 98. For the classifier, two fully connected layers are used with ReLU activation function. Dropout layers with a probability of 0.3 are added to avoid overfitting. Cross-Entropy loss is used for training phase. F1 score is used as performance metrics for validation. Adam optimizer with the learning rate of 2e-5 is used. The model is fine-tuned for 4 epochs. The reported test result is corresponding to the best score on validation set.
In BERT-CRF approach, We initialize the model with
5https://github.com/huggingface/transformers bert-large-wwm 6, and initial learning rates are 5e-5 and 5e2 for BERT and CRF respectively, and the batch size is 16.
In BERT-Span approach, We initialize the model with bert-large-wwm, and initial learning rates are 5e-5, and the batch size is 16.
Our models are trained on 8 Nvidia Tesla P40 cards, and predicted on one GPU card. Take BERT-Span model as an example, average train time of one epoch is 27 minutes and average predict time is 45 seconds for development dataset.
The main result of our models and some baseline is shown in Table 1. Through the Table 1 shown, the pretrained model has more advantages and reaches F1: 90.65% on test dataset, since the train dataset is relatively small. Voting on the results of XLNet-Fostmax, BERT-CRF, BERT-Span reaches F1:93.11% on test dataset. In fact, multi-task of BERT-CRF and BEET-Span gets the highest F1:94.11% on development dataset, but we don’t have enough time to verify on test dataset.
From the results of BERT-CRF and BERT-Span, it is relatively earier to predict the begin and end of short-forms or long-forms. Moreover, based on result of multi-task model of BERT-CRF and BERT-Span, multi-task learning can learn the advantages of the two models.
We attempted numerous small changes to our models. This included some specific attempts particular to each approach and some ablation experiments. For the XLNetSoftmax, we compared cased sub-word and no cased subword. We found that cased sub-word is particular important in this task. We tried ablating the multi-task metric and adversarial Learning work frame respectively. We found that multi-task metric and adversarial Learning work frame slightly improve result. We tried ablating the first letter embedding, capital letter tag and GRU encode for each token’s letters. We found that first letter embedding is slightly effective. We also tried replace last fully connect layers with LSTM or transformer. We found that fully connect layer is ok. As the submission chance is limited, we did not try all these on test data.
masking/tree/main We described the systems used for submission in the Task1 of SDU@AAAI-2021. The task was a sentence level sequence label task. We tried several approaches which used Transformer-based models like BERT, RoBERTa, XLNet. These models are pre-trained and hence, perform well on the small dataset after fine-tuning. There was much future work left. Firstly, transformer-based approach and featurebased approach should be well fused. As we have only about one week to study the task, fusion was unfinished. We test transformer-based approach and feature-based approach respectively on test data. The two approaches reach f1 score of 0.931 and 0.9281, ranked second and third according to the leaderboard. Secondly, inside transformer-based approach, we ensemble different models by simply vote. This was primal method and should be improved. Thirdly, after the competition, we further studied the task. Inspired by the BERTCRF model and BERT-Span model, we proposed a Multitask model with BERT-Span and BERT-CRF, which employ multi-task model based on BERT-CRF model and BERTSpan model with the same BERT layers, and reach a high f1 score of 0.943 on validation set.