An acronym can be viewed as a word which is constructed by taking initial components from a phrase. This study deals with the problem of identification and extraction of an acronym's short and long-form. The proposed approach solves the Acronym Identification (AI) task mentioned in the scientific document understanding (SDU@AAAI-21) task. This paper model the Acronym identification task as the sentence level sequence-labeling problem. The proposed method is computed by an ensemble of various Language Models trained by hyper-parameter tuning. This ensembling technique is then coupled with post-processing steps to extract the best possible predictions. The trained model's performance is evaluated against standard evaluation metrics such as precision, recall, and F1-score. The final model achieves an F1 score of 95.60%, a precision of 93.97%, and a recall of 97.95% on the development dataset. On the test data, the proposed model achieves an F1 score of 92.08%, a precision of 89.70%, and the highest recall of 94.59%, compared to other participants results in the competition.
The acronyms in the technical/scientific domain are
increasing at an exponential rate. This is due to a huge amount of
research conducted in the technical and non-technical
domains. The term “acronym” is defined as the name given to
a particular word or a phrase by taking the first letters of
each word of a phrase
Several techniques have been proposed to extract
acronyms from a given input text corpus. These systems are
rule-based
This paper presents an effective ensembling based
approach to automatically extract acronyms along with their
extended/long-forms. The proposed approach combines
ensembling Language Models + hyper-parameter tuning +
post-processing to get the best possible results. This paper is
structured by giving a survey of related work where a quick
brief about existing approaches used to solve the problem of
Acronym Identification
The existing Acronym Identification systems can be broadly classified into rule-based, features & machine learningbased and Deep Learning-based as depicted in figure 2. The rule-based systems mainly consist of techniques that use rules, patterns and regular expressions to extract acronyms and their expansions. The machine learning-based techniques first extract features required for Acronym Identification, and then a classifier (like Support Vector Machine, Conditional Random Fields) is trained over these features. The Deep Learning-based techniques use state-of-the-art algorithms like Recurrent Neural Networks (RNN), Long Short Term Memory (LSTM), transformer-based models, etc.
The Acronym Identification techniques can be traced back
to the year 1999 where
A popular approach to detect acronyms was proposed
by
1https://github.com/philgooch/abbreviation-extraction
One of the machine learning-based Acronym
Identification systems proposed by
A machine learning-based approach proposed by
Another approach that utilizes Long Short Term
Memory - Conditional Random Field (LSTM-CRF) proposed
by
The rule-based and machine learning-based models were
not able to capture the contextual information. An acronym
extraction system that uses machine learning combined with
a neural network based-contextual model was proposed
by
The proposed system architecture is depicted in figure 3. The main components of Effective Ensembling of Language Models for Sequence Labeling Problem (EELM-SLP) System Architecture are as follows: 1. Data Acquisition/Collection. 2. Finetuning of transformer-based Language Models for sequence-labeling task. 3. Hyperparameter tuning and retraining of Language Models. 4. Ensembling and postprocessing to obtain final predictions. 5. Evaluation on development and test datasets.
The entire data can be downloaded from https://github.
com/amirveyseh/AAAI-21-SDU-shared-task-1-AI. A
sample snapshot of the data is depicted in figure 1. The data
is bifurcated into training, development, and test data. The
training dataset consists of 14006 labeled sentences
The proposed approach uses six transformer-based
pretrained Language Models, who are fined tuned on a
downstream sequence labeling task. The following Language
Models are used: BERT
The configuration used for finetuning Language Models are as follows: • The BERT and CamemBERT model is 12-layered having 768-hidden layers, 12-heads, and 109M parameters. • RoBERTa has the same configuration as that of BERT, except it has 125M parameters. • XLM-RoBERTa has approximately 270M parameters with 12-layers, 768-hidden-state, 3072 feed-forward hidden-state, 8-heads, which is pretrained on CommonCrawl data in 100 languages. • DistilBERT has less parameters of 65M, and it is 6layered. DistilBERT model is distilled from the BERTbased configuration. • Longformer has approximately 149M parameters. Here 4096 represents that the model is pretrained on documents of maximum length 4096.
All the above-mentioned Language Models are finetuned on the training dataset for sentence-level sequence labeling. Every model is trained separately, and the latest iteration is stored.
It was observed that during the inference phase, the output
length was truncated to 128 tokens as it was the default
‘max seq length’ parameter. To preserve the entire token
length, two parameters were used heavily. These two
parameters are ‘sliding window’ and ‘max seq length’. The
sliding window prevents the truncation of sentences by
splitting the input sequence into multiple windows if it
exceeds the default maximum sequence value. The sliding
window problem is that the contextual information is broken
while predicting, and hence it was not used. Some
experiments were carried on with max seq length parameter, and
it was observed that the model was performing better when
max seq length was kept to 350. Other max seq length
values were 128, 256, 300, 400, 450, and 512. The language
models were finetuned by using ‘Simple Transformers’
Table 1 depicts the final list of hyperparameters used to finetune the Language Models on the sequence labeling task. The ‘adam epsilon’ is the epsilon value to use for adam optimizer. ‘best model dir’ is the directory where the best model automatically gets stored after the completion of all epochs. ‘cache dir’ is where the processed data is stored, 2https://simpletransformers.ai/ 3https://huggingface.co/ Hyperparameter adam epsilon best model dir
cache dir train custom parameters only dataloader num workers
do lower case early stopping consider epochs early stopping delta early stopping metric early stopping metric minimize early stopping patience
encoding eval batch size evaluate during training evaluate during training silent evaluate during training steps evaluate during training verbose
fp16 gradient accumulation steps learning rate local rank logging steps max grad norm max seq length multiprocessing chunksize
n gpu no cache no save num train epochs
output dir overwrite output dir
process count reprocess input data
save best model save eval checkpoints save model every epoch
save steps save optimizer and scheduler
silent train batch size use cached eval features use early stopping use multiprocessing warmup ratio warmup steps weight decay classification report labels list lazy loading lazy loading start line Value 1e-08 outputs/best model/ cache/ False
4 False False
0 eval loss
True 3 null 8 False True 2000 False True
1 4e-05 -1 50 1.0 350 500 1 False False 50 outputs/
True
4 True True True True 20000 True False 16 False False True 0.06 2628
0
False B-long, I-long, B-short, I-short, O
False
0
which is consumable by PyTorch
After all the language models were finetuned on training data, the ensembling technique was applied. The advantage of performing ensembling is to construct a robust discriminator or model by using comparatively weaker models. In this paper, a novel-ensembling approach based on RoBERTa prioritizing was used. The table 2 shows that the F1-score of RoBERTa for the development dataset is higher than other algorithms. The steps are as follows: 1. Initially, all the predictions from six pre-trained language models were extracted. 2. For each sentence, token level predictions were extracted for all six models. Hence for each token for every sentence, six predictions were obtained. For simplicity, the predictions can be from any of the following labels - ‘O’, ‘B-short’, ‘I-short’, ‘B-long’ and ‘I-long’. 3. If all the predictions are ‘O’ and any of the models gives any predictions from the bouquet of ‘B-short’, ‘I-short’, ‘B-long’ and ‘I-long’, then the prediction other than ‘O’ is considered. 4. If there is a conflict between ‘B-short’, ‘I-short’, ‘Blong’, and ‘I-long’, then the prediction suggested by RoBERTa is taken into consideration. But if, in this case, if RoBERTa’s prediction is ‘O’, then the prediction with the highest frequency is considered.
The above ensembling technique gave rise to an increase in recall. The ensembling approach’s limitation is that it predicted labels having ‘I-short’ and ‘I-long’ as the beginning tags, which lowered precision for some tokens. To overcome this problem, the following post-processing rules were applied:
By applying the above post-processing rules, the precision improved the proposed, and hence the proposed EELM-SLP system surpassed the other individual trained models on the development dataset.
All the models are trained on 11GB Nividia 2080 Ti GPU for 20-22 epochs, with each epoch taking around three to four minutes. Each model took around 90 minutes to train, and due to early stopping, the training was stopped as soon as the loss stopped reducing. The proposed architecture is efficient and can be integrated into an actual production environment. The prediction pipeline is designed in such a way that it takes advantage of CPU parallelism. So if the input contains a batch of sentences, the predictions will be computed in parallel independent batches and then later combined to form in order as they appear in the original paragraph or list of sentences.
The table 2 depicts the evaluation metrics on the development dataset. It can be seen that the top-3 performers among individual language models based on F1-score are RoBERTa, followed by XLM-RoBERTa, which is followed by BERT. The RoBERTa achieves the highest F1score of 92.18%, while XLM-RoBERTa achieves the F1score of 91.95%. The third best performing algorithm BERT achieves the F1-score of 91.6%. The ensembling technique surpasses the hyperparameter finetuned individual Language Models and achieves a precision of 92.36%, recall of 97.2%, and F1-score of 95.6%. However, the proposed approach of Effective Ensembling of Language Models for Sequence Labeling Problem (EELM-SLP) achieves the highest F1-score of 95.6% recorded on development data. It can be clearly seen that the proposed approach surpasses other individual language models by quite a considerable margin. For all the individual finetuned models and the proposed approach, the class scores are also computed, namely precision, recall, and F1-score for ‘short’ and ‘long’ labels.
In table 3, the evaluation on the test dataset is presented on standard evaluation metrics like precision, recall, and F1score. Initially, the test dataset was evaluated on RoBERTa finetuned model before finetuning. This initial evaluation resulted in precision, recall, and F1-score to be 90.85%, 91.73%, and 91.29%, respectively. The RoBERTa was then finetuned after hyperparameter tuning, which further accelerated the metrics. The hyperparameter tuned RoBERTa achieved the precision of 90.26%, recall of 92.46%, and F1score of 91.34%. One of the significant highlights is the results obtained after applying the ensembling technique, which improved the F1-score from 91.34% to 91.86%. Finally, the highest F1-score was achieved after the application of post-processing. The final proposed system achieved an F1-score of 92.08%, precision of 89.7%, and recall of 94.59%.
In this paper, an Effective Ensembling of Language Models for Sequence Labeling (EELM-SLP) is proposed. The ensembling technique’s importance can be clearly seen based on the development and test dataset results. The final model achieves a precision of 93.35%, recall of 97.95% and F1score of 95.6% on the development dataset and precision of 89.7%, recall of 94.59% and F1-score of 92.08% on the test set. There is an improvement on all the three metrics of precision, recall, and F1-score for the proposed approach on the development dataset. Although the recall and F1-score are highest for the test dataset, the precision is lowered. This might be due to the introduction of false-positives while ensembling and post-processing step. On the one hand, the ensembling and post-processing step increased the recall and F1-score, but on the other hand, it lowered the precision. Hence, the post-processing can be finetuned in-depth. Further, this paper uses all the language models in their ‘base’ configuration and not ‘large’ configuration. In the future, experiments can be carried on ‘large’ configuration based language models, which might improve the scores.