This paper presents our findings from participating in the multilingual acronym extraction shared task SDU@AAAI-22. The task consists of acronym extraction from documents in 6 languages within scientific and legal domains. To address multilingual acronym extraction we employed BiLSTM-CRF with multilingual XLM-RoBERTa embeddings. We pretrained the XLM-RoBERTa model on the shared task corpus to further adapt XLM-RoBERTa embeddings to the shared task domain(s). Our system (team: SMR-NLP) achieved competitive performance for acronym extraction across all the languages.
The number of scientific papers published every year is growing at an increasing rate [1]. The authors of the scientific publications employ abbreviations as a tool to make technical terms less verbose. The abbreviations take the form of acronyms or initialisms. We refer to the abbreviated term as “acronym” and we refer to the full term as the “long form”. On one hand, the acronyms enable avoiding frequently used long phrases making writing convenient for researchers but on the other hand they pose a challenge to non-expert human readers. This challenge is heightened by the fact that the acronyms are not always standard written, e.g. XGBoost is an acronym of eXtreme Gradient Boosting [2]. Following the increase of scientific publications, the number of acronyms is enormously increasing as well [3]. Thus, automatic identification of acronyms and their corresponding long forms is crucial for scientific document understanding tasks.
The existing work in acronym extraction consists of carefully crafted rule-based methods [ 4, 5] and featurebased methods [6, 7]. These methods typically achieve high precision as they are designed to find long form, however, they sufer from low recall [ 8]. Recently, Deep Learning based sequence models like LSTM-CRF [9] have ever, these methods require large training data to achieve optimal performance. One of the major limitations of existing work in acronym extraction is that most prior work only focuses on the English language.
Woodstock’21: Symposium on the irreproducible science, June 07–11,
We participate in the Acronym Extraction task [10] organized by the Scientific Document Understanding workfying acronyms (short-forms) and their meanings (longforms) from the documents in six languages including Danish (da), English (en), French (fr), Spanish (es), Persian (fa) and Vietnamese (vi). The task corpus [11] consists of documents from the scientific (en, fa, vi) and legal domain (da, en, fr, es).
Following are our multi-fold contributions: 1. We model multilingual acronym extraction as a sequence labelling task and employed contextualized multilingual XLM-RoBERTa embeddings [12]. Our system consists of a single model for multilingual acronym extraction and hence is practical for real-world usage.
2. We investigated domain adaptive pretraining of XLM-RoBERTa on the task corpus, which resulted in improved performance across all the languages. 3.
for acronym extraction.
Our sequence labelling model follows the well-known architecture [13] with a bidirectional long short-term memory (BiLSTM) network and conditional random field (CRF) output layer [14]. In order to address the multilingual aspect of the task we employed contextualized multilingual XLM-RoBERTa embeddings [12] in all the experiments. .727/.750/.738 .747/.757/.752 .750/.759/.755 .617/.703/.650 .738/.742/.740 .756/.750/.753 .788/.751/.754 .715/.733/.724 .619/.539/.576 .644/.560/.599 .665/.557/.606 .864/.294/.439 .820/.871/.845 .832/.872/.852 .832/.873/.852 .823/.850/.836 .375/.547/.445 .385/.615/.474 .408/.689/.512 .623/.074/.132 .825/.833/.829 .727/.750/.738 .738/.742/.740 .619/.539/.576 .820/.871/.845 .375/.547/.445 the languages in the corpus. As a pre-processing step, we used spaCy [15] to perform word tokenization and POS tagging.
We do not apply any strategy to explicitly account for low training data of Persian and Vietnamese. Table 3 lists the best configuration of hyperparameters. We compute macro-averaged F1-score using the script provided by the organizers on the development set 1. We employ early stopping and report the F1-score on the test set using the best performant model on the development set.
Hyperparameter Value Table 1 reports the F1-score on the development and test set for all the languages. As a baseline experiment, hidden size 256 we combined the training data for all the languages learning rate 5.0 − 6 and trained a BiLSTM-CRF model using the pretrained training epochs 20 multilingual XLM-Roberta2 embeddings (row r1). This pretraining epochs 3 achieves the overall F1-score of 0.854.
We pretrained XLM-Roberta model for 1 epoch on Table 3 the task corpus using train and development set, which Hyperparameter settings for acronym extraction. results in 0.1 points improvement in the overall F1-score leading to the F1-score of 0.866 (row r2). Increasing the pretraining epochs to 3 results in an improvement of 3.2. Domain Adaptive Pretraining additional 0.1 points in the overall F1-score (row r3). The original XLM-RoBERTa embeddings [12] are trained We also experimented with training the individual on the filtered CommonCrawl data (General domain), models for each language (including separate models for whereas the data of the shared task comprises docu- English scientific and English legal). This results in a ments from scientific and legal domains. In order to significant decrease in F1-score for all the languages (on better adapt the contextualized representation to the tar- average 0.12 points in F1-score, see row r4). This demonget scientific and legal domain, we further pretrained the strates that BiLSTM-CRF with multilingual XLM-Roberta original XLM-RoBERTa model on the corpus data. Our embeddings performs best when trained with several lanexperiments demonstrate improved performance on the guages together enabling efective cross-lingual transfer. task of acronym extraction due to the domain adaptive The F1-score of our submission on the test set are pretraining across all the languages. reported in row r5. Our test submission achieves the F1-score similar to the development set for all the languages demonstrating efective generalization on the test 4. Experiments and Results set; Vietnamese is an exception where F1-score on the test set is significantly worse than the F1-score on the 4.1. Dataset development set (see rows r5 vs r3).
This research was supported by the Federal Ministry for Economic Afairs and Energy ( Bundesministerium für Wirtschaft und Energie: https://bmwi.de), grant 01MD19003E (PLASS: https://plass.io) at Siemens AG (Technology), Munich Germany.
In this paper, we described our system with which we participate in the multilingual acronym extraction shared task organized by the Scientific Document Understanding workshop 2022 (SDU@AAAI-22). We formulate multlilignual acronym extraction in 6 languages and 2 domains as a sequence labelling task and employed BiLSTM-CRF model with multilingual XLM-RoBERTa embeddings. We pretrained XLM-RoBERTa model on the target scientific and legal domain to better adapt multilingual XLM-RoBERTa embeddings for the target task. Our system demonstrates competitive performance on the multilingual acronym extraction task for all the languages. In future, we would like to improve error analysis to further enhance our multilingual acronym extraction models. tional random fields: Probabilistic models for segmenting and labeling sequence data, in: C. E. Brodley, A. P. Danyluk (Eds.), Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001), Williams College, Williamstown, MA, USA, June 28 - July 1, 2001, Morgan Kaufmann, 2001, pp. 282–289. [15] M. Honnibal, I. Montani, M. Honnibal, H. Peters, S. V. Landeghem, M. Samsonov, J. Geovedi, J. Regan, G. Orosz, S. L. Kristiansen, P. O. McCann, D. Altinok, Roman, G. Howard, S. Bozek, E. Bot, M. Amery, W. Phatthiyaphaibun, L. U. Vogelsang, B. Böing, P. K. Tippa, jeannefukumaru, GregDubbin, V. Mazaev, R. Balakrishnan, J. D. Møllerhøj, wbwseeker, M. Burton, thomasO, A. Patel, explosion/spaCy: v2.1.7: Improved evaluation, better language factories and bug fixes, 2019. URL: https://doi. org/10.5281/zenodo.3358113. doi:10.5281/zenodo. 3358113.