Extracting structured data from bibliographic references is a crucial task for the creation of scholarly databases. While approaches, tools, and evaluation data sets for the task exist, there is a distinct lack of support for languages other than English and scripts other than the Latin alphabet. A significant portion of the scientific literature that is thereby excluded consists of publications written in Cyrillic script languages. To address this problem, we introduce a new multilingual and multidisciplinary data set of over 100,000 labeled reference strings. The data set covers multiple Cyrillic languages and contains over 700 manually labeled references, while the remaining are generated synthetically. With random samples of varying size of this data, we train multiple well performing sequence labeling BERT models and thus show the usability of our proposed data set. To this end, we showcase an implementation of a multilingual BERT model trained on the synthetic data and evaluated on the manually labeled references. Our model achieves an F1 score of 0.93 and thereby significantly outperforms a state-of-the-art model we retrain and evaluate on our data.
1. Introduction
rectly communicate with other researchers through
publications [1]. Therefore, accurate citation data is important
for applications such as academic search engines [
and evaluate them against a GROBID model retrained on the same data. Throughout the paper we refer to the reference string parsing module of GROBID as just “GROBID”. To the best of our knowledge, we are the first to train a CFE model, more specifically
in Cyrillic script references. coverage and size, containing only several thousand in- (7%) of English references as well. The data set can be
1. We introduce a large data set of labeled Cyrillic reference strings,1 consisting of over 100,000 synthetically generated references and over 700 references that were manually labeled and gathered from multidisciplinary Cyrillic script publications. 2. We train the very first BERT-based citation field extraction (CFE) model specialized in Cyrillic script references and show the importance of retraining GROBID for Cyrillic script language data.
We achieve an acceptably high F1 score of 0.933 with our best BERT model.
formance are supervised machine learning approaches. Among them, the reference-parsing model of GROBID is typically reported to perform the best. We therefore use GROBID as the baseline in our evaluation.
In recent years, transformer-based models [13] such as BERT [14] have achieved state-of-the-art evaluation results on a wide range of NLP tasks. To the best of our knowledge, there is so far only one paper presenting a BERT-based approach to CFE [15]. The authors achieve state-of-the-art results on the UMass CFE data set [16] by using RoBERTa, a BERT model with a modified training procedure and hyperparameters.
The original BERT model comes in three varieties, one trained on English text only, one on Chinese, and a multilingual model. Furthermore, many ofshoots of BERT
and “citation string” interchangeably. for diferent languages can be found in the literature. For Cyrillic languages, for example, RuBERT is a BERT variant trained on Russian text [17], and Slavic BERT is a named entity recognition model that was trained on four Slavic languages (Russian, Bulgarian, Czech, and Polish) [18]. Both of the aforementioned publications present a performance gain compared to the pretrained multilingual BERT by retraining on task-relevant languages. Because references in Cyrillic publications typically also contain a mix of Cyrillic and English references, we use multilingual BERT in our evaluation.
ating CFE models exist. In Table 1, we show an overview of these citation data sets, including the number of reference strings contained and disciplines covered. In the following, we describe each of the data sets in more detail.
The authors of GROBID [6] provide the 6,835 samples their tool’s reference parser is trained on. These are gathered from various sources (e.g., CORA, HAL archive, and arXiv). New data is continuously added to the GROBID data set2.
Web of Science
One of the most widely used data sets for the CFE task is CORA,3 which comprises 1,877 “coarse-grained” labeled instances from the computer science domain. As pointed out by Prasad et al. [8], a shortcoming of the CFE research field is that the models are evaluated mainly on the CORA data set, which lacks diversity in terms of multidisciplinarity and multilinguality.
The UMass CFE data set by Anzaroot and McCallum [16] provides both fine- and coarse-grained labels from across the STEM fields. Fine- and coarse-grained labels means, for example, that labels are given for a person’s full name (coarse-grained), but also for their given and family name separately (fine-grained).
All of the above manually annotated data sets are rather small and part of them is limited in terms of the scientific disciplines covered. These issues are addressed by Grennan et al. [9] with the data set GIANT, created by synthetically generating reference strings. The data set consists of roughly 1 billion references from multiple disciplines, which were created using 677,000 bibliographic entries from Crossref4 rendered in over 1,500 citation styles.
We see none of the data sets described above as suitable for training a model for extracting citation data from Cyrillic publications’ references, because they are based on English language citation strings only, except for GIANT. However, GIANT does not provide consistent language labels, making the issue of accurate filtering for Cyrillic script citation strings non-trivial.
To the best of our knowledge, no data set of citation strings in Cyrillic script currently exists. It is therefore necessary to create a data set of labeled citation strings to be able to train models capable of reliably extracting information from Cyrillic script reference strings.
for creating an appropriate data set to train and test deep neural networks that extract citation fields, such as author information and paper titles. Grennan et al. [9], Grennan and Beel [19], and Thai et al. [15] found that synthetically generated citation strings are suitable to train machine learning algorithms for CFE, resulting in high-performance models. We use a similar approach to create a synthetic data set of citation strings for model training in the next section. To evaluate the resulting models on citation strings from real documents, we manually annotate citation strings from several Cyrillic script scientific papers. This is described in the subsection “Manually Annotated References.” 3See https://people.cs.umass.edu/~mccallum/data.html. 4See https://www.crossref.org. 5See https://api.crossref.org/works.
6See https://www.webofknowledge.com/. .bib
3 .txt <author>Иван</author> <author>Иванов </author>. <title>Заголовок</title>.
In <journal>Сибирский филологичес... Figure 2 shows a schematic overview of our data set creation, which is described in the following.
To create a data set of synthetic citation strings, a suitable source of metadata of Cyrillic script documents is necessary. Crossref, which is used by GIANT, provides metadata for over 120 million records5 of various content types (e.g., journal-article, book, and chapter) via their REST API. Unfortunately, most of the data either does not provide a language field or the language tag is English. We also considered CORE [20] as a source of metadata. Although CORE provides at least 23,000 papers with Cyrillic script language labels and corresponding PDF files [ 21], it comes with insuficient metadata.
Furthermore, for the relevant BibTeX fields, CORE only provides title, authors, year, and some publisher entries.
We identified Web of Science (WoS) 6 as the most appropriate source of metadata for creating synthetic references and based on the option to gather language-specific metadata. Additionally, WoS provides a filter for the document type, even though it lacks, for example, book types.
The final data set should contain multiple document types to cover various citation fields.
Web of Science provides access to the Russian Science Citation Index (RSCI), a bibliographic database of scientific publications in Russian with roughly 750,000 instances. We chose to gather around 27,000 most recent (i.e., from 2020) article type and around 7,000 most recent (i.e., from 2010-2020) conference proceeding type7 metadata records from the RSCI. The selection is motivated tain level of variety we use the GOST2003, GOST2006,10 by the finding of Grennan and Beel [19] that a model and GOST2008 styles for all references. Since the APA trained with more than 10,000 citations would decrease style cannot handle Cyrillic characters, it is used for nonin performance compared with a smaller training data Cyrillic references only. set. To verify the latter statement in our evaluation, we For each reference, we create a separate PDF rendition. decide to create a data set consisting of 100,000 citation Using various bibliography styles for the same reference strings in total. Last but not least, following the GIANT can result in reference strings that are completely diferdata set, we wanted our data set to consist of around 80% ent in look and structure. For instance, author names articles and 20% conference proceedings. can be abbreviated or duplicated at diferent positions. 11
Based on the language tags in the metadata provided Metadata labels and their counterparts in the PDF referby WoS, a breakdown of the languages of the data we ences are then matched by an exact string match or, altercollected is shown in Table 2. Unfortunately, the RSCI natively, the Levenshtein distance. Exact string matches database by WoS does not provide Ukrainian language are not always possible because some characters are mametadata, but since Russian and Ukrainian are very simi- nipulated by TeX while generating a PDF file or field lar, we expect the model to process Ukrainian language values themselves change during the generation process references comparably reliable to Russian language ref- in various ways, like abbreviations or misinterpreted erences. In our evaluation, we show that our model characters. To store the reference text and reference toachieves similar F1 scores for Russian and Ukrainian lan- ken labels in one file per reference, we create labeled guage references. reference strings as shown in Figure 1.
After converting the WoS data to the BibTeX format In rare cases during the parsing process of the PDFs and filtering out corrupted entries, we enrich the data to text strings using PDFMiner, tokens were garbled and with additional features, such as “Pagetotal”8 and “ad- files could not be read. Consequently, the corresponddress” (publisher city), to get extensive BibTeX entries ing items are removed from the data set, resulting in that are comparable to real references. This process re- slightly varying numbers of references for diferent citasults in a total of 34,228 metadata records in the BibTeX tion styles. In the end, our approach yields about 100,000 format. To generate bibliographic references, we addi- synthetically generated labeled reference strings. A detionally need to identify a set of suitable citation styles. tailed breakdown of the quantity of data for each citation
Based on a CORE subset of Cyrillic script scientific style is shown in Table 3. papers (see next subsection for details), we identify the In Table 4, we additionally show the breakdown of GOST and APA citation styles to be best suited for gen- labels covered by our synthetic references. erating realistic reference strings. The GOST standards9 were developed by the government of the Soviet Union and are comparable to standards by the American ANSI or German DIN. They are still widely used in Russia and in many former soviet republics. To introduce a cer3.2.2. Manually Annotated References Despite the fact that many large scholarly data sets are Parameter Counts publicly available, most lack broad language coverage Number of annotated papers 100 or do not contain full text documents. Investigating sev- Number of reference strings 771 eral data sources, we find that, for example, the PubMed Average reference length (in tokens) 28.00 Central Open Access Subset12 provides mostly English Number of reference related labels 11 language publications,13 just like S2ORC [22]. Further, Number of labeled reference segments 5,080 the Microsoft Academic Graph [ 23, 24] covers millions of publications, but does not contain full texts and therefore also no reference strings. pers. Furthermore, references containing fields outside
We use the data set introduced by Krause et al. [21] as the scope of our labels, like editor or institution, exist. In a source of Cyrillic script papers. After a filtering step to the case of booktitle fields of conference proceedings, we remove papers with lacking or unstructured citations we used the journal label. Lastly, due to the diference in use randomly chose 100 papers to manually annotate. of “№” across citation styles (indicating either an issue
Analyzing the origin of the selected papers, we note or volume number), in ambiguous cases the number after that 80 originate from the “A.N.Beketov KNUME Digital “№” is labeled volume following the GOST2006 citation Repository”14 and five from the “Zhytomyr State Uni- style. versity Library.”15 Origins could not be determined for Table 5 shows the summary statistics of the resulting 15 papers. Figure 3 shows the distribution of papers by data set. In Table 6, we show the labels used and their publication year. A breakdown of the disciplines covered number of occurrences counted in segments (a segment by the data set revealed that the most strongly repre- is the full text range for a label). sented disciplines are “engineering” with 36 papers and Although 65% of the 100 documents are Ukrainian “economics” with 16 papers. The remaining 48 papers are language papers, the references are written in various spread across various fields, such as education, zoology, languages. Nearly 99% are written in Russian, Ukrainian urban planning/infrastructure. and English (see Table 7). Other languages contained are
Using fastText [25, 26] language detection, we find Polish, German, Serbian, and French. that our sample consists of 65 Ukrainian language and While the number of manually annotated references 35 Russian language papers. is not large enough for training purposes, we argue that
Using the annotation tool INCEpTION [27], we label the size and language distribution enable us to perform the references in our 100 PDFs. Regarding manual anno- a realistic evaluation of our models. tation, we note that the real references did not always ift our set of metadata labels. For example, references to 4. Approach patents, legal texts, or web resources might not contain certain elements typical for references to scientific pa12See https://www.ncbi.nlm.nih.gov/pmc/tools/openftlist/. 13See https://www.ncbi.nlm.nih.gov/pmc/about/faq/#q16. 14See https://eprints.kname.edu.ua/. 15See http://eprints.zu.edu.ua/.
them use regular expressions, template matching, knowledge bases, or supervised machine learning, whereby machine learning-based approaches achieve the best results [28]. Furthermore, tools difer in terms of extracted 16 core Intel Xeon Gold 6226R 2.90GHz CPU) takes 1,233 minutes.
To evaluate our fine-tuned BERT model not only on the manually annotated but also on the synthetic references, we remove a hold-out set of 2,000 synthetic references from the training set, with a fixed distribution of citation styles, according to the distribution of the entire data set.
We further evaluate a BERT model trained on 2,000 random instances18—referred to as BERT from here on—regarding individual labels. Since our model is more ifne-grained than the test set, i.e. labels in the synthetic fact that most English language references are formatdata set and manually annotated data set are not the ted in the APA style, where there is no ambiguity in the same, we had to change the pagetotal label to pages and respective labels. the booktitle label to journal. Furthermore, BERT predicts publisher and address
As shown in Table 9, our model performs best on iden- labels worse for English language references than for tifying author tokens with an F1 score of 0.989. Overall, Russian and Ukrainian language references. we observe an F1 score of more than 0.934 for 6 labels (author, year, pages, address, other, and title). 5.2. BERT Evaluation on the Synthetic
We see room for improvement in publisher, journal, volume, and number predictions. The poor performance Hold-Out Set in volume and number predictions can be explained by the Our fine-tuned BERT underperforms in some labels on ambiguity of “№” in the test set (see Section “Manually the manually annotated test set. To evaluate our model Annotated References”). on data with less ambiguity and the same reference docu
We see high recall with low precision values in number ment types it was trained on, we assess the performance predictions and low recall with high precision values on the synthetic hold-out set. in volume predictions. The same observation can be Scores for recall, precision, and F1 score for all 9 trainmade for journal and publisher predictions, but to a lesser ing set sizes evaluated on the hold-out set are visualized degree. in Figure 5. All BERT models achieve F1 scores of over
More than 50% of the actual volume labels are labeled 0.99, even the model fine-tuned with 500 instances. We as number, and around 17% of actual publisher labels are also see a steady increase in the performance, when inlabeled as journal. creasing the training data set size. Best performance
Next, we look into the evaluation on the synthetic hold- regarding the F1 score (0.998) is achieved by the model out set. We evaluate the BERT model depending on trained on 100,000 instances, while this model performs the languages of references (see Figure 4). worst on the manually annotated data set. There are
As mentioned before, our synthetic data set lacks also small diferences in the scores concerning individual Ukrainian language references. Nevertheless, the F1 labels. score of 0.946 for Russian language references is only 2.5% higher than the F1 score of 0.921 for Ukrainian lan- 5.3. GROBID Evaluation guage references. This is potentially due to the high similarity between the Russian and Ukrainian languages. We compare our fine-tuned BERT with the state-of-the
Additionally, for English language references, the pre- art GROBID model. First, we evaluate the of-the-shelf dictions of volume and number labels are much better GROBID on our manually annotated test set. The model than for Cyrillic script references. This is due to the achieves unsatisfying results with an F1 score of 0.09.
Only numeric tokens such as number or year achieve an 18Models trained on 2,000 instances perform best on average. 500 1K 2K 3K 5K 10K 20K 50K 100K
Size of training set than 10,000 references. The best performing GROBID model was trained with 5,000 instances, achieving a F1 score of 0.647. We refer to this best performing GROBID model as GROBID . Compared to the of-the-shelf GROBID results, we managed to increase the F1 score by a factor of seven by retraining GROBID.
Compared to the of-the-shelf GROBID, we see higher F1 scores in almost every label, except for year and number. The best label performance is measured for paper title, with an F1 score of 0.817. A comparison of evaluation metrics of GROBID and BERT is shown in Table 10. Our BERT model outperforms the GROBID model in every label and, consequently, in overall F1 score as well.
19Data used for training of the of-the-shelf GROBID has diferent labels than we have in our synthetic data set. Consequently some labels are condemned to have scores equal zero, e.g. web. Note that GROBID does not provide evaluation scores for other labels.