The article describes the results of experiments in the field of automatic localization of bibliographic descriptions (single and group as part of lists) drown up according to GOST 7.0.100-2018 (or close standards). The experiments were performed on the set of unstructured full-text Russian-language documents of various styles. The proposed solution is based on several parameters of bibliographic descriptions: lengths distribution (in characters), the frequency of prescribed punctuation characters and autocorrelation factors. The use of these features in an explicit form during simple classifiers creation made it possible to obtain criteria of Recall and F1-scores, comparable to previously obtained one using structural recognition methods.
One of the actual tasks in the Natural Language Processing (NLP) is the bibliographic
information extraction, which is used for both identification of source documents and
bibliographic search. This task is a special case of metadata extraction. Primarily, such
functionality is used in automatic quality assessment systems of scientific and
educational papers, their peer review, in applications to Bibliometrics and Plagiarism
detection. Bibliographic information extraction from scientific books, theses and articles
eventually helps in saving researcher’s time while performing literature acquisition. In
addition, «metadata information within citation also carries immense importance
especially in the domain of Scientometrics» [
This task also can be characterized as a special case of Text Mining and Entity extraction within the NLP. More precisely, we can talk about extracting objects represented by a set of named entities. The process of extracting bibliographic information consists of three stages: (i) localization of bibliographic data, (ii) definition of object boundaries and its type, (iii) identification of fields, extraction and interpretation bibliographic data. This paper presents the results of experiments to determine informative features for the statistical localization of single and group (as part of a list) bibliographic descriptions in full-text unstructured documents in Russian. 2
The majority of the texts in Russian with the appearance of bibliographic references and descriptions is subject to the standards GOST 7.1-2003, GOST 7.0.5-2008 and GOST 7.0.100-2018. Primarily, we are talking about scientific, academic and educational publications, technical reports, patents, official documents, etc. Of course, other formats (styles) of presentation of bibliographic descriptions are also in use. However, the article considers the localization of bibliographic descriptions in the notation of the GOST standards. Predominantly we considered bibliographic descriptions of books and articles as the most traditional types of publications.
So, based on 10 thousand random bibliographic descriptions from the Russian State
Library web-portal of the [
According to the above standards, the source of bibliographic information is presented by bibliographic records, bibliographic descriptions (BD) and bibliographic references (citations). At the same time, the bibliographic description is the main part of the bibliographic record, and differs from the bibliographic reference (BR) in that it clearly specifies the sequence and formatting of fields (elements) containing bibliographic data. Also in relation to bibliographic description defined the term «prescribed punctuation» – a set of characters used to separate fields within the BD (see Fig. 1).
is a source of is a part of is a source of is a variant of is a source of is an item of
is a variant of is an item of is a part of
GOST 7.0.100-2018 GOST 7.0.5-2008 GOST 7.0-99 Thus, the extraction of bibliographic information from texts involves the identification of such objects as bibliographic records, bibliographic descriptions and bibliographic references and the interpretation of their constituent fields containing bibliographic data. Taking into account that BD may be considered as an extended version of BR, attention should be focused on identifying signs of localization of this object. It should also be noted that in full-text documents bibliographic descriptions may occur in various presentation forms, according to the classification (see Fig. 2). The classification items on the diagram are ordered in such a way that the complexity of identifying such BDs increases from right to left. Wherein, the greatest difficulty is the automatic recognition of short single-level single BDs, which format does not match any known styles. Accordingly, it is easiest to identify multilevel full group (two or more in a row) bibliographic descriptions drawn up according to GOST.
Taking into account the above classification, the following aspects should be listed that
affect the solution of the of the problem of identifying BD in the text:
1. Bibliographic descriptions may be located both in groups (in lists) in certain places,
and by one in arbitrary places within the text.
2. Generally, bibliographic descriptions can be multilingual information objects.
3. Texts of documents can be structured and unstructured [
A review of the available publications showed that the following approaches to
extracting bibliographic information from texts of various documents in Russian were
implemented in practice:
«Antiplagiat» system [
However, unlike other modules, there is no publicly available description of its
algorithm.
The work [
For metadata extraction, various researchers have considered approaches based on
Hidden Markov Models (HMM), Conditional Random Fields (CRF), Support Vector
Machine (SVM), decision trees and neural networks [
Note that in [
Bibliographic Description GOST 7.1-2003 GOST 7.0.5-2008 GOST 7.0.100-2018
Type of field (named entity)
Header Касаткин, А.С. Касаткин А.С., Немцов М.В. Касаткин, А.С.
Person
Title
Responsibility Edition
Output data Электромеханика: учебник
для вузов Электромеханика: учебник
для вузов Электромеханика: учебник
для вузов Location АМ.С.В..КНасеамтцкоивн, п6е-ереирзадб.,.
6-е изд., перераб.
АМ.С.В..КНасеамтцкоивн, п6е-ереирзадб.,.
Numeric Alpha-numeric Москва:
Вшыксошлаая 2004 Москва: Вшыксошлаая 2004 Москва: Вшыксошлаая 2004
Physical
character
istic
Recognition and selection of named entities is a rather trivial task for structured text,
primarily because at its creation certain conventions and patterns are used. However, in
general we are dealing with poorly structured texts. Therefore, direct recognition of BD
base.
4
by structural methods may be preceded by the localization of places in texts where the
presence of BD is most likely. For this, it is reasonable to use a statistical classifier
based on a compact set of simple features. Therefore, this paper investigates the main
statistical properties of bibliographic descriptions. In contrast to the previously
presented data [
Let us consider universal alphabet Ω – the set of all possible characters that can occur in natural language texts, including the empty character ε. In practice, this can be a Unicode character set. Also given a set of prescribed punctuation characters (PPC) Σ of size |Σ|. In addition, over the alphabet Ω, a string of characters A = a0a1 ... an-1 of length n = |A| is defined.
The statistical features of bibliographic descriptions considered below are based on the following set of statements. BD in the general case is a text fragment with limited length. This fragment is composed by arbitrary sequence of the alphanumeric substrings (fields of bibliographic data), separated by characters from the set Σ. In addition to the usual grammatical punctuation marks, there are PPCs between the BD fields. Therefore, on the length of the BD there should be an increased occurrence of prescribed punctuation characters relative to the text as a whole. Then the localization of bibliographic descriptions can be performed using the sliding window. And the first localization model is specified by three parameters: M1 = <S, O, TPPC>, where S is the window size, O is the offset step for window sliding relative to the previous one, TPPC is the threshold of the relative frequency of the PPC set Σ. It is applicable to Bayesian classification scheme.
Further, because BDs are structured entities, they may demonstrate some cyclic or periodic properties with respect to the positions of prescribed punctuation characters. Therefore, there may be increased values of the corresponding indicators in the areas of BD localization in the text. As such an indicator, consider the autocorrelation.
Let φ: Ω → {ε, 0, 1} be a homomorphism where for character x from Ω ( ) = { 1, ∈ Σ,
, = , 0, ℎ .
The autocorrelation (periodic) of a string A length n is defined by (A) = ∑ −1 ( ) ( + =0 ), ∈ ℕ, 0 ≤ ≤ ⌊ ⌋.
2 Averaged autocorrelation coefficient (AACC) is defined by = 1 ∑ =0 c ( ) , 0 ≤ ≤ ⌊ ⌋. Accordingly, the second localization model is also specified by three parameters M2 = <S, O, TAC>, where S is the window size, O is the offset step for window sliding relative to the previous one, TAC is the threshold value of AACC. It is also applicable to the Bayesian classification scheme.
The following datasets were used for the experiments. A sample of about 350,000
bibliographic descriptions was obtained from marked-up lists of references to texts posted
on the websites of the Russian State Library [
A sample of full-text documents was taken from the Russian National Corpus posted
on the Internet and marked up by experts [
The analytical software package Statistica 10 was used to determine the parameters of the distributions. Based on the Kolmogorov-Smirnov test with a significance level of 0.95, it was found that a lognormal law with various parameters for articles, books and theses describes the BD lengths distribution (see Fig. 4).
As shown, in articles and books, short BDs are usually used (mean – 220 and 244 characters, respectively), while in theses, extended and full BDs are often used (mean – 437 characters). Due to this, the mathematical expectation and variance of BD lengths are noticeably higher for dissertations. For convenience and optimization of further calculations, the width of the BD localization window is taken as a value near the average lengths for books and articles – 256 (as 28).
Next, the assumption that the frequencies of occurrence of prescribed punctuation characters (according to GOST 7.0.1-2018) differ within the boundaries of bibliographic descriptions and in the text as a whole was checked. 0.006 0.005 n o it fnu0.004 c y it s n eD0.003 y iilt b a rob0.002 P 0.001 0 0 Articles - LN(5.39; 0.27) Books - LN(5.50; 0.38)
Theses - LN(6.08; 0.51) 200 400 600 800 Length of Bibliographic description (in characters) 1000 The following Table 1 summarizes data on occurrence frequencies of each prescribed punctuation character (PPC) in texts and within bibliographic descriptions. Since the use of PPC in texts and BD differs (for some characters significantly), this allows the use of relative frequencies of such characters as predictors in the logistic regression model for classifying text blocks. From the initial dataset, using a sliding window with a size of 256 characters and with an offset of 128 characters, blocks of two types were selected: those containing no BDs (text blocks) and containing BDs (bibliographic blocks). Then the parameters of the frequency distributions of the full set of PPCs were determined (see Fig. 5).
5 10 15 20 25 30 35 40
The relative frequency of the full set of prescribed punctuation characters, % As a result of averaging the data over blocks of texts of all styles, the threshold of the relative frequency of the full set of PPCs in various bibliographic descriptions was selected TPPC = 0.045. In addition, the probabilities of Type I errors (α, «false positive») and Type II errors (β, «false negative») are 0.0122 and 0.0054 respectively (see Fig. 6). Evaluation of the first classifier with parameters M1 = <256, 128, 0.045> were carried out on samples of 70,000 text blocks of various styles and 150,000 bibliographic blocks obtained by pulling a sliding window. It was found that a Naive Bayesian classifier at the specified threshold, although it allows identifying nearly all of BD locations, but also captures some syntactic constructs. These are addresses, listings of surname-name groups, tabular data, mathematical and formal expressions, legislative acts. 5.3
Similarly, the parameters for the Bayesian model of recognition of bibliographic blocks were determined based on the value of the average autocorrelation coefficient (AACC) calculated on a string with a length of 256 characters (see Fig. 7).
0.20 0.15 F D P 0.10 0.05 0 0
N(0.024, 0.017)
N(0.14, 0.041) 0.05 0.10 0.15 Since the calculation of autocorrelation is a computationally expensive process, we optimized such a parameter as the maximum index of the autocorrelation coefficient used for the calculation. It is founded that the minimum value of the classifier errors sum is reached with the shift K = 30 (see Fig. 8). In general, the proposed method for taking into account autocorrelation properties for localizing bibliographic references, with some refinement, can be used to visualize the structure of documents (see Fig. 9).
Single BD
Bibliographic List In addition, after applying the Fast Fourier transform (FFT) to the autocorrelation sequences sized 128 symbols was investigated the form of the power spectrum. The experiment was carried out on a sample of 45000 blocks: 15000 plain-text blocks, 15000 blocks with mathematical expressions selected from textbooks of physics, chemistry and mathematics and 15000 bibliographic blocks. Based on the results of averaging the spectra, an integral result was obtained that demonstrates significant differences in the spectral characteristics between the indicated types of content (see Fig. 10). 0,16 0,14 0,12 0,1 0,08 0,06 0,04 0,02 0 0,15 0,1 0,05 0 1
11 Math expressions 21
31 41 Bibliographic descriptions 51
61 Plain text Based on the considered statistical features of bibliographic descriptions in the first approximation, an attempt was made to use them as features (in their original form, without any transformations) when constructing simple classifiers separating text fragments on the width of a sliding window into two classes - text blocks and bibliographic blocks. Thus, it becomes possible to localize bibliographic descriptions in an unstructured full-text document. The first two classifiers are built on the basis of a naive Bayesian approach – the decision is made based on comparison with the threshold of a single feature value.
In addition, two binary logistic classifiers (BLC) were built with a decision threshold equal to zero (two classes are given: -1 for text block, +1 for bibliographic block). The first classifier based on the normalized relative frequencies of the six most commonly used prescribed punctuation characters: comma (x1), dot (x2), colon (x3), semicolon (x4), hyphen (x5) and forward slash (x6). All six factors are significant. 1( ) = −0,94 − 9,40 ∙ 1 + 18,14 ∙ 2 + 22,48 ∙ 3 + 9,55 ∙ 4 + 3,58 ∙ 5 + 62,94 ∙ 6. (4)
The second BLC-classifier is based on same set of features, but the seventh feature is added – the normalized value of the averaged correlation coefficient AACC30 (x7). All seven factors are significant. 2( ) = −1,06 − 9,67 ∙ 1 + 15,49 ∙ 2 + 19,53 ∙ 3 + 9,84 ∙ 4 + 2,25 ∙ 5 + 47,02 ∙ 6 + 4,03 ∙ 7. (5) For experimental verification of the presented solutions, in addition to the described set of initial data, with the involvement of volunteers, 500 full-text documents were annotated – one hundred texts of each style. Each document contained a list of references from 80–150 bibliographic descriptions, and 121 documents contained a single BD, 230 contained paired BDs, and 149 contained single and paired BDs in different parts of the text.
To compare and evaluate different classifiers major evaluation metrics include Precision, Recall and F1-score. The test results of classifiers are summarized in Table 2. It should be noted that the obtained values of the AUC-ROC indicator can be described as good, which indicates the prospects of the statistical indicators considered as classification signs for solving the problem of extracting bibliographic information from fulltext documents. The obtained values of Recall, Precision and F-score are comparable with the results achieved to the results obtained in the above studies using structural recognition methods. At the same time, a compact model of BD localization on simply calculated features is implemented.
To implement stages of the presented study specialized software was developed. It provides the ability to vary various parameters of the models (width of the sliding window, parameters for calculating autocorrelation, symbol sets, etc.) and iterate over them in the specified ranges. In this case, the user can visually view the results of BDs localization – text fragments selected from the input documents (see Fig. 11). The lognormal law describes probability distributions of the lengths of bibliographic descriptions, as well as relative frequencies of the prescribed punctuation characters set. A large number of false positives were identified for scientific-style texts, which explained by an abundance of mathematical expressions with punctuation marks. Therefore, the relative frequencies of PPC is a poor predictor for long size BDs (full), which are typical for dissertations. It is not advisable to use this feature in its original form. The obtained values of Recall, Precision and F1-score of classifiers, developed on the basis of the studied statistical features of BDs, are comparable with the results achieved in previous studies using structural recognition methods. It seems promising to increase the Recall value of the classification of the presented compact models in order to achieve the necessary precision at the next stage of processing – applying regular expressions to selected text fragments. Experimental data indicate that the studied statistical features and models are applicable not only to texts in Russian, but also to other alphabetic languages. Note also that there remains some potential for improving performance within the framework of the presented models by varying the sliding window size, changing the original text string encoding way into a binary sequence, changing character sets, using more advanced metrics. It is also worth considering the possibility of using neural networks to solve the stated problem. The presented results are preliminary and intermediate and will be further refined.