Digital mathematical libraries are today one of the tools for integrating mathematical knowledge. This integration method is based on the use of metadata. Our task was to create methods to programmatically extract the necessary objects from digital mathematical documents, establish semantic relationships between them and generate the necessary sets of metadata. Based on the analysis of the structure of the set of documents under consideration and the stylistic features of their design, an algorithm has been developed for extracting their metadata, creating digital collections and then including them in the corresponding digital library. The algorithm is implemented as a software system and tested on the example of a set of files “Proceedings of the N.I. Lobachevskii Mathematical Center” for 1998-2018. A corresponding digital collection has been created, which is included in the Lobachevskii Digital Mathematical Library (Lobachevskii DML, https://lobachevskii-dml.ru/).
Currently, one of the tools for integrating mathematical knowledge is digital
mathematical libraries (see, for example, [
Periodic journals with a long history of publication have archives of articles that
differ significantly in the composition of the metadata presented in these articles. In
addition, style rules, fixed, for example, by MS Word templates or in .sty-files (for
the TEX system), have also changed many times over the past. For example, in the
journal “Lobachevskii Journal of Mathematics” (the articles of this journal constitute
one of the collections of the digital mathematical library Lobachevskii DML) from
1998 to 2019, foor different .sty-files were used to design articles. Algorithms for
extracting metadata use such stylistic features as heading, annotation fonts, and the
document structure that determines the sequence of structural blocks (name, authors
list, etc.) (see, for example, [
Automated processing of arrays of scientific documents is complicated by the
variety of styles used in journals. As a result, various methods of extracting metadata are
required, which will take into account the structural features of a particular collection
(see, for example, [
This work is devoted to the development of methods for creating digital scientific collections from an array of heterogeneous digitized documents. On the example of processing a set of files containing volumes of “Proceedings of the N.I. Lobachevskii Mathematical Center” (hereinafter referred to as the “Proceedings”) for 1998–2018, describes the process of forming the corresponding digital collection and its inclusion in the digital mathematical library Lobachevskii Digital Mathematical Library (Lobachevskii-DML, https://lobachevskii-dml.ru/), which is currently being formed in Kazan (Volga region) Federal University.
The main purpose of this “Proceedings” is the publication of materials of mathematical conferences. As a result, most volumes of the “Proceedings” contain dozens of articles with a limited (from the modern point of view) metadata composition. Since 1998 (the moment the first volume was released), several style rules for the preparation of materials have been used, which affected the design of articles and the choice of file formats of compiled collections.
Let us single out the main tasks of forming a digital collection. The necessary conditions for creating a digital collection from the “Proceedings” array were: division of volumes into separate articles; highlighting metadata describing each article; generation of additional metadata that contains, in particular, a bibliographic description of the article, a link to the article file in the digital collection, as well as links to the profiles of article authors on academic portals and scientometric databases (kpfu.ru, MathNet.ru, Scopus, DBLP and etc.).
Let us describe in more detail the main stages of the software processing of the selected set of files of collections of articles that have been processed. These stages are as follows: highlighting metadata; their conversion in accordance with XML schemas; creation of a digital collection; its inclusion in the digital mathematical library Lobachevskii-DML. 2.1
First of all, file clustering was performed. As a result, the corresponding volumes were divided into classes according to the similarity of their structure and design. From the moment the first volume was released (1998) until the publication of the last 57th volume (to date), various rules were used to prepare the materials. This affected the file formats of the volumes, as well as the design of the articles themselves. For example, only a number of documents contain indexes of the Universal Decimal Classification (UDC). Table 1 shows the differences in the use of structural elements and the composition of the metadata of this collection. Further, in order to extract metadata describing both the volume as a whole and the articles included in it, the collection files were processed. In particular, the page numbers of all the articles of each volume were determined. To search for pages with article titles, an algorithm has been developed that uses the structural homogeneity of each volume and style uniqueness in the design of articles in it.
An important part of the structural analysis of documents is the allocation of blocks
such as the name, surnames of the authors, their affiliation, abstract, key words and
bibliographic records. Special ontologies have been developed to describe the
structure of scientific documents [
To extract article metadata based on characteristic features, we have defined rules for selecting article blocks. Such features include, in particular, the design of article styles (font, size, use of selections, and a number of others). Improving the quality of metadata extraction provides some additional features that take into account: • text structuring (for example, the location of the word “Annotation” in front of the annotation block); • the type of email address record template used; • the position of the block in the text (for example, the document begins with the title of the article).
As the main elements of the article taken into account by these functions, you can use the position of the block in question in the document, as well as the font used in the text of this block. These functions made it possible to distinguish not only the titles of articles, but also lists of authors, bibliography blocks, and other metadata (for example, e-mail, keywords) if they exist in the text.
Using text analysis methods [
A number of metadata (such as email addresses of authors, their affiliation) was imported and updated from the profiles of authors on academic sites and in scientometric databases. In this procedure, the semantic relationships established during the formation of the digital collection were applied. 2.3
We have developed an XML language for describing digital mathematical collections, consisting of a set of tags and XML schemes based on the Journal Archiving and Interchange Tag Suite (NISO JATS, https://jats.nlm.nih.gov/archiving/). In the notation of this language, based on the data obtained during the processing of an array of files, a description of the collection of “Proceedings” is performed. 2.4
The next stage in creating the digital collection consisted of the procedures for dividing each volume of the “Proceedings” into separate articles. To do this, tags whose attributes indicate the start and end pages of the articles were read from XML-files that contain meta descriptions of volumes. After that, the files were divided into separate documents, which were named in accordance with the rules of the digital collection.
The system of metadata prepared in the process of the above algorithm allowed to
form a digital collection of “Proceedings of the N.I. Lobachevskii Mathematical
Center” and include it in the digital library Lobachevskii DML
(https://lobachevskiidml.ru/) [
The algorithm is implemented in the form of programs in C #, allowing you to
process files in the formats TeX, OpenXML (.docx) and .pdf. TeX-files were
generated using standard functions that implement operations with text strings. To work
with pdf-files, we used the functions of the PDFLib libraries
(https://www.pdflib.com) and iTextSharp
(https://www.nuget.org/packages/iTextSharp/). For documents presented as
docxfiles, the “word/document.xml” file was parsed from the .docx archive in
accordance with the Office OpenXML format (see, for example, [
The process of selecting articles was carried out using a program developed in Python using the functions of the PyPDF2 library (http://pybrary.net/ pyPdf/). 3
For inclusion in the international scientific space of digital mathematical collections of Kazan University, methods of their formation from a set of documents presented in various storage formats are proposed. Based on the analysis of the structure of documents and the stylistic features of their design, an algorithm for the extraction of their metadata has been developed, implemented on the example of “Proceedings of the N.I. Lobachevskii Mathematical Center”.
The work partially contains the results of the project “Monitoring and standardization of the development and use of technologies for storing and analyzing big data in the digital economy of the Russian Federation”, carried out as part of the program of competence of the National Technological Initiative “Center for storing and analyzing big data”, supported by the Ministry of Science and Higher Education of the Russian Federation under the Treaty of Moscow State University named after M.V. Lomonosov with the Project Support Fund of the National Technological Initiative dated 15/08/2019 No. 7/1251/2019. The work was also carried out with the partial support of the Russian Fund for Basic Researches (project 18-29-03086); with the partial support of the Russian Fund for Basic Researches and the Government of the Republic of Tatarstan within the framework of scientific project 18-47-160012.