We develop the document analysis system, which is able to extract text and text metadata (such as font size and style), and restore the document structure. Some parts of the pipeline are based on machine learning thus requiring training and the labeled dataset, creating a training dataset is based on manual labeling. In this article, we describe an approach to the creation of a labeling system in the task of multiclass classification of document lines (paragraphs). The pipeline consists of several stages ranged from getting the source documents to getting a ready-to-learn dataset. An approach to the analysis of scanned documents and documents in docx and txt format is considered. In our work, we focus on intra-team labeling, thus we do not consider some problems, common for the crowdsourcing approach (such as unscrupulous annotators).
As a rule, large documents are not uniform but split into smaller parts. The scientific articles are divided into sections, the novels divided into chapters, etc. The larger parts, in their turn, are divided into smaller parts, as subsections or paragraphs. Thus the document can be represented in the form of the tree.
The physical representation of this logical structure is the Table Of Contents, it serves to facilitate the navigation in the large texts and also can be hierarchical. Thus in this article we can consider the task of logical structure extraction and TOC extraction as synonymous.
Most of the methods of automatic TOC extraction are based on supervised machine learning techniques, thus requiring some labeled dataset. One of the ways of obtaining the labeled dataset is to ask human experts to label the data. These humans are called annotators. One can see main pipeline (in solid lines) and alternative way where feature extraction and classification preformed by the annotator (in dotted lines) in figure 2.
• Each new type of documents has its own type of logical structure, thus it requires its own labeled dataset. For example, we can work with scientific articles but want to work also with financial documents • The existing labeled dataset does not contain some important subcategory of documents.
For example, the scientific articles dataset does not contain any biological articles. In this case, the quality of the logical structure extraction from biological articles probably will be low, and the best way to improve it is the expansion of the collection of labeled documents. • Generally, the enlargement of the training set is the simplest and efective way to improve the quality of any supervised machine learning model.
This paper is organized as follows: • Section 2 describes related work gave a short description of the current state of the task of TOC extraction and the existing approaches to the creation of the training dataset. • Section 3 describes our approach to the creation of the dataset and the way to solve the problem, which emerges in the process of the creation of a labeled dataset for the task of TOC extraction.
One of the early surveys, considering the logical structure extraction task is [
In the 2008–2013 the series of TOC extraction competitions from fiction books [
In the 2019–2021 the series FinTOC competitions were held – TOC extraction competitions
from financial reports [
The approach, used in the Docreader project is based on the XGBoost classifier [
We can conclude that most modern approaches to TOC extraction are based on machine learning methods and, accordingly, require a labeled dataset.
Depending on the specific labeling problem one can rely on crowdsourcing or do the job in-team. Each approach has advantages and disadvantages. Crowdsourcing allows the creation of large datasets with the assistance of external (and often working for the little money) annotators. One can use Amazon Mechanical Turk1 or Yandex Toloka2 as a crowdsourcing platform. The disadvantages of the crowdsourcing approach are includes: • It is impossible to give data containing state or commercial secrets to outsourcing. • Some annotators may work unscrupulous, some external quality control is required. • You would be limited with the platform restrictions.
In the case of in-team labeling the work is performed by team members, who often work for
much more fee then the crowdsourcers. On the other hand in the case of in-team labeling, some
1https://www.mturk.com/
2https://toloka.yandex.ru
problems are not actual. Usually one may not worry about unscrupulous annotators, it is easier
to organize labeling of the secret documents. There are many tools for labeling, one of the most
powerful tools is a Labeling Studio[
1. Extraction of the text lines with metadata (font size and style, indents, etc) from the document. 2. Extraction of the features from lines with metadata 3. Classification of the lines by their type 4. Construction of the document structure We need labeled data to train the classifier 3 (and sometimes the feature extractor 2).
As a rule, the result of a labeling task are pairs of features and the label But the way how we extract lines with metadata and how we extract features may change, and we don’t want to redo the data labeling task every time when any step of the pipeline is changed.
We add special persistent id for each line with metadata, which is not changed during the change of our pipeline. Thus we are able to change our pipeline (for example feature extraction) and do not need to relabel the training dataset. The way how to build the persistent id is diferent for diferent kinds of documents.
Scanned documents: Scanned documents in fact are images, so line detection is a separate task, and we use Tesseract5 for this purpose. The change of the Tesseract version may lead to the change in the text line location algorithm and even to the change of number and the order of lines. In order to avoid the need to perform the labeling task each time when we change the Tesseract version, we save the found lines in the form of the bounding box and the extracted text.
3as of the beginning of 2019 4https://github.com/dronperminov/ImageClassifier 5https://github.com/tesseract-ocr/tesseract
In the future, we use the found bboxes as a result of the work of the Tesseract. Txt document: Line in txt document is defined by the document itself and the number of the line. We define the line id as md5sum(document) + "_" + line_id Docx: Docx is the Microsoft Word format. It represents by a zip archive with files in XML format. One of the files consists of paragraphs, each paragraph contains text and some meta information in the explicit form or as a reference to some style. Paragraphs located in file document.xml, styles defined in the style.xml file, the archive can hold other files also. The paragraph is defined by the XML which produces it and by the docx file itself. We define the line id as md5sum(document) + "_" + md5sum(paragraph_xml)
We hope that the one who is interested in labeled data should maximally simplify the process for the annotators. The annotator should not install the strange software with the cumbersome installation instructions, the annotator should have direct access to the instruction. In our case, we reduce the task to the classification of the image. Annotator gets a zip archive with images of the bounding box (as in picture 3), the annotator instructions, docker file with all the dependencies. The task can be launched with two shell commands (in case if you have docker installed), after launching one can use a web browser to perform the tasks. We use ImageClassifier to create web interfaces for the labeling task. The annotator labels the document line by line, after the first line following the second line, and so on, so the context is holding. After finishing the labeling task annotator gets the archive with the results and is able to upload it into the tasks server. When all annotators have uploaded their tasks, task server merges the answers and adds original documents. As a result, we obtain the collection of documents and labeled pairs line id and the label. The line id should be the same for each run of the pipeline, we describe how to build such id in the subsection 3.1. 3.2.1. Creating tasks images We have to create images with a bounding box around the text line. The way how to create is diferent for the diferent kinds of the documents.
Scanned documents: A scanned document is a picture, we have the coordinates of the line
from the Tesseract and have saved it. Thus the bounding box can be drawn with the help of the
OpenCV [
Txt documents: Txt document containing only text lines without the metadata (such as font size or font style). Thus one may draw text with some image processing library and do not fear losing some important meta information.
Docx documents: Docx document is the most dificult one to obtain the image. Typically the docx document contains a lot of valuable information about the text formatting, so we do not want to draw the document as raw text and lose all the metainformation. On the other hand, the process of the drawings of the docx document is complicated, so only some large libraries are able to do it. The solution may be found in the modification of the internal XML. One may add information to the paragraph that should be concluded into the box. To obtain the coordinates of the box we tried to convert the modified document and the original one into images and subtract the second image from the first one, but note that the drawing of the box leads to the shift of the paragraphs. We also note that the box of the neighborhood paragraphs may be merged if both boxs have the same color. All the above forces us to use more complicated ways of creating images with bounding boxes for docx: 1. Create a pair of documents with the boxes. Each box in the first document have a unique color and the color of the box in the second document alternates. 2. Convert pair of the docx documents to the pair of pdfs and pdfs into list of images. See picture 4 3. We subtract the second image from the first one and obtain the image with only nonzero pixels in the former box.
The task of creating a training dataset occurs regularly in the process of machine learning-based system development. We describe our approach to the creation of the labeled dataset. We have described our approach to the creation of the labeled dataset, describe an approach that enables us to not redo data annotation with every change of our documents handling pipeline, described how to lead the task of line annotation to the task of image classification. We hope that the need for human labeling data is not one time task, but a regularly occurring problem, so the machine learning systems should enable to create such tasks easily.