Extracting information from documents usually relies on natural language processing methods working on one-dimensional sequences of text. In some cases, for example, for the extraction of key information from semi-structured documents, such as invoice-documents, spatial and formatting information of text are crucial to understand the contextual meaning. Convolutional neural networks are already common in computer vision models to process and extract relationships in multidimensional data. Therefore, natural language processing models have already been combined with computer vision models in the past, to bene t from e.g. positional information and to improve performance of these key information extraction models. Existing models were either trained on unpublished data sets or on an annotated collection of receipts, which did not focus on PDF-like documents. Hence, in this research project a template-based document generator was created to compare state-of-theart models for information extraction. An existing information extraction model \Chargrid" (Katti et al., 2019) was reconstructed and the impact of a bounding box regression decoder, as well as the impact of an NLP pre-processing step was evaluated for information extraction from documents. The results have shown that NLP based pre-processing is bene cial for model performance. However, the use of a bounding box regression decoder increases the model performance only for elds that do not follow a rectangular shape.
Natural language processing (NLP) methods are widely used on one-dimensional sequences of text. In some cases, for example, in the extraction of key information of invoice documents, spatial information such as the position of text are crucial to understand the contextual meaning. ? Supported by organization KI Group GmbH.
Convolutional neural networks (CNNs) are already common in computer
vision (CV) models to process and extract relationships in multidimensional
data. CV models have been already combined with NLP models in the past
[
These complex models consist of several components, such as an encoder, a
semantic segmentation and bounding box regression that are trained on di erent
targets with di erent loss functions. All this results in a substantial increase
of the model's parameters and overall training time, e.g., the addition of the
bounding box regression decoder as used in [
In this research, a template based document generator is created to
generate datasets of semi-structured documents, that can be used to compare models
for information extraction from documents. Furthermore, this research evaluates
the possible bene ts of a bounding box regression decoder as the rst step to
analyse the e ect of the individual components. As BERTgrid has shown to
improve model performance by applying a NLP pre-processing step, this research
evaluates the bene ts of an NLP pre-processing step also on a generated dataset
of invoices. Therefore, the main contribution of this work resides in providing a
comparative analysis of state-of-the-art models on a publicly available dataset
[
Chargrid [
Chargrid uses the optical character recognition engine Tesseract [
Another state-of-the-art model that combines CV methods with NLP
methods is CUTIE [
The generation process can be seen in gure 1. First, all information to be
extracted from each document is generated in JSON-Format [
The generated PDF-documents are available online [
The presented model follows the auto-encoder structure with one encoder and up to three decoders which are composed of several CNN blocks. The rst decoder forms the semantic segmentation decoder, while the two others form the bounding box regression decoder. All three decoders are based on the same structure and can be di erentiated from each other by the last convolutional block, ending with a speci c convolutional layer for each decoder. The semantic segmentation decoder uses a softmax activation function with an output dimension of 364 256 F , where F describes the amount of elds to be extracted. The last convolutional layer of the rst bounding box regression decoder, also called the box mask decoder, uses a softmax activation function as well, but with an output dimension of W H (2N ). Where N describes the amount of anchors, with one foreground and one background layer per anchor. The second decoder of the bounding box regression decoder uses a linear activation function. The output dimensions of this decoder are W H (4N ), where positional as well as eld size were encoded per anchor.
The full model of the Chargrid paper further follows an U-Net structure. This step increases the training parameters for the models by nearly the factor two. Due to the size of the models and the resulting training and evaluation time, the impact of this step is not yet evaluated in this research. 3.4
The output of the semantic segmentation decoder creates prediction masks for
each eld that should be extracted. OpenCV [
Recognition performance of ve header elds (company-name, company-address, invoice-number, invoice-amount and invoice-date) and three line item elds (itemname, item-quantity and item-amount) are evaluated in this work.
For the experiments, ten di erent templates are designed to generate 12.000 documents (8000 training, 1000 validation, 3000 test) in total. To create varying documents, each eld is shifted by a random o set. The font "tiro" was used during document creation to receive a constant OCR-Engine performance. All three models were trained and evaluated once with the in Chargrid proposed preprocessing method and once with the SpaCy pre-processed documents. Training took, depending on model size, between 50 to 70 epochs to fully converge with a training time of 8 hours for the smaller models, up to 16 hours for the models using all three decoders.
The header- eld results, which can be seen in table 1, and the list item eld results, which can be seen in table 2, show that the ground truth masks, generated in the pre-processing step, do not reach 100 percent in all cases.
This error rate can be explained due to OCR-Engine miss-classi cations. Characters like "7" were often classi ed as "1" by Tesseract for the used font "tiro".
The highest loss in performance can be observed in the invoice-amount elds (99,8 percent to 95,2 percent) and the item-amount elds (76,3 percent to 75 percent). These elds end often with a single currency character, which does not reach a 50% overlap with the predicted bounding box. Furthermore, the ndings of BERTgrid could be con rmed, as the results show that the SpaCygrid experiments outperform the Chargrid experiments, especially for the list item elds, where the performance is improved by over 50%. It can be seen that in most cases, where the bounding box decoder was added to the model, the results do not reach the accuracy of the experiments without this decoder. The addresselds and the item-names, which do not share a rectangular shape like the other elds, achieve better results with the box mask decoder added to the model.
Since no dataset was available to compare models for key information extraction from documents, the template based document generator was used to generate a benchmark dataset with 12.000 documents using 10 dynamic templates. The experiments have shown that this dataset can be used to train, compare and evaluate di erent models for information extraction of invoices in PDF-format. The model using the semantic segmentation decoder only shows the best overall performance when trained on the generated dataset, where most elds that should be extracted follow a rectangular shape. The bounding box regression decoder can be bene cial for masks with more complex eld structures. This research project could con rm the ndings of the BERTgrid paper regarding the bene ts of a NLP based pre-processing method, as the pre-processing step with SpaCy improved the Chargrid results. Especially the performance gap in the extraction of list item elds between Chargrid and the ground truth masks could be drastically reduced with a performance increasing of over 50%. The experiments have shown that the results for a speci c eld depend on the selected overlap threshold, which has to be adjusted to the elds content to achieve the best performance. It could be observed that the OCR pre-processing step is crucial for the total model performance. The limited amount of di erent templates and the accompanying limited variation of the elds positions result in exceptional scores which are not expected to hold in real-world scenarios. 7
The created dataset with the document generator is a rst step to create a common dataset for document processing. The presented work has shown that it can be used to train, compare and evaluate di erent models for information extraction of invoices in PDF format. Furthermore, it was shown that the use of a bounding box regression decoder is bene cial for masks following a more complex structure than a rectangle. More research with di erent types of templates are needed to further investigate at which point the bounding box regression decoder improves performance. The bene ts of a NLP based pre-processing step could be approved, while the performance of the OCR-Engine is crucial in extracting information from PDF-documents. 8
The extension of the generator with more complex and varying templates would be the next step to improve the relevance of the generator for real-world scenarios. It should be investigated weather a semantic enrichment of the text with domain ontologies can improve the results of the classi cation. With more training data and resources, the e ect of model components that have been neglected in this work can be evaluated as well to get a deeper understanding of the different components e ect on model performance.