In recent years, thanks to Optical Character Recognition techniques and technologies to deal with low scan quality and complex document structure, there has been a continuous evolution and automation of the digitization processes to allow Robotic Process Automation. In this paper we propose a methodology based both on deep learning algorithms (as generative adversarial network) and statistical tools (as the Hough transform) for the creation of a digitization system capable of managing critical issues, like low scan quality and complex structure of documents. The methodology is composed of 5 modules to manage the poor quality of scanned documents, identify the template and detect tables in documents, extract and organize the text into an easy-to-query schema and perform queries on it through search patterns. For each module di erent state-of-the-art algorithms are compared and analyzed, with the aim of identifying the best solution to be adopted in an industrial environment. The implemented methodology is measured with respect to the business needs over real data by comparing the extracted information with the target value and shows performance of 90%, in terms of Gestalt Pattern Matching measure.
With the spread of cameras on mobile devices, more and more images of scanned
documents are collected in order to be digitized for di erent uses. Most of the
digitization processes are still done manually today, however, thanks to recent
advances in machine learning, it is possible to further automate these processes [
When dealing with information extraction from documents, Optical Character
Recognition (OCR) techniques are the key technology, however these alone are
not enough to extract all the visual and structural information from scanned
documents. Moreover their power is limited when dealing with poor-scan
quality documents or with documents with complex structure. In this research we
de ne a methodology for extracting information from scanned or editable
documents, trying to manage and limit the noise coming from the low quality of the
scans and taking into account document structure. The methodology is designed
using real data coming from two di erent companies and is tested by
considering the companies' real business needs. The methodology that we propose rstly
uses Generative Adversarial Network (GAN) [
Information retrieval from documents has been an important research area for
several decades. With the advent of deep learning, OCR systems have become
extremely powerful and usable, thanks to open source systems such as
Tesseract [
The preprocessing step, which aims at eliminating noise in an image without
missing any signi cant information, traditionally was performed with
statistical and computer vision techniques but recently also some deep learning
approaches have been successfully proposed. One of the most used in the eld of
image denoising is represented by GAN [
Algorithms and Approaches
Binarization, skew correction, ltering, thresholding,
compression, thinning [
Rule-based methods [
Extracting information from documents is an active research eld and in recent
years some works on the topic have been published. For instance, in [
The goal of this research is to identify a methodology that allows the creation of an RPA system for extracting speci c information from documents. The information to be extracted are driven by business needs and vary according to the use case of two pilot companies that furnished the data. The goal of these two companies is that of digitizing the information in order to activate a business process of notarization and supplier management. In order to understand the approach to be pursued, in the following the datasets provided to implement the solution are described. 3.1
The datasets used to de ne and test the methodology are composed of a collection of documents from two di erent companies. In one case, the documents are editable pdf of production sheets (in the following we will refer to this data as dataset A), in the other case the documents are scanned pdf of technical product sheets, invoices and transport documents (we will refer to this data as dataset B). In the rst case, the production sheet is composed of di erent pages each composed of a body in the form of a table that contains several production sub-sheets, each representing an order to be performed to speci c suppliers. Production sub-sheets are identi able by a set of contiguous populated lines separated by other production sub-sheets by a white line. The goal is to extract all the lines corresponding to each production sub-sheet to automatically activate the process of supplier selection. In the second case, the dataset is composed of three kinds of scanned documents: invoices, with di erent templates, document of transport with di erent templates, signed and dated by hand, and technical product sheet in the form of a table that contains speci c information about the products (EAN, ingredients, nutrition values, sterility requirements etc). Each document has its own set of information to be retrieved in order to be notarized via block chain. 4
By considering the business needs expressed by the partner companies, a methodology that consists of ve main steps has been identi ed. Starting from the document, a rst phase of pre-processing is envisaged with GANs, with the aim of reducing the noise of scanned documents. Subsequently, a template identi cation module based on deep learning models (CNN or SNN) is implemented. Then a module to detect and extract tables is de ned, looking for the vertical and horizontal lines within the document (using the HT) to trace the number of columns and rows that may constitute them. Follows the OCR phase (with Google Cloud Vision API) to extract the textual content and the content mapping module to encapsulate the information within a matrix schema. Finally, the last module deals with extracting the required information, looking for patterns de ned in advance and depending on the type of document. The goal is to ensure that the methodology can be applied in di erent industrial environments with respect to the business needs expressed. The individual steps of the methodology are described in detail below. 4.1
The rst step of the methodology is represented by the pre-processing phase.
Speci cally, at this stage the goal is to reduce the noise present in the images
of the scanned documents since the quality of the documents in uences all the
next steps of the methodology. As described in chapter 2, one of the most recent
and e ective methods to perform image denoising is represented by GAN [
V (G; D) : minGmaxDEx pdata(x)[logD(x)] + Ez pz(z)[log(1
D(G(z)))] (1)
According to [
D(G(zjy)))] (2)
Using a model of this type it is possible, by providing the model with images of
noisy documents and their respective noise-free, to train the model to produce,
given in input an image of a document, the image of the same document but
with a reduction of the disorder. To perform the training, it is necessary to have
for each input also the target image, i.e. the dataset must be composed of pairs
formed by noisy images and their respective images without noise.
Cycle GAN Another extension of GANs, called cycleGAN [
N M some training examples fxigi=1 where xi 2 X and fyj gj=1 where yj 2 Y . The cycleGAN model includes two mapping functions G : X ! Y and F : Y ! X. moreover two adversary discriminators Dx and Dy are introduced, where the goal of Dx is to distinguish between image fxg and its corresponding translated fF (y)g. The same for Dy with respect to image fyg and the corresponding translated fG(x)g. The goal is twofold: to reduce the opposing losses to align the distributions of the generated images and the distributions of the target images and to reduce the cycle consistency loss to prevent the mapping learned from G and F from being contradictory. 4.2
The second step of the methodology is represented by the document template
identi cation module. This module aims at de ning the template of the input
document and this can be accomplished by any image classi cation algorithm.
In our research we compare two kinds of algorithms: a more consolidated one
based on CNN [
Google Cloud Vision API As an OCR tool we decided to use the Google
Cloud Vision API tool, the documentation of which is reported in [
Hough Transform By analyzing the state-of-the-art for table extraction it
emerges that one of the last trends is based on deep learning techniques.
However, to train such models a lot of data must be available and often it is di cult
to generalize well with public dataset. To overcome this limitation, we follow an
unsupervised approach based on the HT. Before applying the HT to detect lines,
however, we propose to pre-process the image to highlight the lines contained
in it using a computer vision method called Edge Detection [
The fourth module of the methodology takes care of mapping the extracted information into a schema that is easily searchable. In order to quickly access the content of the text, we decided to organize it within a dynamic matrix structure. Indeed, if each content is assigned to a cell whose position is known, it is easy to search for the other contents associated with it in the adjacent cells, since their positions can be used in the text search. As mentioned, thanks to the Google Cloud Vision API, not only the textual content of the document is available, but also the organization and position of the content within the image. Thus, to create the matrix structure we used an approach which assumes that in a document there could be di erent separated tables and each table necessarily has at least one vertical lines. The approach has six steps: 1) Scans all the vertical lines and groups them by considering their ordinates: if lines have overlapping ordinates they are grouped together. 2) For each group of vertical lines add to the group all the horizontal lines that are in the group's range of ordinates. 3) Divide each group into subgroups of directly and indirectly connected lines, where directly connected lines are lines that have a pixel in common, and indirectly connected lines are lines that are not directly connected with each other but are both directly connected with the same line or with a set of indirectly connected lines. Each subgroup thus identi ed represents a table. 4) For each table, detect the number of rows and columns by taking the number of intersections of respectively the vertical and the horizontal lines with the highest number of intersection points. 5) Create a matrix with the identi ed number of rows and columns. 6) Fill in the matrix with the text extracted by the OCR tool selecting the proper cell using the coordinates returned and the coordinates of the table identi ed. This approach generally creates a matrix with more cells with respect to the actual table, since it also consider tables with complex structures (as rows or columns with di erent number of cells). For this reason, in such cases, text is replicated in di erent cells of the matrix if these cells correspond to the same cell of the table. The last step of the methodology deals with retrieving the information of interest starting from the generated content matrix. The goal is therefore to de ne search patterns for each value to be extracted. We de ned two types of patterns: anchor and text. The anchor pattern rst searches for an anchor, i.e. one or more terms from whose position it is then possible to go back to the actual content. An anchor pattern is formed by the elds described in Table 2. The second type of pattern does not provide for the existence of an anchor, but deals with searching directly within the cells for a speci c content that satis es a certain condition. The elds are similar to the previous case but instead of starting searching from the anchor it starts from the rst cell of the matrix. With these two types of patterns it is possible to cover all searches within the matrix and it is therefore evident that the task of retrieving is enormously simpli ed. Indeed, the reorganization of the content in matrices and the two search patterns de ned allows for more complex and exible searches with respect to simple rule-based systems. 5
In the following the experimental results of the methodology are presented. For each step of the methodology, the results are distinguished per dataset A (composed by good quality editable pdf of the same type) and dataset B (composed of scanned pdf of three di erent types) with the aim of verifying that the methodology can be applied to both kinds of dataset. 5.1
To train the GANs algorithms a public Kaggle dataset has been used [
P SN R = 20log p
M SE
M SE =
the class without noise. As far as cGAN is concerned, the network architecture
is composed of 3 layers of convolutions with ReLu activation function, followed
by max-pooling and dropout. As far as cycleGAN is concerned the convolutional
network has been created using a U-Net architecture [
To train the image classi cation module we used the dataset A and B together. Since the dataset A is composed of just one kind of document and the dataset B is composed of three kinds of documents, the outcome of the model is composed of four classes. The images have been divided into training and test with an 80-20 splits. The models compared to perform template identi cation are SNNs and CNNs. As far as the SNN is concerned, the two embedding layers placed in parallel consist of 2 convolutional layers with ReLu activation function followed by fully-connected layer. In order to train the algorithm, training samples have been paired randomly to have 500 paired samples of the same class and 500 of di erent classes. To test the algorithm, all the training images have been used as comparison samples against the test images and, to select the nal class, majority voting has been performed. As far as CNN is concerned, the network is composed of 3 di erent levels of convolution with ReLu activation function, each followed by a max-pooling layer and dropout to avoid over tting and with a nal fully-connected layer. The model has been tested using the same test (3) (4) images of the SNN model. Both the models have been trained using Google Cloud Vertex AI training jobs with automatic hyperparameter optimization. In both cases, learning rate, batch size and number of epochs have been automatically selected by the optimization algorithm. As a comparison metric we decided to rely on overall accuracy (OA). On the four classes, the CNN yielded 93,71% of OA while the SNN model 94,33% of OA. Thus, the SNN model yields slightly better performance, however, its great advantage relies on the fact that, in an industrial environment, this kind of model needs less training data with respect to the CNN, letting it be easier to train and use such a model. 5.3 To evaluate the performance of the table detection algorithm based on the HT, we decided to use the number of lines (vertical and horizontal) correctly extracted, undetected, partially identi ed and exceeding. Furthermore, to understand the actual goodness and usefulness of the pre-processing phase based on cycleGAN, this calculation was done both with and without the application of the cycleGAN model. The results are reported in Table 3.
From the results we can see that the application of GANs does not actually impact the performance of lines detection in the case of dataset A, since these data are software-generated and, thus, perfectly cleaned. Instead, with respect to dataset B, their application enhance the results, showing that the algorithm is able to improve the quality of some scanned documents. Finally, in order to evaluate the overall performance of the methodology, since the nal objective is to retrieve speci c information from documents, we decided to compare each single eld extracted with the target of the extraction. To evaluate the performance of our system we decided to use a metric called Gestalt Pattern Matching (GPM), which assigns a similarity value between two strings S1 and S2, based on the size of the substrings and on the number of matching Km characters between the strings where the matching characters are de ned as the longest common substring (LCS) plus recursively the number of matching characters in the non-matching regions on both sides of the LCS GM P =
2Km jS1j + jS2j (5)
As can be seen from Table 4, in the case of dataset A, the extractor has an exact matching both with the use of the GAN and without. This is likely because the pdf is perfectly clean but also because the search task of these documents is simpler and seems to be less in uenced by a precise identi cation of the tabular scheme (on which, however, there is good performance). As for the dataset B, on the other hand, since the search patterns are more complex and the quality of documents lower, the percentage of matching obtained (better in the case of application of GAN, con rming the e ectiveness of the pre-processing layer) is 0.81. Overall, our methodology has a 90,5% of performance for the task of information extraction in terms of GPM.
This study presents a methodology based on machine learning for the realization of a general and customizable RPA system based on the type of documents and information to be extracted. After an extensive analysis of the state-of-the-art, we developed a modular methodology that could adapt to di erent documents in terms of template and content. For each module we tested di erent options. The identi ed methodology consists of 5 modules: an image denosing module based on cyceGAN; a document template identi cation module, based on SNN; an information extraction module based on table identi cation via HT and on text extraction via Google Cloud Vision API; a custom information mapping module to organize the content into a matrix structure and a query module that extracts the necessary information through search patterns. The methodology has been tested using the GPM score. Overall, the methodology perform well with a score of 0.905 (corresponding to a matching of 90%) and also proved the e ectiveness of the image denoising algorithm. Finally, the implemented methodology has been deployed in di erent industrial environments, with di erent document formats just by ne tuning the template identi cation model and by de ning the search pattern. Some enanchements can be applied for better generalization, such as using a deep learning approaches to detect tables in the document, thus reducing the error in line identi cation and, consequently, in information retrieval, or designing some optimization algorithms to keep the complexity of the SNN model low (due to the necessary scan of the images to perform prediction) that could slow down the extraction of information in an industrial context.