We propose a novel type of document representation that preserves textual, visual, and spatial information without containing any sensitive data. We achieve this by transforming the original visual and textual data into simplified encodings. These pieces of non-sensitive information are combined into a tensor to form the NonDisclosureGrid (NDGrid). We demonstrate its capabilities on information extraction tasks and show, that our representation matches the performance of state-of-the-art representations and even outperforms them in specific cases.
1. Introduction wards successful digitization for many businesses worldwide. The goal is to transform unstructured or semistructured information into a structured form for various downstream tasks, hence streamlining administrative procedures in banking, medicine, and many other domains.
Because of the typically sensitive nature of the data used for document processing, the data available to train state-of-the-art systems is limited. Private companies are often obliged to delete documents collected from customers after a certain time period and may not share the data with providers specialized in automated document processing at all. This restriction prevents them from training and continuously improving machine learning models.
A lot of the current progress in the field of document understanding builds upon the combination of spatial and textual information into a common document representation. This is often achieved using grid-based methods, which preserve the 2D layout of the document and directly embed textual information into the representation. The models can use textual information in an embedded form and still take advantage of the 2D correlations of the document. These methods encode the text in embedding vectors and transpose these vectors into corresponding pixels of the grid.
BERTgrid [
Based on the assumption that simplified encodings can replace the original information in documents and still retain utility for model training, we define components to transform the original data into non-sensitive informational pieces. Some components are based on textual information, and some represent purely visual parts of the document.
State-of-the-art grid-based representations embed the text more or less directly into the grid. Because the character-level encoding and the word or subword embeddings can potentially contain sensitive information, we need to develop other approaches to incorporate textual information.
Layout-only is a binary text mask and the simplest component in our novel representation. This layer contion in the 2D grid based on the token bounding boxes Category Contains alphabetic (a-z, A-Z) Contains numeric (0-9) Contains non-alphanumeric
Value (1, _, _) (_, 1, _) (_, _, 1) = ( , , ℎ , ) the associated bounding box of the = { 1, if ∃ such as (, ) ≺ 0, otherwise
(1) where ≺ means the point (, ) lies within the bounding ≤ + ℎ ). box (formally: (, ) ≺
⟺ ≤ ≤ + ∧ ≤
Alphanumeric Categorization is a strongly simpli- dividing line to separate content or tabular structures. To
ifed text encoding. In our approach, we encode a
token into a three-dimensional binary vector. These three
ifnd lines in document scans, we use the line segment
detector implementation from OpenCV [
The idea behind this approach is that the key infor- into a binary mask. detected from OCR. This reduces the document to its spa- lower than the original embedding dimensions. Nevphabetic character, a numeric character, and one other non-alphanumeric character, respectively. This encoding is summarized in Table 1. mation relevant for tasks like information extraction often has consistent underlying character properties.
For example, the extraction of monetary values from invoices can be supported if we know which tokens contain numbers; e.g., 65.90 or 23.– would, with our approach, most of the time be encoded with (0, 1, 1) (no alphabetic but both numeric and non-alphanumeric characters).
Locality-sensitive hashing (LSH) [
One possible implementation is LSH based on hyperplanes. For this, we randomly sample hyperplanes in the original input space. For each sample in the original space we determine if it’s on the left or right of each hyperplane, resulting in a dimensional boolean vector which forms our hash. We thus reduce word embeddings to a dimensional binary vector, as every hyperplane randomly splits the embedding space into two categories.
The idea behind this hashing is to have textual information without the possibility of reconstructing the original text in the document.
In our experiments we apply this method to BERT
embeddings [
One could argue that depending on the number of hyperplanes, this hash could enable the reconstruction of the original text. We do not expect this to be an issue with the number of hyperplanes chosen significantly ertheless, this is still an outstanding matter and needs further investigation.
Visually-rich documents contain valuable information outside of detected textual information. Visual elements are incorporated into documents to increase their readability for humans. Hence, downstream tasks in automatic document processing can benefit from these visual features.
Line mask is a method to incorporate line segments into a one-channel binary mask. A line in a document can be part of a rectangular box around textual elements or a closing textual information, we only include lines with a length of at least 10% of the document width. With mathematic rounding, the determined lines are incorporated
The automated extraction of key-value information from document scans such as invoices, receipts, or forms can decrease the manual labor needed for many business workflows. We use this task to compare our novel approach to state-of-the-art representations.
Our work is evaluated on three public datasets covering forms and invoices, the two most common applications for document understanding systems.
FUNSD [
XFUND [
RVL-CDIP Layout [
We replicate the chargrid-net architecture from Katti et al.
[
Replicated from the loss used in Katti et al. [
WAR = 1 − #[ins.] + #[del.] + #[sub.] where is the total number of tokens of a specific field (2) in the ground truth instance.
In the following we show quantitative results of our proposed document representation. First we show the impact of single components by carry out an ablation study We report the results of the ablation study in Figure 1. We experimented with diferent combinations of our developed components: Layout-only (LA), Alphanumeric Categorization (AN), Locality-sensitive hashing (LSH), and the Line mask (LI). For the FUNSD and XFUND datasets, we additionaly compare LSH components with 10 and 100 hyperplanes, denoted as LSH(10) and LSH(100), respectively.
To evaluate the model performance for the key infor- to estimate impact of LSH (5, 6, 7) are only carried out for
followed by a comparison among Chargrid [
sizes seem to be influential when it comes to the impact of the components. When using the Line mask (LI) in combination with Layout-only (LA) and the Alphanumeric Categorization (AN), the model performance increases significantly. The Line mask (LI) without the Alphanumeric Categorization seems to be less efective or, in the case of the RVL-CDIP Layout dataset, even worse than the Layout-only (LA) by itself. Combining all components, including LSH with 100 hyperplanes, yields the best model performance for all three datasets. The LSH component with 100 hyperplanes does perform worse when not combined with the other components.
Except for the header fields in the FUNSD dataset, the BERTgrid outperforms the image and the Chargrid on all