Document Information Localization and Extraction (DocILE) is attracting a large amount of attention from the research community due to its potential to significantly reduce manual work. With the explosive growth of technology as they are today, we want to experiment with a method that leverages the advantages of language models in information extraction since it requires an understanding of the contextual information of the text, which large language models are currently successful on. The experiments include using a new combination of published baseline with our model , along with a post-processing step, which helped us achieve a Top 3 position in the competition ranking board.
Extracting information from documents is an indispensable part of human activities in the modern era. However, manual information extraction is time-consuming and labor-intensive. Therefore, automating the process of extracting information has gained much attention from the research community as it has high applicability in reducing workload for workers in manual tasks and creating opportunities for them to focus more on strategic work.
The information extraction process is challenged since it requires an understanding of the semantics, layout, and context of content in the documents. In Machine Learning (ML), the scope of addressing this issue is called Document Information Extraction (IE), a part of Document Understanding.
DocILE 2023 [
The pipeline to localize Key Information and extract them is built upon the provided baselines
for the DocILE competition, and we acknowledge their contributions [
Our focus lies in optimizing their pipeline. Specifically, we leverage diferent versions of
- a large language model that is used to achieve state-of-the-art results on GLUE,
RACE, and SQuAD in Natural Language Processing [
After evaluating several models provided, we see that one model only performs well in certain
categories while the opposite thing happens for the other models. This leads us to the idea of
using Ensemble [
We first implement Ensemble using Average and Max Voting [
From the concept of afirmative ensemble presented in prior research [
Specifically, if any of the models predict that certain content belongs to a particular field type, we consider that content to actually belong to that field type.
In this task, we are provided an abnormally huge amount of unlabeled data compared to the
small amount of labeled data [
Pseudo-Labeling [
Here, we train the model on the pseudo-labeled dataset instead of mixing it with the labeled dataset, because it is a dataset that we have little control over, and it may contain cases that are completely diferent from the training dataset. Training the model on the pseudo-labeled dataset for some epochs helps the model approach more types of data, thereby learning general features. Then, we fine-tune the model on the training dataset to learn the correct features for each specific problem, helping the model improve its efectiveness on that problem.
The models we use struggle in distinguishing information that have the same field type but is a bit far apart. This leads to the problem that even though the information belongs to the same field type and the same bounding box, the model predicts it as multiple diferent bounding boxes. Moreover, after observing the prediction results compared to the ground truth, and experimenting on various documents, we found that it is rare for information of the same field type to be close to each other on the same document.
We first find the distance between the centers of each pair of bounding boxes belonging to the same field type predicted by the model. Then, we experiment with grouping these bounding boxes on diferent thresholds. For each pair of bounding boxes of information belonging to the same field type, if their Euclidean distance is below or equal to the threshold, we will merge those two bounding boxes into a new one, its coordinate is calculated by using formula (1). How the Post-processing work is shown in Figure 3.
= (min(left; l′eft), min(top; t′op), max(right; ′right), max(bottom; b′ottom)) (1)
By doing this, we can reduce the number of false bounding boxes predicted by the model and improve the accuracy of the predictions.
We maintain the same dataset partitioning as provided by the organizer, with the information of each dataset used for the Training, Validating, and Testing processes as described in Table 1.
Most experiments below were conducted on an environment consisting of 4 RTX 2080 Ti 12GB GPUs, along with the following selected parameters and hyperparameters: • Train batch size = 4 • Test batch size = 4 • Gradient Accumulation Steps = 4 • Weight decay = 0.01 • Data Loader workers = 32 • Training Epoch = 500 • Learning rate = 1e-5
At the same time, we use the validation set to evaluate the model and compare the performance
between diferent models based on the results.
3.2. Model
We use 3 models, 2 published baseline models [
For R _ , we fine-tune it from RoBERTa [
For all 3 models, we replaced the default optimizer, from AdamW to Lion Optimizer
[
• R _ trained with FGM technique and Lion Optimizer • R _ _ trained with Lion Optimizer • R _ _ without any changes
Table 4 demonstrates that Afirmative Ensemble produces significantly better results compared to commonly used methods like Max-Voting and Average. Therefore, we will employ the Afirmative Ensemble technique on our three models to predict the output.
We performed the Post-Processing method mentioned in Section 2.3, on the predicted output of each of the 3 models with diferent percentage thresholds of the document width. E.g. For a document with a width is 2000px, the 14.5% threshold means that it will merge two bounding boxes whose distance is below 14.5% of 2000px or 290px.
As shown in Table 5, the threshold of 14.5% of the document width gives the highest result when evaluated on the validation set. From now on, we will use 14.5% as the default threshold. Combining this Post Processing method with the methods we mentioned in Section 2 significantly improves the results.
Due to the large number of documents, we will do this technique on each chunk from provided chunks dataset. Starting with the chunk0 dataset, we pre-process it as follows: • Remove documents belonging to clusters = -1, i.e., documents whose layouts do not appear in the Train dataset. • Remove documents too big, i.e. documents with a size larger than 3000 pixels in any dimension.
• Remove rotated documents.
We then ensemble the 3 models trained on the Train dataset, combining with the Post Processing. Afterward, we predict on the unlabeled chunk0 dataset to generate pseudo annotations for that, which we call pseudo0. After that, we train 3 models on this dataset with the following hyperparameters: • Epoch: 30 • Learning Rate: 1e-5 • Epoch: 300 • Learning rate: 5e-6
Later, we use the Train dataset to train all 3 models more with the following hyperparameters:
The addition of Pseudo-Labeling slightly improved our results as shown in Table 6. However, this method was implemented when the competition was in its final days, which only allowed us to perform it on one chunk of data. Nevertheless, we believe that continuing to use the remaining chunks will continue to improve the final results.
Overall, our results increased significantly compared to the baseline, +0.082 on the Valset and +0.073 on the Testset, which is shown in Table 9. However, we believe there are still many things we can do to further improve the results: • Use more unlabeled data. Currently, only a very small fraction (10k out of almost 1M) of the unlabeled data was used.
• Use models incorporating layout features such as LayoutLMv3, LiLT, etc.
In this paper, we have presented a solution for the tasks required in Track 1 KILE of DocILE
2023. Our improvements to the baseline [