Detection and recognition of text from scans and other images, commonly denoted as Optical Character Recognition (OCR), is a widely used form of automated document processing with a number of methods available. Yet OCR systems still do not achieve 100% accuracy, requiring human corrections in applications where correct readout is essential. Advances in machine learning enabled even more challenging scenarios of text detection and recognition "in-the-wild" - such as detecting text on objects from photographs of complex scenes. While the state-of-the-art methods for in-the-wild text recognition are typically evaluated on complex scenes, their performance in the domain of documents is typically not published, and a comprehensive comparison with methods for document OCR is missing. This paper compares several methods designed for in-the-wild text recognition and for document text recognition, and provides their evaluation on the domain of structured documents. The results suggest that state-of-the-art methods originally proposed for in-the-wild text detection also achieve competitive results on document text detection, outperforming available OCR methods. We argue that the application of document OCR should not be omitted in evaluation of text detection and recognition methods.
Optical Character Recognition (OCR) is a classic problem
in machine learning and computer vision with standard
methods [1, 2] and surveys [
main of documents range from open-source projects such as TesseractOCR [2] and PP-OCR [1] to commercial services, including AWS Textract [18] or Google Document AI [19]. Despite Document OCR being a classic problem with many practical applications, studied for decades [22, 23], it still cannot be considered ’solved’ – even the best engines struggle to achieve perfect accuracy. The methodology behind the commercial cloud services is typically not disclosed. The most popular1 open-source OCR engine at the time of publication, Tesseract [2] (v4 and v5), uses a Long Short-Term Memory (LSTM) neural network as the default recognition engine.
2.2.1. Regression-based Methods Regression-based Methods follow the object classification approach, reduced to a single-class problem. TextBoxes [25] and TextBoxes++ [26] locate text instances with various lengths by using sets of anchors with diferent aspect ratios. Various regression-based methods utilize
an iterative refinement strategy, iteratively enhancing using post-processing on so obtained pixel maps. The
the quality of detected boundaries. LOMO [27] uses an binary, deterministic nature of such pixel classification
Iterative Refinement Module, which in every step re- problem may cause learning confusion on the borders
gresses coordinates of each corner of the predicted bound- of text instances. Numerous methods address this issue
ary, with an attention mechanism. PCR [28] proposes a by predicting text kernels (central regions of instances)
top-down approach, starting with predictions of centres and appropriately gathering pixels around them. PSENet
and sizes of text instances, and iteratively improving the [32] predicts kernels of diferent sizes and forms
boundbounding boxes using its Contour Localisation Mecha- ing boxes by iteratively expanding their regions. PAN
nism. TextBPN++ [
methods from Section 2.2 with diferent approaches: for regression-based methods, we included TextBPN++ as a vertex-focused algorithm and DCLNet as an edge-focused approach. From segmentation-based methods, we selected DBNet and DBNet++ as pure segmentation and PAN as an approach linking text pixels to corresponding kernels. Finally, CRAFT was chosen as a character-level method.
1–7
DBNet [13], DBNet++ [7] and PAN [14] were fine-tuned
for 100 epochs (600 epochs in case of FUNSD) with batch
size of 8 and initial learning rate set to 0.0001 and
decreasing by a factor of 10 at the 60th and 80th epoch (200th
and 400th for FUNSD). Baselines, pre-trained on
SynthText [38] (DBNet, DBNet++) or ImageNet [39] (PAN),
were downloaded from the MMOCR 0.6.2 Model Zoo [40].
DCLNet [12] was fine-tuned from a pre-trained model
[41] on each dataset for 150 epochs with batch size of 4,
initial learning rate of 0.001, decaying to 0.0001. For each
dataset, TextBPN++ [
of document processing, is to recognize the text within the detected instances. Therefore, to evaluate the suitability of popular in-the-wild detectors for document OCR, we perform end-to-end measurements with the following text recognition engines: SAR [20], MASTER [36] and CRNN [21]. The open-source engines were combined with the detection methods in a two-stage manner: the input image was initially processed by a detector, which returned bounding boxes. Afterwards, the corresponding cropped instances were passed to recognition models. As a point of reference, we compare both the detection and end-to-end recognition results of the selected methods with predictions of three common engines for end-to-end document OCR: Tesseract [2], Google Document AI [19] and AWS Textract [18].
3.1 on the datasets from Table 1 are presented in Table 2.
On FUNSD dataset, DBNet++ achieves both the highest detection recall (97.40%) and F1-score (97.42%). The highest precision rate, 97.84% was scored by CRAFT. PAN performed the weakest out of all considered in-the-wild algorithms, scoring just 81.44% F1-score. Despite having achieved better results on FUNSD, segmentation-based approaches were outperformed by regression-based methods on CORD and XFUND. TextBPN++ proved to be the best performing algorithm on CORD in terms of recall and F1-score, scoring 99.74% and 99.19%, respectively. DCLNet, for which the best precision rate on CORD (98.67%) was recorded, achieved superior results 3.3. Metric on XFUND, outperforming the remaining methods with respect to all three measures: precision - 98.22%, recall To measure both detection and end-to-end performance, - 98.17% and F1-score - 98.20%. Out of the considered we used the CLEval [37] metric. Contrary to metrics such popular engines for end-to-end document OCR, AWS as Intersection over Union (IoU) perceiving text on word- Textract presented the best performance on the domain level, CLEval measures precision and recall on character of scans of structured documents – FUNSD and XFUND – level. As a consequence, it slightly reduces the punish- scoring 96.69% and 92.65% F1-score, respectively. Google ment for splitting or merging problematic instances (e.g Document AI generalized remarkably better to distorted dates), providing reliable and intuitive comparison of the photos of receipts from the CORD dataset, achieving quality of detection and recognition. Additionally, the 93.30% F1-score, surpassing the scores of AWS Textract Recognition Score evaluated by CLEval, approximately and Tesseract. The results show that in-the-wild detectors corresponding to the precision of character recognition, fine-tuned on document datasets can outperform popular informs about the quality of the recognition engine specifically on the detected bounding boxes.
be fully representative due to diferences in splitting rules. Recognition Score for AWS Textract reached almost 96%, E.g. Document AI creates separate instances for special surpassing CRNN’s scores by c.a. 2%. This suggests that symbols, e.g. brackets, leading to undesired splitting the recognition engine used in AWS Textract, performof words like "name(s)" into several fragments, lower- ing much more accurately on FUNSD than the CRNN ing precision and recall. On all experimented datasets, model, may have been a crucial reason for the good all fine-tuned in-the-wild text detection models reached results. When evaluated on CORD, models with Difhigh prediction scores, proving themselves capable of ferentiable Binarization scored the highest marks in all handling text in structured documents. Qualitative anal- end-to-end measures: recall (DBNet++), precision and ysis of detectors’ predictions revealed that the major F1-score (DBNet); significantly surpassing the remaining sources of error were incorrect splitting of long text frag- methods. Interestingly, despite obtaining the best recall ments (e.g e-mail addresses), merging instances in dense rate, DBNet++ did not beat the simpler DBNet in terms text regions and missing short stand-alone text, such as of end-to-end F1-score. The predictions of regressionsingle-digit numbers. based approaches, better than segmentation-based ones when pure detection scores were measured, appeared to 4.3. Recognition Results combine slightly worse with CRNN. TextBPN++, however, remained competitive, achieving similar results End-to-end text recognition results combining fine-tuned to DBNet and DBNet++. Recognition Scores of CRNN, in-the-wild detectors with SAR [20] and MASTER [36] regardless the choice of in-the-wild detector, exceeded models from MMOCR 0.6.2 Model Zoo [46], and CRNN 93% on FUNSD and 98.5% on CORD, once again demon[21] from docTR [45] are listed in Table 3. The XFUND strating the suitability of applying these algorithms to dataset was skipped for this experiment since it contains document text recognition. SAR model, not specifically Chinese and Japanese characters, for which the recog- trained on documents, presented poorer performance: nition models were not trained. On FUNSD, the end-to- the highest measured F1-scores on FUNSD and CORD end measurement outcomes followed the patterns from were 86.36% and 85.25%, respectively, both obtained by detection: equipped with CRNN as the recognition en- the combination with TextBPN++. Fine-tuned SAR modgine, DBNet++ proved to be the best tuned model in els achieved slightly higher F1-scores reaching 89.49% terms of CLEval end-to-end Recall (93.52%) and F1-score on FUNSD (equipped with DBNet++ as the detector) and (92.23%), losing only to CRAFT in terms of precision. 93.77% on CORD (combined with TextBPN++ detections). Much higher F1-score (+2%) was measured for AWS Tex- Despite gaining a noticeable advantage over the basetract, whose end-to-end results outperformed all of the line, fine-tuned SAR models did not surpass the perforconsidered algorithms. It is important to note that the mance of the pre-trained CRNN. Similarly to SAR, the pre-trained MASTER model [46] worked the best in com- cess. In particular, fine-tuning models such as DBNet++ bination with TextBPN++, achieving F1 score of 83.00% or TextBPN++ yielded over 96% detection F1-score on on FUNSD and 93.26% on CORD. FUNSD, over 98% detection F1-score on CORD and over 96% detection F1-score on XFUND, with respect to the CLEval metric, outperforming Google Document AI and 5. Conclusions AWS Textract. Moreover, combining these detectors with a publicly-available CRNN recognition model in a twostage manner consistently achieves over 90% CLEval end-to-end F1-score, even without explicit fine-tuning of CRNN. We hope the results will bring more attention to evaluating future Text Detection methods not only in the text-in-the-wild scenario, but also on the domain of documents.
Text detection research has witnessed great progress in recent years, thanks to advancements in deep machine learning. The recently introduced methods widened the range of possible applications of text detectors, making them viable for in-the-wild text spotting. This shifted the attention towards more complex scenarios, including arbitrarily-shaped text or instances with non-orthogonal orientations. With automated document processing remaining one of the most relevant commercial OCR Acknowledgement applications, we stress the importance of determining whether the state-of-the-art methods for scene text spot- We acknowledge the help of Bohumír Zámečník, an exting can also improve document OCR. Our experiments pert on OCR systems, who helped with the supervision prove that detectors designed for in-the-wild text spot- of Krzysztof’s internship project. ting can indeed be applied to documents with great suc