A Survey on Writer Identification and Recognition Methods with a Special Focus on Cultural Heritage

Abstract

This paper reviews the state-of-the-art contributions for writer identification and recognition with a special focus on applications in the domain of cultural heritage. The task of writer recognition has only recently been recognized as a problem that can be solved by the methods available in the computer vision domain. A number of researchers have explored the performance of deep learning and transfer learning techniques for writer identification in historical documents, and for this purpose various datasets have been used, including the Avila Bible dataset, Historical-WI, HisFragIR20, IAM, HWDB and others. This paper analyses relevant methods used for writer identification and recognition in historical and medieval documents. It also makes a distinction between classification based on words, patches, or whole pages. The results indicate that the current literature supports using deep learning and transfer learning methods, as they are found to achieve the highest performance.

Keywords

• 1 Writer Recognition
• Survey
• Image Recognition
• Cultural heritage

1. Introduction

In recent years, we have been witnessing the development and emerging implementation of deep learning methods for solving a variety of problems in different areas. However, one of the most prominent uses of deep learning is in the domain of computer vision, particularly for the task of image recognition where the aim is the distinction of people, places, objects, characters, and actions. For such purposes, convolutional neural networks (CNNs) which represent a type of neural network that automatically extracts features from data, have been successfully utilized.

Handwritten character recognition in the domain of cultural heritage is a complex task that requires well-suited techniques and an extensive dataset. Machine learning approaches have significantly improved the task of recognition, but when using deep learning approaches, and especially transfer learning, the accuracy of the developed recognition models can be greatly increased.

A frequent problem that occurs when working with such datasets is a limited number of documents, especially if the aim is to employ deep learning. However, this problem can be solved by using transfer learning approaches where the models are pre-trained on much wider sets of data that usually consist of thousands and even millions of images, and then are applied to the selected dataset. The main challenge in historical handwritten character recognition is the large variety of handwritten styles between the writers. This challenge is even more complex due to the intense degradation of documents, such as the appearance of page stains, mold, and text fading, which makes them unreadable. However, machine learning and deep learning techniques can address the degradation removal [ 1 ] and thus provide the restoration and protection of ancient documents.

The aim of this paper is to review relevant state-of-the-art approaches used for writer identification and recognition, with a special focus on the domain of cultural heritage and the identification of writers from historical documents.

In the next section, state-of-the-art approaches for writer recognition will be described, with a special focus on the domain of cultural heritage, followed by discussion and conclusion. To the best of our knowledge, we are the first to revisit the topic of writer recognition with a focus on the cultural heritage domain.

2. State-of-the-art approaches for writer recognition

In recent years, due to development of image analysis algorithms as well as superb quality of the digital images, we are observing the growth of new applications within the domain of decision-making systems for writer identification. There is an ongoing research attempt investigating if deep learning approaches suffice as general methodology in designing machine learning systems considering the high volume of data available in the mass digitization era in addition to experts’ time and cost being the most limiting factors for classical machine learning approaches [ 2 ]. At the last Document Analysis and Recognition conference – ICDAR 2021, leading topics of accepted papers were text and symbol recognition, handwriting recognition, and historical document analysis.

2.1.

Writer identification in different types of documents

Recurrent neural networks (RNNs) have proven to be better suited for online writer identification in a developed end-to-end model considering the time dimension of the data [ 3 ]. Accordingly, authors in [ 4 ] present an end-to-end writer identification system based on a global context residual recurrent neural network (GR-RNN) and show that such a system provides a better performance than the state-of-theart based even on limited samples of handwritten data. The method is based on the extraction of information using the global average pooling, while RNNs are used to model the relationship between the sequence of local and fragment-based features. The evaluation of the proposed approach is performed on IAM, CVL, Firemaker, and CERUG-EN datasets, while the best performance was obtained on the Firemaker dataset. The authors concluded that the developed method could extract the detailed information regarding the writing style. However, to apply it on other documents such as historical documents, it needs additional preprocessing steps [ 4 ].

A two-level system of ensembles called Funneling Ensemble Method for Writer Identification (FEM-WI) was developed in [ 5 ]. The proposed method consists of multiple feature dependent base classifiers at the first level, and a meta-classifier at the second level. In addition, the authors proposed four novel feature descriptors. The proposed method was evaluated on IAM and Firemaker datasets and obtained an identification rate of above 90%. An ensemble deep transfer learning model for Arabic (Indian) handwritten digit recognition was proposed in [ 6 ]. However, to the very best of our knowledge, using an ensemble of deep learning transfer models for a writer identification task has not been explored so far.

CNN AlexNet architecture with transfer deep learning from ImageNet was used in [ 7 ] for feature extraction (Fig. 1). It was performed from the text-line images representing handwriting text in English and Arabic languages that were altered producing eight input patches namely, original, contoured, sharped, and sharped contours in addition to their negatives. Hence, deep features were extracted on small image patches in size of 227x227. Prior to data-augmentation, standard preprocessing techniques were performed, such as skew detection and correction, normalization, segmentation, and the sliding window strategy for patches. The classification was performed using a support vector machine (SVM). The used dataset is the QUWI dataset, which consists of 1017 writers with four digitized pages and approximately 60 words written by each writer. The authors extracted features from several freeze layers, in particular Conv3, Conv4, Conv5, Fc6, Fc7, and a combination of Fc6 and Fc7 of AlexNet. The results of the experiments suggested that the highest accuracy was obtained using the freeze Conv5 layer - 92.78% for English, 92.2% for Arabic, and 88.11% for a combination of both languages [ 7 ].

Authors in [ 8 ] proposed using multi-task learning in which word recognition methods and writer identification methods are combined. Writer identification based on a single word image is examined. This is achieved by viewing both implicit and explicit features together, to prevent loss of information as well as to create a model not lacking generalization. Authors model the writer’s general writing style from the set of single handwritten word images. The CNN used for multi-task learning in this study is an adaptation of the AlexNet architecture with two pathways for transferring specific features from the secondary to primary (writer identification) task in an end-to-end system for the purpose of achieving better performance metrics for writer identification. Additionally, they evaluate three secondary tasks important for writer identification: (i) word recognition, (ii) word-length estimation, and (iii) characterattribute recognition, as well as their combination. The methods were evaluated on CVL and IAM datasets. The performances of writer identification (Top1 and Top5) are presented, and the authors concluded that adaptive learning can improve the performance of writer identification while deep adaptive learning, by capturing complex relationships, can furthermore improve the performance of writer identification.

Authors in [ 9 ] use a deep learning approach based on multi-task learning, similar to their previous work [ 8 ], and employ a CNN architecture with two branches: feature pyramid and fragment branches. Feature pyramid branch is used to extract feature maps, while a fragment branch is trained for writer identification based on fragments extracted from the word image as well as from the feature maps.

In [ 10 ] the authors proposed a deep learning-based framework for offline text-independent writer identification. The proposed method includes the ResNet architecture and a new descriptor which analyses the handwriting thickness. The framework is evaluated on IAM, Firemaker, CVL, and CERUG-EN datasets and obtained accuracies of 97.50%, 99.61%, 96.16% and 88.95%, respectively, thus proving the suitability of the proposed framework for handwritten character recognition.

Writer identification was performed using a CNN in [ 11 ], where the feature vector was generated by cutting off the classification layer and using the output of the second last fully connected layer as a feature vector. In [ 12 ], the authors proposed DeepWriter – a deep multi-stream CNN for textindependent writer identification. The method is based on local handwritten patches which are used in pairs as input. The training data was augmented to improve the performance of the proposed method, and the obtained accuracies demonstrated good applicability on both Chinese and English characters. 2.2.

Writer identification in historical documents

A comprehensive experimental study comparing deep learning and classical machine learning methods for writer identification in historical documents was performed in [ 13 ]. The aim of this research was to prove that deep learning approaches can be used as general methodology in designing end-toend machine learning systems that could extract the information useful to identify different writers using only images of text belonging to the ancient manuscripts. The results of the study show that deep learning approaches in comparison to classical machine learning models provide at least comparable if not better results.

Authors in [ 14 ] employed transfer learning to perform medieval manuscript writers’ identification using a dataset of digital images obtained from the ‘Avila Bible’. Moreover, the developed system is considered end-to-end since it provides a writer classification (output) based on a single page image (input). Writer recognition was performed using well known architectures including MobileNetV2, VGG19, ResNet50, InceptionV3, InceptionResNetV2, and NAS-NetLarge, as these architectures have reached state-of-the-art classification performances at many computer vision applications. Hence, detecting rows of text in a manuscript page is a similar problem to detecting an object in a scene in computer vision. Traditional machine learning models were trained on Avila Bible and Trento Bible datasets with great performance as confirmed by the same authors in [ 15 ].

The sample that the authors used in [ 14 ] consisted of 749 pages written by 8 identified authors. 96 of those pages, 12 per author, were manually labeled and used to train the model. Hence, the model was trained on 12099 rows (each row was one image). The rest of the pages were used to evaluate the performance of the writer identification system. In particular, writer recognition was performed in three consecutive steps: first, automatic row detection of the text line within each page; second, feature extraction needed for a reliable row classification using neural networks, and the last step is majority vote row-decision combiner that connects a writer to each page considered as shown in Fig. 2 [ 14 ]. In this figure, rp is the number of rows detected and classified in a page, and d and a are the output vectors of the row classifier for each row in a page. Row detection was performed using a MobileNetV2 architecture adapted to object detection using a Single shot detector for detection-map generation. Row classification (feature extraction plus classification) was performed using five remaining transfer learning models. The developed end-to-end system obtained a writer identification accuracy of 96.56%.

Authors’ contribution in [16] is twofold: creating a dataset based on manuscripts of ancient Arabic writers and performing classification on that dataset. The dataset included 8638 images from 64 manuscripts written by 52 authors. Four deep learning transfer models were used: MobileNetV1, DenseNet201, ResNet50, and VGG19 and the focus of the paper was on experiments with deep learning models as well as tuning the learning of hyper-parameters aiming for the performance metrics improvements. Authors made assumptions that deep learning transfer models with fine-tuning of hyperparameters would have better performance metrics. In the first experiment, they established the base level for model accuracy. Hence, they evaluated pre-trained models without fine-tuning of the hyperparameters and, as expected, obtained not satisfactory results (partially apart from VGG19 model that obtained a validation accuracy of 87.37% and average F-score of 83.62%). Furthermore, they performed optimization of the hyper-parameters based on three strategies: (i) minimization of the learning rate to improve the learning process, (ii) increasing the number of final dense layers to improve the classification accuracy, and (iii) increasing the number of neurons in the final dense layer to improve the learning process performance metrics.

The authors found that employing the first and third strategies for classification and recognition improves the model accuracies, as the highest accuracy was obtained using a learning rate of 1e-6, and 1024 neurons. At the same time, the second strategy was not found valuable, since increasing the number of dense layers did not improve the performance of the models [16]. Furthermore, the results from this part of the analysis served as an input to fine-tune the pre-trained models. The results obtained an accuracy higher than 95%. In conclusion, DenseNet201 correctly classified 26 authors, followed by the VGG19 (24 authors), ResNet50 (23 authors), and MobileNet (21 authors) [16]. The architecture of the writer identification system proposed in [16] is shown in Fig. 3.

In the HisFragIR20 [17] competition on image retrieval from historical handwritten fragments, most methods use deep learning approaches for writer identification but nevertheless the evaluation results (mean average precision) are still below 35%, therefore leaving a great margin for model improvement. Therefore, the authors in [18] use HisFragIR20 dataset and develop the A-VLAD model with architecture shown in Fig. 4, for identifying writers from fragments of historical documents providing better evaluation results. In this figure, H, W and D are, respectively, height, width and feature dimension, while the final encoded vector of A-VLAD is of size (K x D).

Considering the ICDAR2017 [19], the test dataset for this competition included 3600 document images originating from the 13th to 20th century. The organizers argued that those participants who used transfer learning may face poor results because the deep learning network was initialized using the weights of the pre-trained model. One method submitted to the competition and presented in [19] included a pre-trained ResNet18, but the highest top-1 precision of 76.4% was obtained by a model that is based on oriented Basic Image Features.

By contrast, the authors in [20] used the ICDAR2017 dataset to perform, among other tasks, writer recognition using pre-trained models, but also models from scratch. In particular, the authors used several datasets, namely the Kuzushiji-MNIST, CLaMM (ICDAR2017 Classification of Medieval Handwritings in Latin Scripts), DIVA-HisDB (ICDAR2017 Competition on Layout Analysis for Challenging Medieval Manuscripts), and Historical-WI (ICDAR2017 Historical Writer Identification dataset). It was found that pre-trained models obtain higher performances, specifically the best performance was achieved by the DenseNet121, followed by InceptionV3, VGG19, and ResNet152 [20].

Historical-WI dataset was also used to evaluate the performance of the method proposed in [21]. Here, the authors employed surrogate classes to train a deep residual network and showed that the proposed unsupervised feature learning technique outperformed the current methods available in the literature, especially because it does not require training labels.

A deep learning-based model for automatic writer identification from historical documents was proposed in [22]. The model uses U-Net for binarization, extracts the features using the ResNet50, and obtains global descriptors using an optimized learnable residual encoding layer. The results showed that fine-tuning the U-Net does not improve the performance, however a combination of binarization, feature extraction and weighted average by means of deep generalized max pooling (DGMP) aggregation performs the best.

Authors in [23] developed a novel method for writer identification from historical handwritten documents that is based on a single feature extraction method. In particular, the authors extracted patches from SIFT descriptors, applied the principal component analysis (PCA) to reduce the dimensionality of the descriptors, used mini batch K-means clustering to group the descriptors, trained a CNN to map image patches to their labels, performed encoding using multi-VLAD and, finally, applied exemplar SVM in order to compare the results. On the ICDAR2019 dataset, they obtained an accuracy of 97% without any preprocessing technique.

Finally, authors in [ 24 ] employed an ensemble of CNN models for writer identification and retrieval from historical documents. The ensemble model was built by a combination of InceptionResNetV2 pretrained architectures. The image set was composed of 170 document images grouped in 34 classes representing the writers, extracted from the ICDAR2019 benchmark dataset. The results proved the efficacy of the ensemble model in overcoming single pre-trained models, with an accuracy of 96%.

3. Discussion

State-of-the-art methods aimed at writer identification include deep learning, transfer learning and end-to-end modeling. Most papers are focused on deep learning approaches, particularly CNNs, but many papers also employ transfer learning, as these architectures are already trained, and perform well even on small datasets. From the selected papers, we can see that writer recognition is mostly performed on words and whole pages, as opposed to patches which are used in only five papers. In ten out of twenty-one papers, end-to-end modeling was performed, while deep learning methods are used in twenty-one papers demonstrating the importance of such methods for machine vision. The use of pretrained architectures was found in ten papers (Table 1). [ 15 ] Avila Bible

Trento Bible [16] Collected by the

authors [20] Kuzushiji-MNIST

CLaMM DIVA-HisDB

Historical-WI [21] Historical-WI [22] Historical-WI 12 1.

4. Conclusion

This paper has provided a review of relevant state-of-the-art methods for writer recognition and classification in the domain of cultural heritage. The current literature proposes the application of deep learning and transfer learning methods for such a task, and the results have been promising with high accuracy even on datasets with a high number of authors. The most frequently used datasets for writer recognition in historical documents include the Avila Bible and Historical-WI. The performance of writer recognition strongly depends on the applied methodology, where deep learning-based methods proved to be the most accurate with classification accuracies between 90% and 99%. However, from the reviewed literature, none of the methods obtained a 100% accuracy, which implies that there is still the need for future work in this direction.

5. Acknowledgements

This work was partially supported by the Ministry of Education, Science and Technological Development of the Republic of Serbia through the Mathematical Institute SANU.

6. References

References

[1] M. O. Tamrin , M. El-Amine Ech-Cherif , M. Cheriet , A two-stage unsupervised deep learning framework for degradation removal in ancient documents , in: Proceedings of the 25th International Conference on Pattern Recognition, ICPR '20 , Springer, Cham, 2021 , pp. 292 - 303 .

[2] A. Rehman , S. Naz , M. I. Razzak , Writer Identification Using Machine Learning Approaches: A Comprehensive Review , Multimedia Tools and Applications 78 ( 8 ) ( 2019 ) 10889 - 10931 .

[3] X. Y. Zhang , G. S. Xie , C. L. Liu , Y. Bengio: End-to-end online writer identification with recurrent neural network , IEEE transactions on human-machine systems 47(2) ( 2016 ) 285 - 292 .

[4] S. He , L. Schomaker, GR-RNN: Global-context residual recurrent neural networks for writer identification , Pattern Recognition 117 ( 2021 ).

[5] E. O. Omayio , I. Sreedevi , J. Panda , Funneling Ensemble Method for Writer Identification (Fem- Wi ) , 2022 , Available at SSRN 4022914.

[6] R. S. Alkhawaldeh , M. Alawida , N. F. F. Alshdaifat , W. Z. A. Alma 'aitah, A. Almasri, Ensemble deep transfer learning model for Arabic (Indian) handwritten digit recognition , Neural Computing and Applications ( 2021 ) 1 - 15 .

[7] A. Rehman , S. Naz , M. I. Razzak , I. A. Hameed , Automatic visual features for writer identification: a deep learning approach , IEEE access 7 ( 2019 ) 17149 - 17157 .

[8] S. He , L. Schomaker , Deep adaptive learning for writer identification based on single handwritten word images , Pattern Recognition 88 ( 2019 ) 64 - 74 .

[9] S. He , L. Schomaker , Fragnet: Writer identification using deep fragment networks , IEEE Transactions on Information Forensics and Security 15 ( 2020 ) 3013 - 3022 .

[10] M. Javidi , M. Jampour , A deep learning framework for text-independent writer identification , Engineering Applications of Artificial Intelligence 95 ( 2020 ) 103912 .

[11] S. Fiel , R. Sablatnig , Writer identification and retrieval using a convolutional neural network , in: Proceedings of 16th International Conference on Computer Analysis of Images and Patterns , CAIP '15 , Springer, Cham, 2015 , pp. 26 - 37 .

[12] L. Xing , Y. Qiao , Deepwriter: A multi-stream deep CNN for text-independent writer identification , in: Proceedings of 15th International Conference on Frontiers in Handwriting Recognition, ICFHR'16 , IEEE, 2016 , pp. 584 - 589 .

[13] N. D. Cilia , C. De Stefano , F. Fontanella , C. Marrocco , M. Molinara , A. S. Di Freca , An Experimental Comparison between Deep Learning and Classical Machine Learning Approaches for Writer Identification in Medieval Documents , Journal of Imaging 6 ( 9 ) ( 2020 ).

[14] N. D. Cilia , C. De Stefano , F. Fontanella , C. Marrocco , M. Molinara , A. S. Di Freca , An end-to-end deep learning system for medieval writer identification , Pattern Recognition Letters 129 ( 2020 ) 137 - 143 .

[15] N. D. Cilia , C. De Stefano , F. Fontanella , M. Molinara , A. S. Di Freca , What is the minimum training data size to reliably identify writers in medieval manuscripts? , Pattern Recognition Letters 129 ( 2020 ) 198 - 204 .

[24] R. Janković Babić , A. Amelio , I. R. Draganov , Writer Identification From Historical Documents Using Ensemble Deep Learning Transfer Models , in: Proceedings of 21st International Symposium INFOTEH-JAHORINA, INFOTEH'22 , IEEE, 2022 , pp. 1 - 5 .