-Traditional forensic document analysis methods have focused on feature-classification paradigm where a machine learning based classifier is used to learn discrimination among multiple writers. However, usage of such techniques is restricted to availability of a large labeled dataset which is not always feasible. In this paper, we propose a Cotraining based approach that overcomes this limitation by exploiting independence between multiple views (features) of data. Two learners are initially trained on different views of a smaller labeled training data and their initial hypothesis is used to predict labels on larger unlabeled dataset. Confident predictions from each learner are used to add such data points back to the training data with predicted label as the ground truth label, thereby effectively increasing the size of labeled dataset and improving the overall classification performance. We conduct experiments on publicly available IAM dataset and illustrate the efficacy of proposed approach.
Keywords-Writer Identification, Co-training, Classifier, Views, Labeled and Unlabeled data
Writer Identification is a well studied problem in forensic document analysis where the goal is to correctly label the writer of an unknown handwriting sample. Existing research in this area has sought to address this problem using Machine Learning techniques, where a large labeled dataset is used to learn a model (supervised learning) that efficiently discriminates between various different writer classes. The key advantage of such learning approaches is their ability to generalize well over unknown test data distributions. However, such generalization provides greater performance only when used with a large labeled data. In real-world scenarios, generating large labeled datasets requires manual annotation which is not always practical. The absence of such datasets also leads to inefficient usage of available unlabeled data that can be exploited to provide a greater classification performance. To address these issues, we propose a Co-training based learning framework that learns multiple classifiers on different views (features) of smaller labeled data and uses them to predict labels for unlabeled dataset which are further bootstrapped to the labeled data for enhancing the prediction performance.
Existing literature on writer identification can be broadly
classified into two categories. First category is of text
dependent features which capture properties of writer
based on the text written. In this writer identification is
done by modeling similar content written by different
writers. This reliance on text dependent features poses
challenges of scalability. In real world application such
data is seldom available which limits the usability of these
techniques for practical purposes. said et al. [
Likewise, writer identification can also be divided
under two major approaches. First is statistical analysis
of several features such as edge hinge distribution. Edge
hinge distribution captures the change in the direction of
writing samples. Second approach is model based writer
identification. In this predefined models of strokes of
handwriting are used. Prime focus of these techniques was
on making a better system for identification using different
techniques for modeling and analysis. Various techniques
such as Latent Dirichlet Allocation(LDA) were proposed
for higher accuracy for identification[
Existing techniques and methods did not make use of
unlabeled data for the identification. Information tapped in
the unlabeled data can make a significant improvement in
the performance of the system. This information can be
extracted using different techniques such as transductive
SVMs[
The organization of the paper is as follows. Section 2 provides an overview of Co-training based framework. Multiple data views in form of writer features are described in Section 3. Section 4 illustrates the proposed approach. Experimental results are described in Section 5. Section 6 outlines the conclusion.
Co-training is a semi supervised learning algorithm
which needs small amount of training data to start. It
reiteratively labels some unlabeled data points and again
learns from it. blum et al. [
Co-training is an iterative bootstrapping method
which increases the confidence of the learner in each
round. It boosts the confidence of score like Expectation
Maximization method but it works better than EM[
Selection of data points is crucial in the performance
of the algorithm. New points added in each round should
make learner more confident in making decisions about
the labels. Hence, several selection algorithms have been
tried to make a better system as system’s performance can
vary if selection method is changed. Different methods
out performs each other depending on the kind of data
and application. One approach to select points was based
on performance[
Some other methods has also been employed like
choosing the top k elements from the newly labeled cache.
This is an intuitive approach as those points were labeled
with the highest confidence by the learner. However, hwa
et al. [
Selection of uncorrelated views is important in the
working of Co-training. blum et al. [
Co-training fits naturally for the task of writer identification as any piece of writing can have different views. Contour angle features and structural and concavity features are two such different views for any handwritten text. They can be considered uncorrelated enough to fit the task of writer identification in Co-training framework. Co-training also needs to have two learners to learn over two views. We used two different instances of Random Forest as learners to normalize the effect of learner over views.
Angle features were used to train first classifier and SC were used to train the other one. Then in each round a cache will be extracted from unlabeled data. This cache would be labeled by both learners and some data points will be picked from newly labeled cache by selection algorithm. Selected data points will be added to the training set and the learners are retrained while remaining data points in the cache are discarded. This process is repeated unless the unlabeled set is empty. Below is the pseudo code for the Co-training algorithm.
Algorithm 1 Co trainingAlgo Require:
L1 Labeled View One L2 Labeled View Two U Unlabeled Data H1 First Classifier H2 Second Classifier Train H1 with L1 Train H2 with L2 repeat
Extract cache C from U U U C Label C using H1 and H2 d selection algo(C) where d add labels(d,H1,H2) L1 L1 [ view one of d L2 L2 [ view two of d Retrain H1 on L1
Retrain H2 on L2 until U is empty C
Selection algorithm used for selecting data points was
based on agreement of both learners over data points. Points
on which the confidence of both learners was above certain
threshold were selected. In case of documents accuracy
of classifier would be high if two different views will
indicate same label for any data point. Selection method
based on randomly selecting data points and checking their
performance as used in [
We used IAM dataset which has total of 4075 line images written by 93 unique writers. We conducted two experiments to test the performance of Co-training against the baseline systems. In first we compared the accuracy of classifiers after Co-training against baseline methods by adding the scores of both learners. In this scores of the class distribution of the two learners were added for each data point and a joint class distribution score was generated. Class label with the highest score was assigned to that data point. Second experiment was based on the maximum of the confidence score of the label assigned by each learner. In this each classifier assigns a class label to the data point. This assignment is based on the highest value of the confidence score distribution over all classes. Class label with the higher score between the two is assigned to the data point.
Our goal is to show that Co-training can be used to label unlabeled data even if a small amount of labeled data is present in the beginning. Therefore experiments were run on dataset of different sizes. We conducted experiments with four different settings of data. System was initially trained over full, half, one fourth and one tenth of the total training data. In one tenth training data only three samples per class were present. Table shows that after Co-training accuracy of classifiers is better than the baseline system with all sizes of datasets in both experimental settings.
In this paper we presented a Co-training based framework for labeling a large dataset of unlabeled document with the correct writer identities. Previous work in writer identification was focused on either on developing a better feature selection algorithm or to use different techniques for modeling the text of the document. All the work was based on a assumption that sufficient amount of labeled data is available for training a system. In our work we address the problem of limited amount of labeled data present in real life applications. Our method tries to iteratively generate more labeled data from unlabeled data. Experimental studies show that accuracy of learners on the dataset labeled by Cotraining was better than the baseline system. This proves the effectiveness of Co-training for labeling a large dataset of unlabeled documents. In future we would like to address this problem of limited data by using other semi supervised learning methods.