In this working note, we introduce our participation at the ImageCLEF 2016 Handwritten Document Retrieval Task. We mainly focused on hyphenation detection using line images and information retrieval using n-best results. The hyphenation detection step utilizes extracted image features from beginning and end of a line and a binary classi er to determine if a line contains hyphenation. Then the spell correction step is used to eliminate spelling errors from the concatenation of a broken word from the end of a line and the beginning of the next line. The nal text retrieval step employs a su x stripping algorithm to normalize the word tense and form and TF-IDF scheme to rank the retrieved relevant segment results of our submission.
For the ImageCLEF 2016 Handwritten Scanned Document Retrieval Task [
Ground Truth: 6. The evidence of the engagement, consigned to a portable 1-best: 6 . The evidence of the engagement consigned to a /*missing*/ Ground Truth: instrument, instead of a fixed Book. Taken from Exche
1-best: instrument , instead of a by Each Silver from The Ground Truth: quer Bills,- Differs from Stock Annuities- agrees with
1-best: our Bills differs from Stock Annuities degrees
with=the features of query images with the indexed features of existing images in the
dataset or transcribe the document before retrieving the query string. Language
models like n-gram and HMM [
Although the handwritten document recognition has attracted much attention in the areas of image processing and machine learning, the impact of hyphenated words is not well studied. Hyphenated words, also known as broken words, are the words supposed to appear at the end of a text line while broken and continued at the beginning of the next line because of the manuscript writer's intention to save space. Hyphens will be used to mark such broken words at the end of line by \-" or \=" and at beginning of a line by \:". Hyphenations may cause recognition errors if they are not processed properly. Example line images, recognition errors in 1-best transcripts and the corresponding ground truth from the task dataset are shown in Fig. 1. We are interested in this working note to describe a solution to detect words with hyphenation in segmented line images, which can further be used to improve the retrieval result of given strings of queries.
The rest of this working note is organized as follows. First, we introduce our proposed methods in Section 2, including steps of image preprocessing of line segmented image, hyphenation detection, spell correction using training transcript and text retrieval. Second, the evaluation of the e ectiveness of hyphenation detection methods and the o cial task evaluation of our submitted run are discussed in Section 3. Finally, we conclude our work and proposed some future works and improvements. 2
In our proposed solution to ImageCLEF 2016 Handwritten Document Retrieval Task, we mainly focused on how to detect the hyphenated words, how to do spell correction for detected hyphenated words and how to retrieve the relevant segments based on 1-best results. In this section, we will elaborate the details of each step. We rst describe the preprocessing step of how the segmented grayscale line images is normalized into xed-height binary images (Section 2.1). Then using normalized line images, a hyphenation detection methods is proposed (Section 2.2) to detect lines with hyphenation, followed by spell correction (Section 2.3) and text retrieval step (Section 2.4). 2.1
Preprocessing In this step, our goal for the preprocessing step is to obtain noise and slant free binary line images. In some related works, skew correction may also be necessary before the recognition or word spotting step. However, in the task dataset, lines are well segmented and with only negligible slope, which ease us from skew correction. The slant of written lines are removed by applying a two dimensional a ne transformation to the original line images. The transformation matrix is arbitrarily chosen based on the observation from random selected line images in the training set. Afterwards, a global threshold is applied to the slant corrected grayscale line images to generate the binary line images. We also resize the line images to a xed height of 30, and the width is scaled proportionally. 2.2
Hyphenation Detection In order to detect lines with beginning and ending hyphenations, image windows at both the beginning and end of each line image are obtained. Several image features are then extracted from the image windows, and various binary classi ers are used to detect hyphenations. As a binary classi cation problem, according to the writing style of the document writer, lines containing both the end hyphens and beginning hyphens in the next line are considered lines with hyphenation. Lines with such hyphenations are labeled as positive, while the others are labeled as negative. This strict labeling rule is helpful to eliminate false positives in the prediction results.
Several binary classi cation methods are used on the image features in the training set. To evaluate the performance of these methods, precision, recall and F-score are tested as metrics in the development set. In this task, the ground truth transcripts of both training and development set are provided, thus the ground truth labels can be obtained and used to compare with the hyphenation detection results.
To represent the binary line image windows as feature vectors, a set of local
features are extracted from the beginning windows and ending windows for
each line. These features have been used in previous works on keyword spotting
approaches [
The classi ers investigated are implemented in Scikit-Learn [
Spell Correction Once the hyphenation of a certain line is detected, the last word of the current line and the rst word of the next line are concatenated. The text from the two word windows are extracted from the 1-best result, and all the special characters are removed before concatenation.
From the 1-best documents in both the training and development dataset, the accuracy of recognition without hyphen is relatively high. However, for hyphenated words, word spotting algorithms tend to predict the given word image as the most similar complete word, instead of considering it as only the front part or the back part of a word. For example, the word \testimony", if broken as \testi-mony" into the end and beginning of two lines, the latter part is more likely to be predicted as "many" which is a complete word, instead of "mony" which is a su x in ground truth. Fig. 1 also shows some of such examples.
To correct the recognition errors caused by hyphenation, a spell correction step is applied after the concatenation of two broken words. The ground truth QUERY
TRANSCRIPTS
PROCE SSING
DOCUMENTS INDEXING
SEGMENTS RETRIEVAL MODEL
RETRIEVAL REQUEST INDEX RANKING SYSTEM
RESULT
transcript from the training set is used to generate a dictionary of the dataset. All
the words in training and development set are converted to lower case before put
into the dictionary. A spell corrector is then used. The spell corrector considers
the corrections of edit distance no larger than 2 from the original word. The
corrections with maximum likelihood to the current prediction in the dictionary
is chosen. If no matched correction under these criteria is found, the original
word is remained unchanged. The implementation of the spell corrector can be
found in [
Y; (Format 1) where L is the line number, T represents the word, W and H are the width and height of the bounding box, and X and Y are the left-top coordinate of the box, respectively.
We used the software package Elasticsearch1 to index the segments. A total of 16939 segments were indexed into three elds: \ID", \contents" and \annotations". In the \ID" eld, the ID of the rst line in each segment was used to distinguish the segments. In the \contents" eld, we only indexed the words, i.e., W in format 1, in the segments. We applied the following hyphen-rule to take the hyphenated words into consideration:
If line i ends with any of =, -, and :, and line i + 1 starts with either - or :,
then the last term in line i and the rst term in line i + 1 are
considered a hyphenated word.
(Rule 1)
The broken words are concatenated together after removing the hyphens. The
spell corrector described in previous section is then used. Before indexing, we
also applied the Porter stemmer [
In the retrieval component, we utilize the Term Frequency - Inverse
Document Frequency (TF-IDF) scheme [
In this section, we elaborate the details of experiments and the performance of hyphenation detection and ImageCLEF 2016 Handwritten Scanned Document Retrieval Task evaluation.
The task dataset is a subset of scanned and manually transcribed manuscripts
written by Jeremy Bentham under the Transcribe Bentham project [
For the hyphenation detection step, the ground truth are extracted from transcripts of training and development set. As expected, the dataset is 1 https://www.elastic.co/products/elasticsearch signi cantly unbalanced. In the training set, there are only 810 positive samples in 9645 lines, the proportion of positive samples is 8.4%. In development set, there are only 853 positive samples in 10589 total samples. The percentage of positive samples is 8:0%. The trained models from training set are used for the classi cation of the development set. The precision, recall and F-score metrics of the development set is shown in Table 1. From the experiment results we can observe that AdaBoost is the best performed classi er for hyphenation detection in the proposed feature set. The detection algorithm does not perform well in development set, thus we do not include it into the nal submission. The spell corrector is still used in text retrieval step to handle the hyphens already in the 1-best results.
We submitted one run as the task submission, and the o cial evaluation results are shown in Table 2. We noticed there is a signi cant drop from the results of the development set to those of the test set. The similar performance decreases can be also found in the results of the baseline system, which uses exact string matching and should be robust among di erent datasets if these datasets are of similar characteristics and quality. The reason for this signi cant performance di erence is because the test set is considerably more di cult than the development set, where the bounding box is much less accurate and the images quality is lower.
We discussed our participation of ImageCLEF 2016 Handwritten Scanned Document Retrieval Task. Our submission run is based on a three step process, hyphenation detection, spell correction and information retrieval. The hyphenation detection step utilizes extracted image features from beginning and end of lines and a binary classi ers to determine if a line contains hyphenation or not. Then the spell correction step is used to eliminate spelling errors the concatenation of a broken word from a line ending and the line beginning of the next line. The spell correction step uses only the vocabulary from the transcript of training set. A nal information retrieval step employs a su x stripping algorithm to normalize the word tense and form and TF-IDF scheme to rank the retrieved 30 segments as the output of our system.
There are several future works that can be investigated to improve the performance of our system. First, more line image features can be considered as the input of supervised binary classi cation methods. Second, larger lexicon can be utilized in both the hyphenation detection step and spell correction step. Due to the task restriction, only the vocabulary in the training set can be used, and the use of external data for learning a language model is prohibited. A larger lexicon of the whole dataset or external data rather than only the training set will improve the e ectiveness of the spell corrector. 5
The authors gratefully acknowledge the support from the National Library of Medicine (NLM) grant R01LM11934.