This report presents the work carried out for the image classi cation task in the course of the CLEF-IP 2011 competition. Based on the visual content, patent images are automatically classi ed into several drawing types, such as abstract drawings, tables, ow chart and graphs. For that purpose, a series of SVM classi ers, multi-modal fusion schemes and a variety of content-based low-level features for black and white images were used. The overall reported performance was promising. Our best runs achieved a true positive rate of over 66% and the reported average area under curve is over 0.9.
There are many di erent types of images in patents, such as technical
drawings, diagrams, photos, ow charts and graphs [
The rest of this paper is organized as follows: Section 2 describes the used content-based features in detail, Section 3 outlines the classi cation process. Results are presented in Section 4. 2
The content-based features described in this section are the basis for the image classi cation. All features were extracted globally for each image in the training and test set.
Local Binary Patterns (LBP) The LBP [
MPEG-7 Edge Histogram (EH) The edge histogram descriptor[
Optical Character Recognition (OCR) The following feature values were obtained by applying an optical character recognition toolkit 4 on the images: font size, number of of text blocks, ratio between area of text blocks and image area image size, number of ' g' occurrences, number of 'tab' occurrences, number of digits and image orientation.
Binary Image Features (BIF) For each of the images a descriptor with total length of 12 was computed. It comprises the image's Euler number, mean, standard deviation, variance, skewness, kurtosis, perimeter, area, number of (4- and 8-) connected components, thinness ratio and density.
The range of all feature vectors in the training set was linearly scaled between 0 and 1. The feature values in the test set were rescaled accordingly. 3
Our approach to patent image classi cation is based on training support vector
machines (SVMs) since they had achieved satisfactory performance in related
tasks over the past few years. The challenge of classifying a patent image into
one of the 9 classes, see Section 1, was regarded as a sest of two-class problems
(i.e. ' ow chart' or 'no ow chart' and 'graph' or 'no graph'), where for each
problem the positive and negative examples were extracted from an existing
annotated training set. The maximum of the scores yields the nal classi cation
for an image. Since for all classes there were more negative examples than
positive examples, the SVM training data for a speci c class was composed of all
its positive annotations with an equal number of negative annotations randomly
selected from the remaining classes. The training data, contains between 310
(for ow charts) and 5983 (for dna) training images per class. For better
comparability we assured that the random selection produced the same annotations
across di erent runs. In total, 36 SVMs (9 classes * 4 feature sets) were produced
using the LIBSVM software package [
The list below describes the speci c con guration of the 8 runs produced. For
runs with a single modality, only one of the previously mentioned content-based
features was chosen. The other 4 runs apply various simple late fusion methods
4 http://www.leadtools.com/sdk/ocr/
on the output of the base classi ers. Late fusion rst reduces unimodal features
to separately learned scores, then these scores are integrated to produce nal
scores for the classes [
In order to get a rst impression of the image classi cation task, exemplary classi cation results from the test data are presented in Figure 5. For each of the 9 classes (from top to bottom row: abstract drawing, chemical, code program, gene sequence/dna, ow chart, graph, maths, table, character/symbol) the 5 images with highest classi cation scores are shown. The images with chemical structures, mathematical formulas and symbols are apparently easy to discriminate. On the contrast, tables and code programs seem visually similar. The last image in the row with gene sequences was incorrectly classi ed. The second image in the row of graphs shows an abstract drawing and was also misclassi ed which is possibly due to its graph like curved structure. The analysis has to deal with images with multiple gures (of the same class in our case), see last image in row with graphs. Another challenge is the variable and unknown orientation of the images. As can be seen in the row with tables and graphs, some of the images are rotated.
The test data consists of 1000 unclassi ed images. For each of the images the type of the image was classi ed. Figure 2 shows the performance per run and class using the Area under Curve (AUC) and True Positive Rate (TPR) measures. From the submitted 8 runs, 5 runs achieved satisfactory performance indicated by an AUC value of 0.9 an above. The best run according to the TPR measure is alphacentauri with an accuracy of 66.3% and AUC of 0.96, slightly outperforming the runs with score fusion (TPR between 62.4% and 65.3%). The good performance is obviously due to the feature LBP. Interestingly, a fusion with other features did not improve the classi cation results on average. As can be seen in Figure 2, the fusion with OCR and EH was only bene cial for the classes ow chart (improvement from 44.7% to 82.8%), abstract drawing (from 54.2% to 75.3%) and table (from 68.7% to 76.5%). The single modalities using OCR, EH, or BIF only did not provide satisfactory results on average. Most or even all of the characters/symbols were correctly classi ed even when applying runs that use features that generally came of badly.
A detailed view on the performance of the run alphacentauri is shown in Figure 3. Very promising results could be achieved for characters/symbols, chemical structures and math formulas (AUC value over 0.95). The image types abstract drawing, ow charts and graphs are di cult cases mainly due to their intra-class variance.
A confusion matrix for run alphacentauri is given in Figure 4. Actuals belong on the side of the confusion matrix and predictions are across the top. For the classi er many image types (all but chemical, maths and characters) seem to be di cult to distinguish from tables. Similarly, graphs were di cult to classify. No or hardly any confusion was attained for the types chemical, maths and characters. 5
This paper presented the experiments for our participation in the CLEF-IP image classi cation challenge. The type of images in patents was automatically classi ed by using SVM classi ers and simple multi-modal fusion schemes. A variety of content-based low-level features for black and white images were used. Generally, training the SVM models and in particular the parameter tuning is a computationally expensive process and must not be neglected. The performance of the classi ers tested on 1000 patent images was promising. Our best runs achieved a true positive rate of over 66% and the reported average area under curve is over 0.9 for 4 of the 8 submitted runs. Some classes have been identi ed better by using only a single input feature and others by late fusion. Consequently, for di erent classes di erent features and fusion methods should be applied.
This work was supported by the Austrian Research Promotion Agency (FFG) FIT-IT project IMPEx 5 Image Mining for Patent EXploration (No. 825846).