In this paper, we propose a texture based technique to detect text in grey level natural scene images. This work is a part of the project called Intelligent Glasses. It is a wearable system to facilitate navigation and to assist the blind and visually impaired persons in real world. It has three parts, a bank of stereovision, a processing unit for visual perception and a handheld tactile surface. In its original form, it will be able to provide information about different types of obstacles and their position with respect to user. Our textual/symbolic information interpretation module to the vision system of the Intelligent Glasses will recognize the text from the captured scene and textual and/or symbolic information will be displayed on the handheld tactile. Initial results are encouraging with a text detection rate of 64%.
In this article, we have proposed a texture based technique to detect text in grey level natural scene
images. This work is a part of the project called “Intelligent Glasses” (
(a) (b) (c) (d) Figure 1 Intelligent Glasses : (a) concept (b) scene (c) environment perception (d) representation One of the salient feature of this tactile surface of the system is its compact representation of the environment and space. Using the visual (acquired by stereovision system) and inertial information, it changes with the time and with the scene observation point. With the help of these two dimensions, this interface can represent a simplified 2D version of 3D world. It is also possible to estimate the relations between real world and its tactile representation such as the size of an object, an obstacle distance with respect to user etc. Moreover, it can represent different geometrical shapes (square, rectangle, circle, arcs, curves….) making it feasible to represent contours, edges and arcs etc which are generally present in a scene.
However, all these information are not sufficient to give enough autonomy to a blind/visually impaired person. As an important form of human beings language, visual texts are widely used in our daily life. For example, different sign boards, directions, shop names etc contain textual and/or symbolic information that is perceived by a human being to facilitate his knowledge of environment and perhaps also help in his navigation. The need to interpret textual and/or symbolic information becomes evident in the case of blind or visually impaired persons. With this perspective, we want to add textual/symbolic information interpretation module to the vision system of the Intelligent Glasses that will recognize the text characters from the captured scene text and finally, textual and/or symbolic information will be displayed on the handheld tactile. However, before recognition, we have to know about the text existence (text detection) and text location in an image (text localisation). As the tactile surface is reprogrammble, it is possible to adapt it for the representation of textual and symbolic information persent in the scene.
This paper addresses the problem of scene text detection in grey-level images and presents our
preliminary work in this domain. Our algorithm generates potential/candidate text regions that can later
be verified by a validation/verification scheme. We follow a general framework of candidate regions
generation and validation as employed by various researchers. Our detection method is based on six
texture features computed from grey-level co-occurrence matrix (GLCM), proposed by Haralick
The rest of the paper is organized as follows. In section 2, we persent some of the previous work in this domain. We describe the proposed texture coding and classification technique in section 3. Six features namely contrast, homogeneity, dissimilarity, entropy, energy and correlation from grey-level co-occurrence matrix (GLCM) are computed over a small region of image. Hence capturing local textural details present in the image, we can build a system which can differentiate text and non-text regions. Image database, experimental setup and results are included in section 4. Finally section 5 concludes this paper.
Most of the existing systems use voice synthesizer to help a blind or visually impaired person. The
system proposed by Ezaki et al.
In recent years, digital cameras/ camcoders and PDA are increasingly popular and they have shown
potential as alternative imaging devices. The researchers working in document analysis and
recognition have changed their orientation and instead of working with traditional scanner captured
document images, they are concentrating on analysis of images taken from camera. This change
accompanies with lot more challenges such as low resolution, uneven lightining, perspective
distortion, nonplanar surfaces, wide angle lens distoration, complex background
We have proposed a simple texture coding method to detect scene text in grey-level natural scene
images. We have used spatial histograms computed from grey-level co-occurrence matrix (GLCM) for
texture coding. Grey level co-occurrence matrix M(x, y)(d, θ) or second order histogram (which consider
the relationship between groups of two pixels in the original image) was initially defined by Haralick
Generally, GLCM is computed over a small square window of size N centered at a certain pixel (x, y) and then window is moved by one pixel in the same manner like convolution kernel. Fine texture description requires small values of d and/or small window size, whereas coarse texture requires large values of d and/or large window size. An average over all orientations is taken so that these matrices are rotation invariant. Figure 2 (a) shows an example of construction of grey-level co-occurrence matrix for d = 1 and θ = {0°, 180°} and {90°, 270°}. The matrix M(0, 0) (1, 180°) is just the transpose of M(0, 0) (1, 0°). So to cover all orientations (8 in this case), we need only to compute first four orientations.
It is known that feature based algorithms are generally more stable than raw data (grey levels) based algorithms so a number of features can be calculated using the co-occurrence matrix (containing G2 elements) for texture discrimination. Haralick defined 14 such features. We have used six out of these fourteen features in our work. These features are: contrast, homogeneity, dissimilarity, energy, entropy and correlation and are listed in figure 2(b). As we can see, to calculate different features, the joint probability density of grey level co-occurrence computed by GLCM is weighted differently. The first three features (contrast, dissimilarity, homogeneity) can be grouped and can be named as “Contrast Group” .They compute quantity of contrast in a window. The second group called “Orderliness Group” contains features (energy and entropy) which indicate how regular (orderly) the pixel values are within the window. Correlation measures the linear dependency of grey levels on those of neighboring pixels.
To classify pixels as text or non-text, we tested various generative and discriminative classifiers. The generative classifiers are parametric, based on the assumption that features joint probability distribution for a certain class is multi-variate gaussian. In our experiments, we used one or more (two in our case) gaussians for each class. Given two classes – text and non-text and number of gaussians, the classifiers are: mono gaussian for text and non-text class (TNTSG), mono gaussian for text class (TSG), multi gaussian for text and non-text class (TNTMG), multi gaussian for text class (TMG). In multi gaussian models, a gaussian mixture model containing two gaussians, is trained using Expectation-Maximization (EM) algorithm. Gaussian parameters i.e. mean and covariance matrix for each class are estimated on a training database by using maximum likelihood estimator. In test, Mahalanobis distance is used to compute likelihood estimate.
The discriminative classifier (NNC) is a two-layer perceptron with 6 inputs, NH hidden cells and 2 outputs. The number of hidden cells (NH) used in our analysis are 20, 40 and 60, resulting in three discriminative classifiers. Texture features values from training database are normalized between –1 and 1 before feeding to neural classifier and the desired outputs are example labels: 1 for text, -1 for non-text. The network is trained for 10000 epochs with cross-validation stopping. One fourth of the training database is used in cross-validation.
We have used ICDAR 2003 robust reading and text locating database
We compute GLCMs over a small square window of size N centered at a certain pixel (x, y) and then window is moved by one pixel in convolution kernel manner. GLCMs are computed in 8 directions (E, NE, N, NW, W, SW, S, SE) or (d = 1, θ = 0°, 45°, 90°, 135°, 180°, 225°, 270°, 315°) and an average is taken so that these features are rotation invariant. In actual implementation only four orientation matrices are needed to be calculated and the other four orientations can be computed using transpose of these matrices. Moreover, five different square windows with size N = 5x5, 7x7, 9x9, 11x11, 13x13, 17x17 are used.
Due to intensive nature of computations, reduction of number of intensity levels (quantizing the image to a few levels of intensity) helps increase the speed of computation with negligible loss of textural information. The grey-levels are uniformly quantized to G levels between minimum and maximum grey levels of a window. We choose 32 grey-levels in our implementation. Once GLCMs are computed, texture features can be calcluted by the equations in figure 2(b). Average feature calculation time for an image of 640x480 pixels using un-optimized C++ code running on Pentium IV 3.2GHz for various windows is given in table 1.
43 7x7 51 9x9 67
To evaluate the performance of a certain text detector, we adopt a pixel based evaluation mechanism. The target images (binary in nature) in ICDAR 2003 database contains one or more black (pixel values = 0) rectangular areas representing text regions. The pixel value for non-text pixels is 1. The database is designed for text localisation. However, in our scheme, due to absence of localisation step which generates rectangles around text strings, we have to evaluate performance of text detector with the given target images where text regions are represented by rectangular regions and figure-ground data is missing. Hence, such an estimate is biased and the actual detection rate is higher than the calculated. The text detector generates either 0 (for text) or 1 (for non text) for each pixel of the input image. In pixel based evaluation mechanism, the output of text detector is compared with the target and a confusion matrix is created. For evaluation, two quantities, text detection rate and false alarm rate are computed.
In this paper, we have employed a texture features computed from grey-level co-occurrence matrices text detection in natural scene images. Our text detectors work on a wide range of text font sizes and fonts (see figure 3). Although, the performance is evaluated on a small test database of 50 images but the results are encouraging and we hope that performance evaluation of these text detectors on a larger database will validate these results and conclusions. Different classifiers have been employed and we have found that mono gaussian models are more robust than multi-gaussian models and two class (text and non-text class) model is better than single class (text class) model. The best classifier is multi-layer perceptron which gives a text detection rate of 64% and false alarm rate is 25%. Till now, we have not filtered any detected text region by applying validation methods e.g. geometrical and spatial constraints, baseline detection, character alignment etc. We believe that such validation schemes will lower the false alarm rate. Furthermore, we are exploring gradient methods as they can differentiate text and non-text regions. Gradient methods are rapid in calculation so one such method can be used to generate candidate text regions which can further be processed by our proposed texture scheme, thus making overall process fast.