The paper shows the use of artificial neural networks for the task of scene text recognition. A comparative analysis of the effectiveness of the use of various neural network architectures was presented. Experimental studies were performed on the MNIST, IIIT 5K, SVT and Synth 90k datasets. Nowadays, a lot of work in the framework of computer technology based on deep learning methods is devoted to the subject of text recognition [1-4]. Irregular text is widely used. However, it is considerably difficult to recognize because of its various shapes and distorted patterns. [5]. Examples of scene text can be photographs of street signs, stills from movies with subtitles, data in robot navigation systems, etc. Among deep learning methods, convolutional neural networks (CNN) are the most studied [6-11]. Often, a popular MNIST dataset, which is a database of handwritten text samples, is used to conduct experimental research in text detection works. In [12], modifications of convolutional neural networks were studied to improve the accuracy of the classification of handwritten numbers. Authors [13] seek to characterize the learning architectures exploited in biological neural networks for training on very few samples, and port these algorithmic structures to a machine learning context. They carried out modeling of the structure of the neural network of smell of moths, then the created computational models are taught to read handwritten numbers. Most methods based on convolutional neural networks extract image features at the last level of the network using a single CNN architecture with an unlimited number of quantization approaches. What limits the use of intermediate layers to identify local features of the image.
calculated, where the summands are the product of the pixel values on the values of the kernel cell that covered the pixel. The sum is divided by the sum of all elements of the convolution kernel. The resulting value is the new value of the selected pixel. By applying a convolution operation to each pixel of the image produces an effect that depends on the selected convolution kernel. In this work, we have constructed CNN for the task of recognizing handwriting in Arabic numerals and the scene text recognition. 2.1
Neural networks of the following architectures have been developed to recognize handwritten digits. 2.1.1
- First fully connected layer, ReLU activation function, 512 neurons; - Second fully connected layer, Softmax activation function, 10 neurons; - Cross-entropy loss function. 2.1.2
- First fully connected layer, ReLU activation function, 1024 neurons; - Second fully connected layer, ReLU activation function, 512 neurons; - Third fully connected layer, Softmax activation function, 10 neurons; - Cross-entropy loss function. 2.1.4
Architecture 3. Convolutional Neural Network with two Dropout Layers
- First convolution layer, ReLU activation function, convolution kernel 5x5, (28, 28, 32);
- Second convolution layer, ReLU activation function, convolution kernel 5x5, (28, 28, 32);
- First max-pooling layer, kernel 2х2, step 2x2, (
Architecture 3. Convolutional neural network with thee Batch Normalization layers
- First convolution layer, ReLU activation function, convolution kernel 5x5, (28, 28, 32);
- Second convolution layer, ReLU activation function, convolution kernel 5x5, (28, 28, 32);
- First Batch Normalization layer;
- First max-pooling layer, kernel 2х2, step 2x2, (
The input gray-scale image of size 28x28 pixels (for neural networks without the convolutional layers have to be reduced to the vector of size 784) were subjected to normalization by aligning interval of the brightness values of the pixels to the range from 0 to 1. This image was sent to the input of the neural network along with the corresponding class label. The class label was converted into a unitary code (one-hot encoding) before being fed to the network input for the correct operation of the Softmax function and the cross-entropy function. Adam was chosen as the optimizer with a training step of 0.001, the batch size, for which a one-time adjustment of the scales is 128. The network weights before training are initialized to normally distributed values with a mathematical expectation of 0 and a standard deviation of 0.1. The bias of the layers is initialized to zero. For extreme pixels in convolutional layers, the values are automatically filled with zeros to obtain the same dimension at the output of the layer as at the input.
To solve the problem of scene text recognition, it is proposed to use a combination of a convolutional neural network, a recurrent neural network (RNN) [16, 17], and Connectionist Temporal Classification (CTC-loss) [3]. Feature extraction using a convolutional neural network is shown in Figure 1.
The output of the convolutional network is a three-dimensional array that must be converted to a two-dimensional array, the second dimension of which must be equal to the maximum number of characters in the input image. After conversion, each column of the matrix is fed to the input of the corresponding LSTM-cell. The recurrent layer is used to represent the input features as a sequence and the network recognizes the characters exactly in the order in which they appear in the search text. Each vector is transferred to a fully connected layer and to the Softmax activation function. The dimension of a fully connected layer is equal to the length of the array of all possible recognizable characters with the addition of a blank marker. This marker CTC-loss algorithm uses to indicate the absence of a character in the text, because most of all input sequences of characters will be less in length than the maximum possible. The softmax function represents the input vector as a probability distribution vector for all possible symbols, including a blank symbol marker. The principle of the recurrent layer is shown in Figure 2.
Thus, the output is a matrix with distributed Softmax function by probability columns, where the number of columns is the maximum possible word length, and the number of rows is the length of the “alphabet” L + 1 (blankmarker). This matrix is input to a decoding algorithm that removes all duplicate characters, and then blank markers. The final alphabet is the set of all possible characters in a word and a blank marker by Eq.2:
L L {blank}.
(
The probability of the path Y is equal to the product of the probabilities of all activated cells of each column of the matrix:
T p(Y ) t1 yYtt , YL'.
Example of calculating the probability of the path “ap-pl-ee”:
p("ap plle)" y1a y2p y3 y4p yl5 y6 ye7 ye8 .
The decoding algorithm is shown in Figure 3. Many paths always correspond to one word, and to calculate the probability of a word, it is necessary to calculate the sum of the probabilities for all possible paths.
The error function is similar to binary cross-entropy, only the word probability is used here as the class probability by Eq 4.
CTCloss ln( p("apple")) .
For the considered example, in which the alphabet consists of 6 characters, and the maximum number of characters is 8, the number of possible paths is 68 = 1679616 possible paths. When solving practical problems, the length of the alphabet and the maximum number of characters are significantly larger. Therefore, dynamic programming is used to calculate the word probability. The word probability value is used in the back propagation algorithm. 2.2.1
For scene text recognition developed a neural network combining convolutional, recurrent layers and CTC error function (CRNN). Details of the CRNN architecture are given below.
- First convolution layer, ReLU activation function, convolution kernel 3x3, (31, 100, 64);
- First max-pooling layer, kernel 2х2, step 2x2, (
As an optimizer, Adam was chosen with a training step of 0.001, the batch size for which there is a one-time
adjustment of weights is 64. The length of the alphabet is 43 characters, the maximum possible length of the word is
16 characters. The network weights before training are initialized to normally distributed values with a mathematical
expectation of 0 and a standard deviation of 0.1. For extreme pixels in convolutional layers, use automatic zero-fill to
get the output layer of the same dimension as the input. The input image was reduced to the size of 31x100 pixels in
grayscale, and then fed to the neural network input along with the corresponding word.
(
Experimental studies were performed on the test set of MNIST and for scene text images separately. For experimental research, a computer program was developed in the Python programming language using the Tensorflow [18] framework.
For experimental studies, the MNIST dataset [19] (Modified National Institute of Standards and Technology) – a database of samples of handwriting numbers was used. Images from this dataset are processed images from another dataset - NIST. NIST samples had a dimension of 20x20 and were taken from the U.S. census Bureau with the addition of test samples written by students of American universities, and then were normalized, smoothed and reduced to the size of 28x28 pixels. MNIST database contains 60000 training and 10000 test gray-scale images of size 28x28 Examples of images are shown in Figure 4.
According to the results of the experiment on the test part of the MNIST data set, the maximum accuracy for the best epoch was 99.6%. Such precision showed a convolutional neural network with thee Batch Normalization layers on mini-batchas, this network converges significantly faster than others, while having the highest total precision
In this paper, the dataset Synth 90k, proposed and developed in the article, is used to train neural networks [1]. This dataset is a program-generated image in grayscale, variable width and 31 pixel height. The training part of the set contains more than 7 million images with 90 thousand unique English words. Image Samples from Synth 90k dataset are presented in Figure 6.
Two data sets were used to recognize text using a trained network: IIIT 5K [20] and Street View Text - SVT [21]. The IIIT 5K dataset is localized text in photographs of storefronts, billboards, advertisements, street signs, etc. The dataset contains 2000 training and 3000 test images. Image Samples from IIIT 5K dataset are presented in Figure 7.
Images with text on storefronts, billboards, signs of various institutions also prevail in the SVT dataset. This dataset contains 249 images of graphic scenes and 647 images of localized text. Image Samples from SVT dataset are presented in Figure 8.
Training sample was 80%, test 20% of the total number of images. The neural network was trained 1 epoch using 7 224 612 images from the Synth 90k dataset. Used graphics processor NVIDIA GeForce GTX 650, the learning process took 110 hours. Figure 9 shows the results of an experimental study to solve the scene text recognition problem using various data sets. The Synth 90k dataset used for training.
The comparison of the obtained results scene text recognition accuracy with the results of other authors was implemented using IIIT5K and SVT datasets. The comparative values are placed in Table 2.
Conclusions 50 words 24.3 64.1 95.5 97.6 99.5 96.7
IIIT5K 1000 words 57.5
The paper presents the application of artificial neural network technologies for the problem of text recognition. Comparison of various neural network architectures to recognition for handwritten digits from the MNIST dataset has been carried out. The best accuracy was shown by the сonvolutional neural network with three Batch Normalization layers. A further development of this approach is scene text recognition. Scene text images are characterized by the absence of clear criteria for distinguishing the background from the text, the heterogeneity of the background, the high probability of various distortions and noise. Such text can be of variable quality, different font, slope, shape, thickness and texture. The neural network architecture combining convolutional, recurrent layers and CTC error function is proposed for recognition of such text. At the stage of experimental research, the effectiveness of the developed CRNN was compared with the data of other authors. Experimental studies conducted on specialized datasets confirm the validity of the use of technologies convolution neural networks for text recognition tasks.