This article discusses several classification algorithms of recognizing numbers from photographic images or with manual input, namely: support vector machine (SVM), Knearest neighbors (KNN), random forest (RF) and several variants of neural networks. The success rates of the algorithms in the field of handwriting recognition were compared. Six variants of recognition technology were analyzed and tested: using classifier from Scikitlearn package and using deep learning neural networks. To construct and train neural networks or train classifiers, a well-known and rather complete base of handwritten digits MNIST was chosen. Two types of neural networks were considered: sequential and convolutional. The training of neural networks was carried out using a variable number of steps (epochs). Recognition images were scaled to a size of 28x28 (784 cells in onedimensional representation). Preliminary processing of images (filtering, scaling, etc.) was carried out using the OpenCV library. For recognition, each image of a digit was converted to a 28x28 size and fed to the input of a pre-trained neural network. A technique to select the area of interest in photographs containing hand-written digits for further recognition has been devised. For handwritten digit recognition, the best recognition accuracy is provided by a convolutional neural network, as 97.6% of car ladle digits were recognized correctly with it. To improve the recognition accuracy for handwritten digits, it is necessary to perform two additional stages of image preprocessing and dataset transformation. After building recognition models using all the algorithms mentioned above, the recognition accuracy of all handwritten digits on the test program turned out to be within 98-100%. For industrial images regardless of the used neural network version, the recognition accuracy was 96-98%.
There are a few examples of technical application of optical number or digit recognition from photographs, i.e. recognition of railway carriage numbers, car license plates, product marking, readings of recording devices, etc.
This article discusses several classification algorithms for recognizing numbers in photographic images or with manual input, namely: support vector machine (SVM), K-nearest neighbors (KNN), random forest (RF) and several variants of neural networks. The algorithms success rates in the field of handwriting recognition were compared. In the methods section of this article, brief information is given about handwriting recognition and compared machine learning methods. In the experimental section, the values obtained as a result of the study were compared. Evaluations were made on the compared machine learning algorithms.
Optical Character Recognition (OCR) is an important area of research in artificial intelligence and
character recognition [
By now a wide range of studies have been carried out, including a comprehensive study and implementation of various popular algorithms for recognizing handwritten text, including handwritten numbers. The task of segmentation and recognition of image areas containing handwritten or printed characters is relevant due to the presence of numerous technical applications.
For example, license plate recognition in work [
Character recognition is a classification task that includes recognizing a set of characters in an image, dividing them into 10 classes in case of numbers or 26 classes in case of the Latin alphabet letters.
Many systems are currently available for identifying printed text. However, according to [
There are some problems that hinder the implementation of character recognition, namely: low quality of photo or scanned copy, but this problem is partially solved by image preliminary processing. presence of distorted characters, especially when working with handwritten documents or numbers, due to the peculiarities of character style. similarity between outlines of some characters.
For example, in work [
In works [
In work [
Improving the accuracy of handwritten digit recognition is achieved by increasing the complexity
of the used deep learning neural networks. For example, in [
This paper provides a reasonable understanding of machine learning and deep learning algorithms like SVM, KNN, RF, CNN, and MLP for handwritten digit recognition. Furthermore, it provides information about the algorithm which is efficient in performing the task of digit recognition. The related work that has been done in this field followed by the methodology and implementation of all the algorithms for their better understanding will be discussed in the following sections of this paper. Next, it presents the conclusion and result. The last section of this paper contains used citations and references.
Goals of this research: 1. Evaluation of recognition accuracy for machine learning and deep learning algorithms such as SVM, KNN, RF, CNN and MLP in relation to real sets of handwritten numbers. 2. Analysis of algorithms influence of an image preliminary processing on recognition accuracy. 3. Evaluation of the possibilities to use the considered algorithms for solving technical problems associated with the processing of handwritten digits noisy images (on the example of recognizing cast iron ladle numbers).
The comparison of the algorithms (Support vector machines, KNN, Random Forest, Multi-layered perceptron & Convolutional neural network) is based on the characteristic chart of each algorithm on common grounds like dataset, the number of epochs, complexity of the algorithm, accuracy of each algorithm, specification of the device (Ubuntu 20.10, i5 9th gen processor, 8GB memory) used to execute the program and runtime of the algorithm under ideal condition.
To build and train the model, a well-known and fairly complete base of handwritten digits MNIST
was selected [
The Support Vector Machine (SVM) was first proposed by Vapnik and since then has attracted a
high degree of interest in the machine learning research community [
Decision trees form a classification model as a tree structure for the solution of a problem. The tree structure and rules are easy to understand. This simplifies the implementation of the algorithm. Decision trees method consists of simple sequential decision making operations. One of the most important steps in creating the tree structure is choosing the attribute value for which the branching in the tree will be determined [20,21].
A multilayer perceptron is a class of feedforward artificial neural networks that consists of at least three layers: input, hidden, and output. Except for the input neurons, all neurons use a non-linear activation function.
The number of hidden layers can be increased to any number according to the problem, without limitation on the number of nodes. The specific number of hidden layers or the number of nodes in a hidden layer is difficult to determine due to the unstable nature of the model and is therefore chosen experimentally. Each hidden layer in the model can have different activation functions for processing. For teaching purposes, it uses a supervised learning method called backpropagation. In MLP, a node connection consists of a weight that is tuned to synchronize with each connection as the model is trained. [22]
KNN is one of the classification methods. Nearest neighbor searching is the following problem: We are given a set S of n data points in a metric space, X, and the task is to preprocess these points so that, given any query point q X, the data point nearest to q can be reported quickly. This is also called the closest-point problem and the post office problem. Nearest neighbor searching is an important problem in a variety of applications [23 The KNN algorithm is used to assign a new observation to the one of the classes that is most common among the k neighbors of a given element, the classes of which are already known. Classification is made according to the threshold value determined with the average of the k data that appears to be the closest. The performance of the method is influenced by the closest neighbor number, threshold value, similarity measurement and sufficient number of normal behaviors in the learning cluster [24-25].
Random forest algorithm can be used in both classification and regression problems like decision trees. The logic of work is to create more than one decision tree and produce average results with the help of these trees. The reason why this algorithm is called random is that it offers extra randomness during the creation of the tree structure. When splitting a node, instead of looking for the best attribute directly, it looks for the best attribute in a subset of random attributes. This situation creates more diverse trees [21, 26-27].
Deep convolutional neural networks (CNNs) are a specialized kind of ANNs that use convolution in place of general matrix multiplication in at least one of their layers [28]. CNN is a deep learning algorithm that is widely used for image recognition and classification. Unlike neural networks with a simpler architecture, which have one or more hidden layers, CNNs are composed of many layers. Such a feature allows them to compactly represent highly nonlinear and varying functions [29]. CNNs involve many connections, and the architecture is typically comprised of different types of layers, including convolution, pooling and fully connected layers, and realize form of regularization [30]. In order to learn complicated features and functions that can represent high-level abstractions (e.g., in vision, language, and other AI-level tasks), CNNs would need deep architectures. Deep architectures, and CNNs, consist of a large number of neurons and multiple levels of latent calculations of nonlinearity. According to [31], each level of CNN architecture represents features at a different level of abstraction defined as a composition of lower-level features.
CNN uses a filter (kernel) which is an array of weights to extract features from the input image. CNN employs different activation functions at each layer to add some non-linearity [32].
Many works have noted the importance of image preprocessing for recognizing handwritten numbers or letters.
In particular, [33] applied various preprocessing methods to improve the performance of CNN models. Translations, rotations and elastic deformations have been studied for a variety of in the area of machine learning goals [34-36].
Acording to the authors [33], preprocessing with the combination of elastic and rotation improves the accuracy of the three analyzed networks up to 0.71%.
According to [37], many of today’s OCR systems are built following traditional approaches to image processing and work great with printed text but if use them for handwritten text recognition in images it can get unexpected results with poor recognition quality.
The quality of a learned system is primarily dependent of the size and quality of the training set. In work [36] was proposed a simple technique for vastly expanding the training set on base of elastic distortions. In [38] the MINST dataset size was increased by four times by using affine transformations and the prior knowledge of transform invariant properties. In the raised method, the elastic distortions are applied to each sample of the training set to extend nine new samples.
These distortions improve the results on MNIST substantially.
To compare the algorithms based on working accuracy, execution time, complexity, and the number of epochs (in deep learning algorithms) this paper used three different classifiers: Support Vector Machine Classifier, KNN Classifier, Random Forest Classifier, Multilayer Perceptron Classifier.
This set of algorithms was implemented using the Scikit-Learn package in the Python programming language [38].
More complicated Multilayer Perceptron Classifier and Convolutional Neural Network Classifier were implemented using the TensorFlow with Keras frontend package in the Python programming language.
The Keras library contains numerous implementations of widely used building units of neural networks, such as layers, target and transfer functions, optimizers, and many tools to simplify the work with images and text. 5.1.
To select areas of images containing recognizable digits, we used OpenCV library tools. The findContours function or the Maximally stable extremal region extractor (mser) algorithm were used to highlight the contours of the digits.
To recognize numbers in photographs of real cast iron ladle cars, an algorithm based on boundary detection was used [39]. The algorithm for preprocessing the image and highlighting the area containing the digits of the number included the following stages: 1. image filtering to reduce the noise level (a Gaussian filter was used - cv2.GaussianBlur function); 2. binarization of the image to cut off noise (the cv2.threshold function was used, its parameters were adapted to reliably select the outlines of the digits); 3. highlighting the contrasting borders on the image (the Canny edge detector was used cv2.Canny function, for which the values of the maximum and minimum values of the gradient were preliminarily selected); 4. filtering to reduce the effect of image heterogeneity (the median filter was used cv2.medianBlur function); 5. image binarization (cv2.threshold function was also used); 6. morphological transformation (dilatation - function cv2.dilate); 7. selection of contours and their sorting (selection of contours was carried out using the cv2.findContours function); 8. image segmentation, i.e. selection of recognition areas in the form of rectangle set containing previously selected contours of digits (cv2.boundingRect functions were used).
Recognition images to be scaled to a size of 28x28 (784 cells in a one-dimensional representation). Each pixel value of the images lies between 0 to 255 followed by Normalizing these pixel values by converting the dataset into 'float32' and then dividing by 255.0 so that the input features will range between 0.0 to 1.0. Next, one-hot is performed encoding to convert the y values into zeros and ones, making each number categorical. 5.2.
Classical classification algorithms were implemented on the basis of the package Scikit-learn [40]. It is an open source machine learning library that supports supervised and unsupervised learning. It also provides various tools for model fitting, data pre-processing, model selection and evaluation, and many other utilities.
To implement the Support Vector Machine algorithms, the module sklearn.svm was used. The support vector machines in scikit-learn support both dense (numpy.ndarray and convertible to that by numpy.asarray) and sparse (any scipy.sparse) sample vectors as input. However, to use an SVM to make predictions for sparse data, it must fit for such data. For optimal performance, use C-ordered numpy.ndarray (dense) or scipy.sparse.csr_matrix (sparse) with dtype=float64.
For binary and multiclass dataset classification, scikit-learn implements the SVC, NuSVC, and LinearSVC classes. SVC and NuSVC are similar methods, except for slightly different sets of parameters and they have different mathematical formulations (see section Mathematical formulation). On the other hand, LinearSVC is another (faster) implementation of Support Vector Classification for the case of a linear kernel.
The implementation of C-Support Vector Classification is based on libsvm. The fit time scales at least quadratically with the number of samples and may be impractical beyond tens of thousands of samples.
The tuning of the classifier was carried out by choosing the type of kernel, the regularization parameter and Kernel coefficient for ‘rbf’, ‘poly’ and ‘sigmoid’ kernel types.
A random forest classifier is a meta estimator that fits a number of decision tree classifiers on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting. The sub-sample size is controlled by the max_samples parameter if bootstrap=True (default), otherwise the whole dataset is used to build each tree. The adjustment of this classifier was carried out by selecting the optimal value of max_samples parameter (it is a number of trees in the forest).
Scikit-learn package implements two different nearest neighbors classifiers: KNeighborsClassifier implements learning based on the k-nearest neighbors of each query point, where k is an integer value specified by the user. RadiusNeighborsClassifier implements learning based on the number of neighbors within a fixed radius r of each training point, where r is a floating-point value specified by the user.
The k-neighbors classification in KNeighborsClassifier is the most commonly used technique. The optimal choice of the value k is highly data-dependent: in general, a larger one suppresses the effects of noise, but makes the classification boundaries less distinct.
In this work, KNeighborsClassifier was used with the choice number of neighbors required for each sample.
The original Multilayer perceptron was also implemented using scikit-learn package. As the experience of setting up MLPClassifier showed, the following parameters have the main influence on the accuracy of character recognition: hidden_layer_sizes represents the number of neurons in the i-th hidden layer; solver represents the solver for weight optimization; learning_rate_initdouble represents the initial used learning rate. It controls the step-size in updating the weights (only used when solver=‘sgd’ or ‘adam’).
It was found that a result acceptable in terms of accuracy is achieved when choosing a sufficiently large number of neurons in the hidden layers, sgd or adam optimizers, and a moderate initial training speed.
All variants of the classifier models after tuning on the MNIST dataset were saved using the joblib module.
The other variant of Multilayer perceptron was implemented using TensorFlow package with Keras interface.
The Keras library contains numerous implementations of widely used building units of neural networks, such as layers, target and transfer functions, optimizers, and many tools to simplify the work with images and text.
The model for recognizing digits included the input and output layers, as well as one or more hidden layers. The model was trained using a variable number of steps (epochs). Recognition images to be scaled to a size of 28x28 (784 cells in one-dimensional representation), so the number of neurons in the input and hidden layers was assumed to be 784.
The output level is a layer with 10 nodes tf.nn.softmax, which returns an array of ten probability estimates, the sum of which is 1. Each node contains an estimate that indicates the likelihood that the current image belongs to one of 10 classes.
For recognition, each image of the digit was converted to a size of 28x28, and then fed to the input of a pre-trained neural network.
When setting up the model according to the MNIST test data or a set of images for further recognition, it was found that the recognition accuracy increases slightly with the increase in the number of hidden layers. The model setup is much more sensitive to the choice of the type and parameters of the optimizer and the number of training epochs.
The implementation of handwritten digit recognition by Convolutional Neural Network was done using TensorFlow and Keras.
Variants of structures of the deep learning neural network based on Keras framework, which were used to recognize car number elements, are shown in Fig. 1.
а) sequential neural b) convolutional neural network (CNN) network (feed-forward network - FNN) with three
dense inner layers Figure 1: Structures variants of the deep learning neural network based on Keras framework, which were used to recognize car number elements
A simpler version of a fully connected neural network (Fig. 1, a) in the test set provided a recognition error level of 1.3-1.5%, but in real examples, the recognition accuracy did not exceed 70% [33].
A more accurate digit recognition of areas of interest was achieved using a convolutional neural network (CNN, see Fig. 1b). The disadvantage of this type of networks is a significantly longer duration of training.
In the sample of 15 ladle car numbers, which included a total of 42 digits (12 three-digit and 3 two-digit numbers), 41 digits were correctly recognized using a convolutional neural network, so that the recognition accuracy was 97.6%. Recognition errors are connected with poorly written numbers. n o iit n g o c e r 0,98 0,96 tcceau scsea 00,,9924 r a f o itno 0,9 rcaF 0,88
The stages of constructing and training a model and recognition of practical samples are easily separated due to the possibility of exporting models. To do this, the model.save method (filepath) was used to save the Keras model in one HDF5 file, which contains: architecture of the model, allowing to restore the model; model weight; training configuration (loss calculation function, optimizer) state of the optimizer, which allows to resume training exactly where it was stopped.
To restore the model, the function keras.models.load_model (filepath) was further used. This function also allows to build a model using the saved training configuration (if the model has never been compiled).
However, when using customized models for recognizing handwritten numbers for other technical objects – carriage numbers, photographs of meter readings, etc., for all model variants (including those based on CNN), the number of errors increased sharply – up to 40-50%, depending on the dataset.
With proper tuning of all the models mentioned above (from KNN to CNN – as the complexity grows), the accuracy of the MNIST test sample estimate was 97.5-98.5%, which is slightly inferior to the best results achieved using preconfigured or convolutional neural networks.
The results of the SVC classifier as part of the scikit-learn package are shown in Fig. 2. 1 0,984 rbq linear poly
0,982 a) comparison of recognition results for different b) comparison of recognition results kernels (kernels «rbq», «linear», «poly» were used ) for different values of the regularization parameter Figure 2: The results of the recognition accuracy evaluation for SVC classifier with different calibration parameters
As can be seen from the results shown in Fig. 2, the best accuracy of image recognition from the MNIST sample using the SVC classifier is achieved for the «rbf» kernel and a regularization parameter of at least 50.
Similar studies were carried out for other classification options - KNN Classifier, Random Forest Classifier, Multilayer Perceptron Classifier. The results of the analysis of the influence of the tuning parameters of the KNN Classifier and Random Forest (RF) Classifier classifiers are shown in Fig. 3.
As can be seen from fig. 3, a decrease in the number of nearest neighbors for the KNN Classifier or an increase in the number of trees in the forest for RF Classifier leads to an increase in recognition accuracy. However, both of the considered classification algorithms do not have any reserves for improving the recognition accuracy by varying the settings.
The settings of the multilayer perceptron had a great influence on the recognition accuracy. The results of the computational experiment are shown in Fig. 4. As can be seen from the graphs presented, the recognition accuracy of digit samples from the test sample increases with an increase in the number of neurons in the hidden layers (it was assumed to be the same for all layers) and with an increase in the number of hidden layers.
The graphs in Fig. 4 are built using the MLP Classifier method from the scikit-learn package. For a similar classifier built using the Keras package, similar results were obtained for the effect on the recognition accuracy of the number of hidden layers and the number of neurons in a layer. 0,974 0,975 0,972 a) KNeighborsClassifier results b) Random Forest Classifier results Figure 3: The results of the recognition accuracy evaluation for KNeighborsClassifier and Random Forest Classifier with different calibration parameters trea se ccu csa a n fo iito ion gno t rca rce F
0,99 0,985
0,98 0,975
0,97 0,965 0 200 400 600
800
Figure 4: The results of the recognition accuracy evaluation for MLP Classifier (scikit‐ learning package) with different parameters
When testing various recognition models, some features of the MNIST dataset were established (fig. 5): all images of numbers have a clearly defined border area; images are grayscale, the brightness of pixels varies across the width of the digit image; the images of the numbers are centered in relation to the 28x28 area.
Figure 5: Sample images from the MNIST database
However, the area of interest images for recognition are actually black and white, either due to the binarization of the image at a preprocessing stage or are initially black and white.
To improve the recognition accuracy, it is necessary to perform two additional stages of image preprocessing: - after highlighting the area of interest contour exactly along the boundaries of the digit, this part of the image is centered in the square area; - the border of the image is added with a width of 15-25% of the size of the square area.
The width of the border area significantly affects the reliability of digit recognition (both printed and handwritten).
Results of the study of influence of the added width of the boundary region in the recognition accuracy set of handwritten characters 0-9 are shown at Fig. 6. A computational experiment for constructing this curve was performed using Scikit-learn implementation of the SVC algorithm.
An analysis of the effect of the width of the added boundary region was carried out for variants of digit recognition using neural networks. The shape of the curve in Fig. 6 for variants with a multilevel perceptron or a CNN network remained exactly the same as for the classification algorithms. An example of recognizing a set of handwritten digits 0-9 is shown in Fig. 7.
Border width 20% of the maximum image size. Recognition accuracy is 100%. b) Border width 50% of the maximum image size. Recognition accuracy 80% (recognition errors is indicated by arrows).
Figure 7: Examples of recognition results with CNN deep learning network for a group of handwritten digits with different widths of the added border area
To assess the reasons of the recognition failure, a test program in Python was developed that allows to visually draw numbers and monitor the state of the recognition area and the result (Fig. 8).
Figure 8: Appearance of the program window for testing handwritten character recognition
After building recognition models using all the algorithms mentioned above, the recognition accuracy of all handwritten digits on the test program turned out to be in the range of 98-100% (one or no errors per 50 drawn digits).
Another reserve for improving the recognition accuracy is setting up the base for building models to work with black and white images. For this, images from the MNIST dataset were converted to black and white.
The results obtained are largely related to the peculiarities of highlighting areas of interest containing digits in the image. The selection of the contour (the findContours function in OpenCV) is carried out exactly along the border of the founded contour. The selected area in the form of a rectangle described around the contour of the digit, without region of interest additional processing, noticeably differs from the base for recognition (MNIST).
A similar result has been achieved in industrial images. Regardless of the used neural network version, the recognition accuracy was 96-98%. For elements of the ladle car number binarization and morphological “closing” operations (MORPH_CLOSE in terms of the OpenCV library) were performed before recognition.
This research paper has implemented some models namely Support Vector Machine Classifier, KNN Classifier, Random Forest Classifier, Multilayer Perceptron Classifier, Multi-Layer Perceptron and Convolutional Neural Network for handwritten digit recognition using MNIST datasets. It compared them based on their working accuracy.
1. After a simple setup, all these algorithms demonstrated almost the same accuracy of handwritten digit recognition, differing within +1%. 2. It was found that CNN gave the most accurate results for handwritten digit recognition, but the only drawback is that it took an exponential amount of computing time. 3. To improve the recognition accuracy, it is necessary to perform two additional stages of image preprocessing and dataset transformation: after highlighting the area of interest contour exactly along the boundaries of the digit, this part of the image is centered in the square area; the border of the image is added with a width of 15-25% of the size of the square area; converting images from the MNIST dataset to black and white form.
After building recognition models using all the algorithms mentioned above, the recognition accuracy of all handwritten digits on the test program turned out to be in the range of 98-100% (one or no errors per 50 drawn digits). A similar result was obtained when recognizing the generated sets of digits with different shapes – the recognition accuracy reaches the recognition accuracy on the MNIST test sample. For industrial images regardless of the used neural network version, the recognition accuracy was 96-98%. 7. References