The study presents the results of applying the main known metrics used to evaluate the performance and accuracy of algorithms and neural network models on different classes for the task of graphic content recognition in security systems. For the analysis, different classes of images processed by the neural network algorithm were compared. То evaluates the quality of the algorithm's training based on the results of graphical pattern recognition, nine different metrics for the five conducted correct classification computational experiments were used. The sample used in research, the CamVid benchmark video dataset for training the neural network model, shows different training results for different recognition classes, with this indicator ranging from 38.15% to 97.07% when using the VGG-16 function. At the same time, the highest standard deviation of accuracy, with a value of 0.030351419, was recorded only for the “Pavement” class. This indicates the imperfection of the CamVid training dataset. It should be modified to improve recognition quality by increasing the size and number of test images.
Machine learning and neural networks are closely related,
as neural networks are one of the primary technologies in
the field of machine learning [1–3]. These algorithms are
particularly widely used in security systems. In machine
learning, several key metrics are used to evaluate model
performance. These metrics help to understand how well
the model is performing the given task and to identify areas
where it can be improved. There are several metrics for
evaluating different neural network algorithms [4]. All of
them are used to analyze the recognition of various
properties and characteristics of neural network recognition
algorithms [5]. These are useful for creating an optimal
model of a graphic information recognition system. The
most important ones are the metrics for evaluating the
quality of learning [
Therefore, it is of particular interest to understand
whether there is a correlation between the weight
coefficient of the presence of a particular classification
object in graphic object recognition and the accuracy of
such recognition. For example, in the works [
Therefore, it is important to assess how accurately
graphical patterns are recognized in a specific practical task
[5]. The same systems are used in specific tasks, such as
security systems. In particular, the corresponding modules
are part of intelligent access control systems [
Now mostly part of more complex practical application systems, which are known as Image Identification and Recognition Systems (IIRS). IIRS are often used both for detecting defects on parts within quality control systems according to ISO-9000 standards and for detecting and recognizing the values of vehicle license plates. Based on the results of the IIRS module, the intelligent system can automatically make decisions about granting or denying access to a secured area for a specific object. Another application of such systems is machine vision systems. The common principle of construction for all such systems is: 1)
The technical part of acquiring and initial processing of the image.
0000-0002-5151-9203 (A. Sahun); 0000-0002-4805-8880 (V. Khaidurov); 0000-0001-9695-4543 (V. Lakhno) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 3)
The technical or software part for analyzing and classifying image elements.
The subsystem for registration/identification and summarization of recognition data.
In all similar IIRS systems, this intelligent module with a neural network-based algorithm plays a central role. The accuracy of this module determines the overall performance of the entire system.
For those practical tasks where IIRS is now mostly used, a
mathematical apparatus based on neural networks with
different types of training is applied [
The test model chosen is the neural network model described in [2]. This model has several layers of neurons (Fig. 1). Given the practice of using neural network-based algorithms in recognition and identification systems, a deep-learning neural network model was chosen. This is due to several existing advantages of such models for graphic identification/recognition tasks [1, 3].
The main goal of the study is the evaluation of the accuracy of a neural network algorithm in the task of recognizing graphic content.
The neural network diagram of the IIRS shown in Fig. 1 operates with the Haar feature. This approach is most effective when using a deep-learning neural network.
In the basic model described in [2], the input layer of neurons receives initial data, such as the intensity of each pixel and Haar features for various graphical objects to be identified (bushes, trees, cars, roads, sky, sidewalk elements, fences, pedestrians, etc.).
In the Matlab environment, there is a built-in function vgg16() which implements the architecture of a deep neural network. There is also a function analogous to it, vgg19(). The first function operates with 16 convolutional and fully connected layers of neurons, including 13 convolutional and 3 fully connected layers. This function is used for image classification in the process of pattern recognition. The vgg16() function in MATLAB returns a neural network object but does not contain a specific method for computing distances (metrics) between feature vectors for processed images.
The vgg19() function also implements the architecture of a deep neural network and has an input size of 224×224×3. Unlike vgg16, the neural network in the vgg19 network is trained and fine-tuned on a dataset of graphical data containing over 1,000,000 images and 1000 classes. This allows this neural network to have more powerful capabilities for feature extraction in images. To define metrics based on VGG19 in MATLAB, we first need to load and prepare the VGG19 model, and extract image features from a specific layer of the neural network. After this, both vgg16 and vgg19 functions must use different metrics to compare these features. That is, neither function has builtin distance metric determination.
To use distance metrics with feature vectors extracted from the VGG16 model in MATLAB, we have to follow these steps: 1)
Loading and preparing the VGG16 Model (use the pre-trained VGG16 model to extract feature vectors from images. 2) 3)
Extracting Feature Vectors (feed your images through the VGG16 model to get the feature vectors). distance metrics to compare the feature vectors). = where is the number of categories.
Confusion matrix. This matrix shows the number of
correct and incorrect classifications for each class. It
shows the proportion of correctly classified objects among
all objects. This metric is well suited for tasks where classes
are balanced. The expression below provides an example of
obtaining the accuracy
metric in
machine learning
algorithms [
measures the proportion of correctly classified positive objects among all objects classified as positive. This metric is important when the cost of false positive results is high. In (2) we present the expression for computing the accuracy metric in machine learning.
= .
Below are the main known metrics used to evaluate the includes TP, FP, TN, and FN for each category. performance of algorithms and neural network models on different classes of graphic content recognition. These metrics are used in machine learning [2].
Area under the ROC curve. The ROC curve shows the relationship between TPR and FPR at different thresholds. The area under the curve (AUC) measures the model’s Accuracy metric in machine learning. Accuracy ability to distinguish between classes (7).
False Negative Rate (FNR). The FNR measures the negative classifications, FP (False Positive) is the number of proportion of false negative results among all positive incorrect positive classifications, and FN (False Negative) is examples during training.
= = . .
(6) (7) (8) (9)
False Positive Rate (FPR). The FPR measures the proportion of false positive results among all negative examples during training.
The above-mentioned metrics help objectively assess the quality and effectiveness of the model for identifying graphical objects in a video surveillance system based on neural networks, as well as choosing the most efficient algorithm for specific conditions and tasks.
In this research, all the evaluation metrics (1)–(9) listed above were used to assess the quality of model training. The quality metric values of the algorithm training obtained in 5 computational experiments
Class name Exp #1 (1) (2) (3) (4) (5) classes). 0,8698 0,4188 0,4342 0,3251 0,4921 0,8988 0,758 0,8145
Exp #2 0,9320 0,8647 0,9397 0,9867 0,4468 0,4347 0,3264 0,5825 0,9218 0,8281 0,8172
Exp #3 0,9479 0,9181 0,9483 0,9551 0,6394 0,4896 0,4621 0,6245 0,9542 0,9104 0,9492
Exp #4 0,9348 0,9126 0,9455 0,9749 0,5463 0,4549 0,3698 0,5978 0,9594 0,8328 0,8576
Exp #5 0,9818 0,8786 0,9541 0,9848 0,5070 0,4465 0,4243 0,6582 0,9732 0,8972 0,8207 .
⋅ + .
. 1 − = 2 ⋅
The F1-score metric of recall. The F1-score is the harmonic mean between precision and recall. It is useful when balancing these two metrics is necessary. It is calculated according to the expression provided below:
metric. IoU is used to evaluate the quality of segmentation and object detection by measuring the ratio of the intersection area of predicted and ground truth objects to their union area. It is calculated according to the expression provided below:
Recall metric in machine learning. Recall measures the proportion of correctly classified positive objects among all training obtained in 5 computational experiments actual positive objects. This metric is important when the cost (calculated result of correct classification of objects for all of false negative results is high. The following expression is used to compute this metric (2):
Mean Average Precision metric. Average precision is calculated for each category and then averaged across all categories. This metric is often used for object detection tasks. Such a metric is particularly relevant for evaluating the training quality of this neural network-based model. The metric value can be determined using the expression provided below:
Fig. 2 shows weights coefficient indicators object recognition for the test video data segment. The significance of the Intersection over Union (IoU) metric, calculated for each of the semantic classes, lies in its ability to measure the accuracy of the neural network’s recognition performance. IoU assesses how well the predicted segmentation overlaps with the ground truth segmentation for each class. Higher IoU values indicate better performance, meaning the predicted areas closely match the actual areas. This metric is crucial for evaluating the effectiveness and reliability of the neural network in accurately recognizing and segmenting different semantic classes within the graphical content. In Fig. 3 we can see values of the IoU Accuracy evaluation metric. As can be understood from above, the most important and resultant indicator of model training quality is the IoU (Intersection over Union) metric. The result of the correct classification of objects for each class in the 5 conducted computational experiments values for different detection classes are presented in Figs. 4–6.
Considering that the model was trained on 421 images, it can be considered that its training level may be sufficient for the graphical identification task at hand. But we see that the training quality even for the same semantic classes varies significantly across the different 5 experiments.
The smallest value of such a deviation will be for objects of the “Bicyclist” class at 0.76%, and the largest will be for objects of the “Fence” class at 25.25%.
Such a difference can be explained by various reasons. For example, the imperfection of the algorithm or the insufficient quality or length of the training data sample. As shown by the calculations obtained in Table 1, the most accurate results of the learning algorithm NM were obtained for the classes: “Road”—97.06%, “Sky”—94.46%, and “Car”— 94.16% accuracy of correct recognitions. At the same time, the recognition quality of images of the type “SignSymbol” was 38.15%, and “Tree” had 45.19% accuracy of correct recognition. The average learning quality of this algorithm on the test fragments was 75.42%.
Analyzing the data presented and visualized in Table 1 and Figs. 4–6, it can be said that the quality of the learning algorithm described in [2] significantly depends on the accuracy of the training. The accuracy of image recognition in neural network-based algorithms is highly dependent on the quality of training. Here are some key points of this dependence: Training Data Quality; Training Data Quantity; Preprocessing; Algorithm Complexity; Training Process. The sample used in the study [2] CamVid benchmark video dataset for training the neural network model shows different training results for different recognition classes. This indicator ranges from 38.15% to 97.07% when using the VGG-16 function. It can be noted that all the provided training quality metrics on the same recognition classes yield approximately the same accuracy values. While the variance (standard deviation) indicator is highest only for the “Pavement” class. It amounts to 0.030351419.
The obtained average recognition accuracy of graphical objects at 75.42% is comparable to the recognition rate of 98.7%. This indicates insufficient training quality due to the shortcomings of the training dataset.
It can be assumed that the simplest way to improve
recognition accuracy could also be using a more complex
neural network algorithm. Such one present in MatLab is
called VGG-19 [