Introduction

Supervised learning is used in many areas, such as: classification [1,2], regression [3], patterns recognition, natural language processing and image encryption [4,5,6]. Examples of problems that can be solved using supervised learning are: classifying whether an email is spam or not, weather forecasting, classify text, whether a review is positive or negative [7,8].

The popular algorithm used during training model for example in the case of regression or SVM classifier, is the gradient descent, which minimizes loss function, by adjusting the parameters of our model in the direction of the decreasing gradient of the loss function [9,10]. The goal of supervised learning is to achieve high accuracy to make right predictions on unknown data. There are many interesting improvements to such models for application systems. In [11] was presented how to use machine learning for imbalanced data inputs, while in [12,13] was presented positioning of technical systems for power electric models. We can find also many applications for complex input data structure ie. [14] gave it for the graph based input relations compositions.

In classification problem we also distinguish models based on deep neural networks [15]. The architecture of neural networks is: weights, activation functions [16], loss function and optimizer. In the case of classifier SYSYEM 2023: 9th Scholar's Yearly Symposium of Technology, Engineering and Mathematics, Rome, December 3-6, 2023 ap307985@polsl.pl (A. Polowczyk); ap307986@polsl.pl (A. Polowczyk) KNN and Gaussian Naive Bayes there is no learning with weights. Using KNN model on large dataset, it can lead to high consumption of computing resources. In [17] was proposed strategy, which improve the efficiency of KNN classifier on Big Data.

In this paper we will compare three classifiers: KNN, Gaussian Naive Bayes and SVM, that were built on three various data:

• model uses PCA to reduce the dimensionality of the data • model uses two features selected by us • model uses all the features We will check the effectiveness of the above models, in the case of the KNN for different metrics and for the SVM model we will test the performance for various kernels. We will summarize whether reduction of the dimensions of our data allows us to get satisfactory results, leading to a decrease in computational complexity.

Raisin database

The database that we used to build various classifiers contains samples that were described by 7 morphological features. These features were obtained after previously processing the photos.Values are continuous and we can see that each feature has value from different ranges. There are also high values of standard deviations for example, for Area and ConvexArea features, indicating that the values for these features are highly dispersed from their mean.

Agnieszka Polowczyk et al. CEUR Workshop Proceedings 9-16

Standardization

Normalization or standardization are used to improve the efficiency and effectiveness of the model. In the case of KNN model, that uses distance measures to classify data samples, if the values weren't normalized or standardization, features with higher values could have greater impact on model's result, which could lead to low accuracy. Therefore, an important and recommended action is to use one of the data processing techniques before creating KNN and SVM models. Mainly for Gaussian Naive Bayes doesn't use data standardization, because this algorithm doesn't depend on distance, so doesn't require scaling of features. We used standardization exceptionally in Gaussian Naive Bayes classifiers in which the PCA technique were used and in classifiers in which used twodimensional data to plot decision boundaries.We used standardization, which transforms our data in a way that the mean is equal to 0 and the standard deviation is equal to 1. At the beginning for everyone feature we calculated the mean value and standard deviation and then we used the results to calculate new values using the formula below:

𝑥𝑛𝑒𝑤 = 𝑥 − 𝜇 𝜎 (1)

Model based on PCA

One of the popular dimensionality reduction techniques is PCA. The task of PCA is to return n-features that we can create a model with high accuracy. Interesting improvements to PCA models composed for graph based classifiers were presented in [18]. In our models were used PCA, which returns to us new training and test data reduced from seven to two dimensions.

Model based on two features

Another way to prepare data for the model is to reduce dimensionality based on correlation analysis. Correlation defines the relation between two variables. Correlation value close to 1 or -1 mean a strong correlation, but value close to 0 mean weak correlation. The Extent feature was removed from our training and testing data, because its correlation value with our target feature was only 0.28. Additionally, the following features were eliminated: ConvexArea, Perimeter, Area, MinorAxisLength, because these attributes had strong relation with other features and didn't contribute relevant information to the classification models. Finally, our classifiers were built on other two features: MajorAxisLength and Eccentricity. The Fig. 1 shows correlation plots between two features.

Model based on all features

For each classifier, we also built a model based on all seven features. Sometimes training a model on the basis of all attributes can be a disadvantage, because this approach lead to slower learning of our classifier. However, the advantage of including all features is that in some cases it can lead to very high efficiency of our machine learning algorithm, because we don't lose any relevant information. Fig. ?? illustrates our feature and correlation graphs.

Methods

KNN

Formulas

Euclidean distance:

𝐷(𝑥, 𝑦) = ⎯ ⎸ ⎸ ⎷ 𝑚 ∑︁ 𝑖=1 (𝑥𝑖 − 𝑦𝑖) 2(2)

Manhattan distance:

𝐷(𝑥, 𝑦) = 𝑚 ∑︁ 𝑖=1 |𝑥𝑖 − 𝑦𝑖|(3)

Minkowski distance:

𝐷(𝑥, 𝑦) = (︃ 𝑚 ∑︁ 𝑖=1 |𝑥𝑖 − 𝑦𝑖| 𝑟 )︃ 1 𝑟(4)

Canberra distance:

𝐷(𝑥, 𝑦) = 𝑚 ∑︁ 𝑖=1 |𝑥𝑖 − 𝑦𝑖| |𝑥𝑖| + |𝑦𝑖|(5)

Chebyshev distance:

𝐷(𝑥, 𝑦) = 𝑚 max 𝑖=1 |𝑥𝑖 − 𝑦𝑖|(6)

9-16

Cosine distance:

𝐷(𝑥, 𝑦) = 1 − ∑︀ 𝑚 𝑖=1 𝑥𝑖 • 𝑦𝑖 √︀∑︀ 𝑚 𝑖=1 𝑥 2 𝑖 • √︀∑︀ 𝑚 𝑖=1 𝑦 2 𝑖 (7)3

Gaussian Naive Bayes

Formulas

Bayes' Theorem in our model:

𝑃 (𝑦|𝑥1, 𝑥2, ..., 𝑥𝑛) = 𝑃 (𝑦) • ∏︀ 𝑛 𝑖=1 𝑃 (𝑥𝑖|𝑦) 𝑃 (𝑦|𝑥1, 𝑥2, ..., 𝑥𝑛)(8)𝑃 (𝑥𝑖|𝑦) = 1 √︀ 2𝜋𝜎 2 𝑦 • exp(− (𝑥𝑖 − 𝜇𝑦) 2𝜎 2 𝑦 )(9)

Sample variance:

𝜎 2 = 1 𝑛 − 1 𝑛 ∑︁ 𝑖=1 (𝑥𝑖 − 𝑥 ¯)2 (10)

Algorithm

Gaussian Naive Bayes is probabilistic model, that uses Bayes' Theorem to determine the probability of sample belonging to a specific class. The classifier assumes, that the features are independent. We used this type of classifier, because our data is continuous and the data is approximately normally distributed. During the training process, our model calculated the mean and variance for each attribute from every class and the "a priori" probabilities for each class. When predicting a test data, two probabilities are returned because we have binary classification. We chose the highest probability with its label.

SVM

Formulas

Hinge Loss:

𝜀𝑖 = max(0, 1 − 𝑦𝑖𝑓 (𝑥𝑖))(11)

Updating weights and bias:

𝑤𝑡 = 𝑤𝑡 − 𝜂∇𝑤𝐶𝑜𝑠𝑡(𝑤𝑡) (12) 𝑏 = 𝑏 − 𝜂∇ 𝑏 𝐶𝑜𝑠𝑡(𝑤𝑡)(13)

Minimizing the cost function using Stochastic Gradient Descent (SGD):

𝑚𝑖𝑛𝐶𝑜𝑠𝑡(𝑤𝑖) = 𝜆||𝑤|| 2 + max(0, 1 − 𝑦𝑖𝑓 (𝑥𝑖))(14)𝜆 = 1 𝑁 𝐶(15)

Algorithm

SVM classifier (Support Vector Machines) creates a hyperplane that maximizes the distance between the closest points of two classes (support vectors). When creating a hyperplane, two techniques are used: soft margin and hard margin. Soft margin during the process of training allows our algorithms to make mistakes, so it allows points to be on the wrong side of the hyperplane or inside the margin. Hard margin during the process of training doesn't tolerate any errors, so points cannot be on the wrong side of the hyperplane or inside the margin. In our case, where our data isn't completely linear separable, we used the soft margin technique and used various kernels to transform our data to higher dimensionality. We also used regularization parameter C. We created models, in which one of the classes is equal to 1 and the other is equal to -1. Then, using Stochastic Gradient Descent, we updated our weights and b after each data sample. Finally, we tested our models, if the predicted values were negative, they were assigned the label -1, if non-negative, they are assigned the label 1. We compared the performance of classifiers using different kernels such as: linear, poly, rbf, laplacian and sigmoid.

9-16

Experiments

KNN

We created the KNN models for different metrics for which the prediction is based on 3 nearest neighbors.

SVM

We created SVM models for different kernels for specific parameters. These nuclei are: linear, polynomial with degree of 7, rbf with a gamma of 2, laplacian with a gamma of 2 and sigmoid with a gamma of 1.

Conclusion

After analyzing our results for three classifiers, we conclude that using the PCA technique to reduce the dimensionality from 7 to 2 features supported performance of our models, also achieving high accuracies, comparable to the results of models built on all features. After analyzing the correlation of our data, we were able to find two features for which the models had similar accuracy compared to the PCA-based models, these features are: MajorAxisLength and Eccentricity. The accuracy for the KNN classifiers with and without PCA are very similar, ranging from 82% to 88% depending on the metric. In the case of Gaussian Naive Bayes classifiers the accuracy result obtained using PCA and using 7 features gave the same value of 85% , which only confirms the fact that the reduction in dimensions didn't contribute to the loss of significant information. The last type of classifier, that was analyzed was SVM. After analyzing for different kernels, the sigmoid kernel turned out to be the best, which in models with and without PCA indicated the best accuracy of 88%.