Supervised learning is one of the main types machine learning in which model is trained from data which consists of features (input data) and labels, that is target values. Using our training data, the parameters of our model will be adjusted until loss function reaches a low value or until we get high accuracy on the validation data. Before we start building model, we need make data preprocessing. PCA is often used, to reduce numbers of dimensions our data. Models in which data have been reduced using PCA often have high accuracy. In this article, we will look at how well-known classifiers work such as: K-Nearest Neighbors, Gaussian Naive Bayes and Support Vector Machines, that using PCA. We will also check the performance of the classifiers for which the data has been reduced to fewer dimensions by analyzing correlation tables and we will look at models whose data contain the original number of features. We will evaluate their efectiveness based on the Raisin database and show how decision boundaries built in models that were constructed after our analysis.
KNN and Gaussian Naive Bayes there is no learning with
weights. Using KNN model on large dataset, it can lead
Supervised learning is used in many areas, such as: clas- to high consumption of computing resources. In [17] was
sification [ 1, 2], regression [3], patterns recognition, nat- proposed strategy, which improve the eficiency of KNN
ural language processing and image encryption [
Examples of problems that can be solved using supervised
learning are: classifying whether an email is spam or not, In this paper we will compare three classifiers: KNN,
weather forecasting, classify text, whether a review is Gaussian Naive Bayes and SVM, that were built on three
positive or negative[
the data • model uses two features selected by us • model uses all the features
In classification problem we also distinguish models The database that we used to build various classifiers based on deep neural networks[15]. The architecture contains samples that were described by 7 morphological of neural networks is: weights, activation functions [16], features. These features were obtained after previously loss function and optimizer. In the case of classifier processing the photos.Values are continuous and we can see that each feature has value from diferent ranges.
There are also high values of standard deviations for example, for Area and ConvexArea features, indicating that the values for these features are highly dispersed
Normalization or standardization are used to improve the eficiency and efectiveness of the model. In the case of KNN model, that uses distance measures to classify data samples, if the values weren’t normalized or standardization, features with higher values could have greater impact on model’s result, which could lead to low accuracy. Therefore, an important and recommended action is to use one of the data processing techniques before creating KNN and SVM models. Mainly for Gaussian Naive Bayes doesn’t use data standardization, because this algorithm doesn’t depend on distance, so doesn’t require scaling of features. We used standardization exceptionally in Gaussian Naive Bayes classifiers in which the PCA technique were used and in classifiers in which used twodimensional data to plot decision boundaries.We used standardization, which transforms our data in a way that the mean is equal to 0 and the standard deviation is equal to 1. At the beginning for everyone feature we calculated the mean value and standard deviation and then we used the results to calculate new values using the formula below: = −
One of the popular dimensionality reduction techniques is PCA. The task of PCA is to return n-features that we can create a model with high accuracy. Interesting improvements to PCA models composed for graph based classifiers were presented in [ 18]. In our models were used PCA, which returns to us new training and test data reduced from seven to two dimensions.
Another way to prepare data for the model is to reduce dimensionality based on correlation analysis. Correlation defines the relation between two variables. Correlation value close to 1 or -1 mean a strong correlation, but value close to 0 mean weak correlation. The Extent feature was removed from our training and testing data, because its correlation value with our target feature was only 0.28. Additionally, the following features were eliminated: ConvexArea, Perimeter, Area, MinorAxisLength, because these attributes had strong relation with other features and didn’t contribute relevant information to the classification models. Finally, our classifiers were built on other two features: MajorAxisLength and Eccentricity. The Fig. 1 shows correlation plots between two features.
For each classifier, we also built a model based on all seven features. Sometimes training a model on the basis of all attributes can be a disadvantage, because this approach lead to slower learning of our classifier. However, the advantage of including all features is that in some cases it can lead to very high eficiency of our machine learning algorithm, because we don’t lose any relevant information. Fig. ?? illustrates our feature and correlation graphs. (2) (3) (4) (5) (6) 3.
3.1. KNN
Euclidean distance: Manhattan distance: Minkowski distance: ⎯⎸ (, ) = ⎷⎸∑︁( − )2 (, ) = ∑︁ | − | =1 =1
Canberra distance: Chebyshev distance: (, ) = ︃( ∑︁ | − | =1
)︃ 1 (, ) = ∑︁ | − | =1 || + ||
(, ) = m=a1x | − | Cosine distance:
(, ) = 1 −
∑︀
=1 · √︀∑︀=1 2 · √︀∑︀=1 2 (7)
Updating weights and bias: = − ∇()
= − ∇() Minimizing the cost function using Stochastic Gradient Descent (SGD): KNN classifier is mathematical model, that doesn’t require training. New, unknown points are predicted based on the k-nearest points voting. When classifying a new () = ||||2 + max(0, 1 − ()) (14) sample, model calculates distances between the sam- 1 ple and each point in the specified n-dimensional space. = (15) Among all the distances the model chooses k-smallest and voting takes place. The class that occurs most frequently among the selected points becomes the predicted 3.3.2. Algorithm class for the new sample. We compared performance of SVM classifier (Support Vector Machines) creates a hyKNN classifier for diferent distance measures (metrics): perplane that maximizes the distance between the closest Euclidean, Manhattan, Minkowski for = 3, Canberra, points of two classes (support vectors). When creating Chebyshev and Cosine. a hyperplane, two techniques are used: soft margin and hard margin. Soft margin during the process of training allows our algorithms to make mistakes, so it allows 3.2. Gaussian Naive Bayes points to be on the wrong side of the hyperplane or inside 3.2.1. Formulas the margin. Hard margin during the process of training doesn’t tolerate any errors, so points cannot be on the Bayes’ Theorem in our model: wrong side of the hyperplane or inside the margin. In our (|1, 2, ..., ) = (()|· ∏1︀,=21, ...(,|)) (8) cuasseed, twhheesroefotumradragtianitsenc’htncoiqmupeleatnedlyulsiendeavrasreiopuarsakbelern,weles to transform our data to higher dimensionality. We also (|) = √︀21 2 · exp(− (2− 2 ) ) (9) wusheidchreognuelaorifzathtieonclpasasreasmiesteerquCa. lWtoe 1craenatdedthme oodtehlesr, iins Sample variance: equal to -1. Then, using Stochastic Gradient Descent, we 2 = 1 ∑︁( − ¯) 2 (10) wupedtaetsetdedouoruwremigohdteslsa,nidf tbhaeftperreedaicctheddavtaalsuaems pwlee.reFinneagllay-, − 1 =1 tive, they were assigned the label -1, if non-negative, they are assigned the label 1. We compared the performance of classifiers using diferent kernels such as: linear, poly, 3.2.2. Algorithm rbf, laplacian and sigmoid.
Gaussian Naive Bayes is probabilistic model, that uses Bayes’ Theorem to determine the probability of sample belonging to a specific class. The classifier assumes, that the features are independent. We used this type of classifier, because our data is continuous and the data is approximately normally distributed. During the training process, our model calculated the mean and variance for each attribute from every class and the "a priori" probabilities for each class. When predicting a test data, two probabilities are returned because we have binary classiifcation. We chose the highest probability with its label. 3.3. SVM
Hinge Loss: = max(0, 1 − ()) (11)
4.1. KNN We created the KNN models for diferent metrics for which the prediction is based on 3 nearest neighbors.
4.3. SVM We created SVM models for diferent kernels for specific parameters. These nuclei are: linear, polynomial with degree of 7, rbf with a gamma of 2, laplacian with a gamma of 2 and sigmoid with a gamma of 1.
After analyzing our results for three classifiers, we conclude that using the PCA technique to reduce the dimensionality from 7 to 2 features supported performance of our models, also achieving high accuracies, comparable to the results of models built on all features. After analyzing the correlation of our data, we were able to find two features for which the models had similar accuracy compared to the PCA-based models, these features are: MajorAxisLength and Eccentricity. The accuracy for the KNN classifiers with and without PCA are very similar, ranging from 82% to 88% depending on the metric. In the case of Gaussian Naive Bayes classifiers the accuracy result obtained using PCA and using 7 features gave the same value of 85% , which only confirms the fact that the reduction in dimensions didn’t contribute to the loss of significant information. The last type of classifier, that was analyzed was SVM. After analyzing for diferent kernels, the sigmoid kernel turned out to be the best, which in models with and without PCA indicated the best accuracy of 88%.