In machine learning we can emphasize models based on the such as supervised learning and unsupervised learning. Mainly algorithms based on learning without teacher are used to clustering process. This algorithms are used to split ours data to smaller groups, clusters with similar and comparable attributes. Guided learning is utilized to create many classifiers. On the basis of previously prepared training data, the classifier learns certain relations and dependencies so that it can correctly predict target values later. In our paper we will look at two rule-based models that use decision rules to classify data samples. Examples of models are Decision Tree and Random Forest which are created for diferent hyperparameters. We will also show how the reduction of dimensionality afects to efectiveness and eficiency our models by using PCA technique and correlation analysis to select the most relevant features.
before training the algorithm is preparing data. This
preparing is based on standardization or normalization
Artificial intelligence is used in many areas. In image our dataset. In the case of high dimensionality of the
processing for example is applied in feature correction data, various dimensionality reduction techniques are
and encryption [
We will also check the efectiveness of above, our models.
In the case of KNN model we can establish number of In the case of Decision Tree for diferent measure:
ennearest neighbors. However, often the problem is to de- tropy and gini, and for various depths. For the Random
termine the optimal value k-nearest neighbors, in [
The database that we used to build various classifiers contains samples that were described by 7 morphological features. These features were obtained after previously processing the photos.Values are continuous and we can see that each feature has value from diferent ranges. There are also high values of standard deviations for example, for Area and ConvexArea features, indicating that the values for these features are highly dispersed from their mean. The Fig. 1 shows a table containing the statistics of our attributes.
To improve the efectiveness and eficiency of the model, data normalization or standardization is used. However rule-based models don’t require transformations to a single scale, because this classifiers make predictions based on specific rules. Nevertheless, in our case, we have standardized for:
ize the data. • models that were built from lower dimensionality data using the PCA technique. When using this technique, it is recommended to before standard• models that were based on two features that we chose. Standardization data contributed to changes in values to a similar range which helped in the creation of decision boundary charts.
In our classifiers, we used standardization that transforms the data in such a way that its mean is equal to 0 and the standard deviation is equal to 1. First for every attribute we calculated its mean and standard deviation. Later, we used the obtained results to compute the new values using the below formula: = − (1)
One of the popular dimensionality reduction techniques is PCA. The task of PCA is to return n-features that we can create a model with high accuracy. PCA model can be improved for sophisticated data on the input, [16] presented denoising of the input for improved processing. In our models were used PCA, which returns to us new training and test data reduced from seven to two dimensions.
Another way to prepare data for the model is to reduce dimensionality based on correlation analysis. Correlation defines the relation between two variables. Correlation value close to 1 or -1 mean a strong correlation, but value close to 0 mean weak correlation. The Extent feature was removed from our training and testing data, because its correlation value with our target feature was only 0.28. Additionally, the following features were elimi- Information gain: nated: ConvexArea, Perimeter, Area, MinorAxisLength, because these attributes had strong relation with other features and didn’t contribute relevant information to the Gini coeficient: classification models. Finally, our classifiers were built on other two features: MajorAxisLength and Eccentricity.
The Fig. 3 shows correlation plots between two features.
Information gain: before − after − ∑︁ ·
2 =1 − =1 ∑︁ ()
before − after
For each classifier, we also built a model based on all seven features. Sometimes training a model on the basis of all attributes can be a disadvantage, because this approach lead to slower learning of our classifier. However, the advantage of including all features is that in some cases it can lead to very high eficiency of our machine learning algorithm, because we don’t lose any relevant information. Fig. 2 illustrates our feature and correlation graphs.
3.1.1. Formulas
Entropy after: − − ∑︁ · log2() =1 =1 ∑︁ () 3.1.2. Algorithm A Decision Tree is a directed model that consists of a root, nodes, leaves and edges. Root is top of the tree, passing through the edges, we come to the nodes and finally to the leaves, to the lowest layer of the tree. Leaves contain the answers, predictions of our model, to which class our data sample is classified. Nodes contain rules that are used to make decisions during testing. Rules are created using impurity measures. These are: entropy and gini coeficient. Our classifiers will create rules that will divide our sets into more pure subsets. The final conditions are those for which the information gain is the greatest. The Decision Tree has a tendency to overfitting, so we used the following as regularization parameters: number of max depth is 2 and 3, and the minimum amount of data in the set before the division can not be less than 2.
Random Forest algorithm creates a forest in a random manner. This “forest” you can think of as an ensemble of Decision Trees, most of the time trained with the “bagging” method. The general idea of the bagging method is that a combination of learning models increases the overall result. The Random Forest starts by selecting random samples from the given dataset. It selects these random subsets with replacement, meaning that some samples may be used multiple times in a single subset. e features at each split in the tree. This randomness in feature selection is what gives the Random Forest its name. The Random Forest consist of many decision trees. Test data is classified by decision trees. Next, voting takes place and we look at which class/forecast occurs most (2) frequently. Random Forest is better option than Decision Tree, because this classifier has not a tendency to overfitting to training data. Our models include several dozen decision trees, where each of them has been trained for (3) diferent training data that has been previously randomized from the main dataset intended for training.
neighbors, IEEE Transactions on Neural Networks and Learning Systems (2018) 1774–1785.
After an in-depth analysis carried out on Decision tree [8] G. D. Magistris, C. Rametta, G. Capizzi, C. Napoli, and Random Forest models, it can be concluded that us- Fpga implementation of a parallel dds for wide-band ing PCA to reduce dimensionality for our dataset is good applications, volume 3092, 2021, pp. 12 – 16. idea. Presented models of decision trees achieve high [9] H. S. Parmar, S. Mitra, B. Nutter, R. Long, S. Anaccuracies for a depth equal of 2 at level 83 %, which tani, Visualization and detection of changes in brain were trained on a training dataset using PCA. In addition, states using t-sne, 2020 IEEE Southwest Symposium after analyzing the correlation, we were able to find two on Image Analysis and Interpretation (SSIAI) (2020) features for which the models made predictions as good 14–17. as the models for which PCA were used. Random Forest [10] C. Napoli, G. Pappalardo, E. Tramontana, A hyis the model which make even more efective predictions. brid neuro-wavelet predictor for qos control and Classifier of this type achieved an accuracy of 90 % using stability 8249 LNAI (2013) 527 – 538. doi:10.1007/ 100 decision trees. Additionally, an important element 978-3-319-03524-6\_45. turned out the right choice of impurity measure, our [11] M. Woźniak, M. Wieczorek, J. Siłka, Bilstm deep research confirm that classifiers using the entropy mea- neural network model for imbalanced medical data sure gave better accuracy results than models that used of iot systems, Future Generation Computer Systhe gini coeficient. To sum up, the use of PCA for our tems 141 (2023) 489–499. database allowed us to achieve equally high accuracies, [12] F. Bonanno, G. Capizzi, G. L. Sciuto, C. Napoli, while reducing computational complexity. G. Pappalardo, E. Tramontana, A novel clouddistributed toolbox for optimal energy dispatch management from renewables in igss by using wrnn References predictors and gpu parallel solutions, 2014, pp. 1077 – 1084. doi:10.1109/SPEEDAM.2014.6872127. [13] A. Sikora, A. Zielonka, M. F. Ijaz, M. Woźniak, Digital twin heuristic positioning of insulation in multimodal electric systems, IEEE Transactions on
Consumer Electronics (2024). [14] S. Pepe, S. Tedeschi, N. Brandizzi, S. Russo, L. Iocchi, C. Napoli, Human attention assessment using a machine learning approach with gan-based data augmentation technique trained using a custom dataset, OBM Neurobiology 6 (2022). doi:10.
21926/obm.neurobiol.2204139. [15] A. Alfarano, G. De Magistris, L. Mongelli, S. Russo,
J. Starczewski, C. Napoli, A novel convmixer transformer based architecture for violent behavior detection 14126 LNAI (2023) 3 – 16. doi:10.1007/ 978-3-031-42508-0\_1. [16] W. Dong, M. Woźniak, J. Wu, W. Li, Z. Bai, Denoising aggregation of graph neural networks by using principal component analysis, IEEE Transactions on Industrial Informatics 19 (2022) 2385–2394.