A dataset containing over 13k samples of dry beans geometric features is being analysed using machine learning (ML) and deep learning (DL) techniques with the goal to automatically classify the bean specie. The obtained geometrical data has quite a lot redundancy. Many features are strongly correlated. This work analyses the influence of data dimension reduction (DDR) (elimination of excess strongly correlated features) and features scaling (FS), often called normalization, on the machine learning performance (measured in terms of accuracy and approximate training time). Additionally also an influence of activation function (sigmoid vs. ReLU) on artificial neural network performance has been checked.
Classification of dry beans is of some economic importance. Manual classification is labour
intensive, etc. Over 13 k samples of dry beans of 7 various species were photographed and their
geometry was measured via computer vision techniques in [
The dataset used in [
The research question examined in this work is: How do data dimension reduction, feature scaling
and activation function influence machine learning performance? The above question is related to
concurency, specification and programming in the following way. Among topics of CS&P 2021 one
can find: Model checking and testing - this work checks different ML models, knowledge discovery
and data mining - machine learning belong to this field, soft computing - artificial neural networks are
are categorized as a kind of soft-computing.
1.1.
work [
In the handbook [
compared: ReLU and sigmoid.
In
GB
RAM, no graphical processing unit (GPU) acceleration.
Majority of experiments performed were shallow learning that do not need GPU support. As the dry beans dataset is relatively simple, the artificial neural network (ANN) applied was also rather simple and GPU support was not crucial for ANN training. Training times were in range from milliseconds to a few minutes.
The dataset under study consists of 13611 samples. A sample amounts to 16 geometrical features and a label identifying the specie of the bean. The species are as follows: Barbunya, Bombay, Cali,
The features are:
Correlation analysis (see table 1) has shown that several of the features are strongly (positively or
negatively) correlated. This is due to the fact that basically all of them are kind of geometric
measures. In the original work [
It is also sometimes suggested that feature scaling (often called normalization) can improve
performance [
The methods tried were: Naive Bayes Classifier, Decision Tree, Random Forest, Support Vector Classifier.
Results for Gaussian naive Bayes classifier are shown in table 2. One can see that DDR or FS has small effect on training time. Using DDR or FS (or both) significantly increased accuracy from 77.23% to 89.83-91.00%.
Results for decision tree are shown in table 3. Decision tree applied was limited to 16 leaf nodes and maximum depth of 5. One can see that FS has no effect on accuracy and little effect on training time. This probably connected with the fact that DT analyses one feature at the time, so it not cares what is the ratio of specific feature range to other features. DDR shorten training time with limited accuracy decrease.
Decision tree is known to be sensitive for data “rotation”, see [
Results for the random forest (RF) are shown in table 4. The random forest consisted of 150 trees. No limits (max leaves, max depth and etc.) were put on trees. One can observe that training times are longer that for single decision tree (which is reasonable as here we have a set of decision trees). The accuracies are high. DDR shortened training time and allowed for slightly higher accuracy (0,14-0,18 % point). This is quite interesting that although DDR slightly reduced accuracy on single tree it improved accuracy on RF. Similarly to decision tree, SF practically has little effect on training time.
93.06% 93.24% 93.10% 93.24%
4.69 s 2.69 s 4.79 s 2.59 s
Results for support vector classifier (SVC) is shown in table 5. Polynomial kernel has been used. Generally SVC is much more “heavier” model than gaussian classifier, decision tree or random forest. Training times much longer. One can see that DDR or FS has small effect on SVC accuracy. DDR on not scaled features reduced training time. Feature scaling significantly increased training time and increased accuracy a little (about 1% point). The longest training time was observed for DDR and SF data. The training time was 9 times longer than for DDR and not SF data. The author cannot explained this effect.
For an artificial neural network (ANN) the data needs additional treatment. First, the names of
bean species were labelled with numbers and then these numbers 0-6 were codded as so called
”onehot”. The reason of using ”one-hot” encoding is well explained for example in [
Three experiments has been performed to analyse: 1) influence of data dimension reduction,
2) influence of features scaling and 3) influence of activation function (sigmoid vs. ReLU). The ANN
architecture was kept similar (as much as possible) to described in [
Influence of data dimension reduction, data scaling (or normalisation) and activation function has been investigated. The influence depends on machine learning technique used.
Generally data dimension reduction reduces training time with rather limited influence on accuracy Data scaling is a must in case of artificial neural network. Omitting data scaling decreased accuracy from about 93% to about 56%. In case of shallow learning techniques its influence is smaller, it sometimes help a little with accuracy, sometimes not.
Generally scaling had no effect on decision tree and random forest performance. In case of support vector classifier scaling resulted in huge training time increase. Author cannot explain this effect.
The highest accuracy observed was 93,24%. It was obtained 3 times with: 1) random forest with 8 features (scaled and not scaled), 2) ANN, 8 features, scaled and 3) SVC, 16 features, scaled. It is quite intriguing that exactly the same, maximum result repeated 3 time. 7. References