A dataset containing over 13k samples of dry beans geometric features is being analysed using machine learning (ML) and deep learning (DL) techniques with the goal to automatically classify the bean species. First the original dataset was reduced to eliminate redundant features (too strongly correlated and echoing others). Then the dataset was visualised and analysed with machine learning techniques: Multinomial Bayes, Support Vector Machines, Decision Tree, Random Forest, Voting Classifier and Artificial Neural Network. The overall accuracies obtained were in range: 88.35 - 93.61%.
Correlation analysis has shown that several of the features are strongly (positively or negatively) correlated. This is due to the fact that basically all of them are kind of geometric measures. The decision was taken to drop some features to avoid correlations over 0.9 (or negative correlation below -0.9) between them. The benefits of such a decision should be: 1) a significant reduction of the computational complexity 2) a lower risk of overfitting 3) ease of visualisation. The disadvantage is a limited risk of loosing some valuable information and, as a result, a decrease in accuracy.
Extent -0.0781 0.1460
Thus, in this work it was decided to limit the set of features list to these 8 members:
MajorAxisLength, MinorAxisLength, AspectRatio, Extent, Solidity, Roundness, ShapeFactor2,
ShapeFactor4, and to exclude: Area, Perimeter, Eccentricity, ConvexArea, EquivDiameter,
Compactness, ShapeFactor1, ShapeFactor3. The issue of high correlations among some features was
not addressed in [
It shows that the Bombay species is trivial to classify as its beans are significantly bigger than others. the classification of other species seems to be much more difficult, and we can expect more errors. The correlations between pairs of the selected features are listed in Table 1.
In this work the following techniques were used: Multinomial Gaussian Classifier, Support Vector Classifier, Decision Tree, Random Forest, Voting Classifier, Artificial Neural Network (Multilayer Perceptron or MLP).
The full dataset was divided into the training and test subsets. 80% of samples were used for
training and 20% for testing. Division of all available samples into the training and test subsets is
crucial for a correct methodology. The aim of all ML or DL methods is to achieve a ”generalization”
ability. Thus it is important to check the accuracy of classifying new samples, ones that have not been
used during training. Otherwise, there is a very serious risk that the model will suffer form overfitting.
Overfitted models perform very well on the training data but much worse on new data. Overfitting (as
one of the most important issues in ML) is widely discussed in ML handbooks [
Naive Bayes models are based on Bayes's theorem. They are extremely fast and simple, but on the
other hand, their performance is usually limited. They can be used as a baseline for classification
problems (see [
The overall accuracy obtained with Multinomial Bayes Classifier was 64,30 %. The problem was to classify into 7 different classes. Thus the blind (random) classification should result in about 1/7 = 14,29% accuracy. As one can see, classification is more difficult if there are more classes. Random classification should give accuracy equal to about 1/(number of classes). Thus we can see that even this simple model perform about 50 percent points better than the random approach. 4.2.
Support vector machines (SVM), which can be used as regressors or classifiers, are considered a
very powerful and flexible algorithms. On the other hand. they may need a lot of computing power
(see [
The results are quite similar for all kernels. The accuracy can be further improved to some extent (tenths of %, maybe 1%) by increasing C, but this will also significantly increase the training time.
A decision tree (DT) belongs to the class of so called non-parametric algorithms. The term nonparametric can be misleading. In fact, a decision tree has parameters, but their number is not constant.
During the learning phase, a decision tree tries to find the best questions partitioning the dataset in
order to reduce information impurity (the measure is the Gini index or information entropy). The great
advantage of decision trees is that they are extremely intuitive. On the other hand, a decision tree has
no limited degrees of freedom, so it is easy to overfit (if the user is not aware of that). The splits made
by a decision tree are always orthogonal (made on one feature at a time), so the decision tree is very
sensitive to data rotation (see [
In [
Figure 2 shows the decision tree obtained under the above limits. The overall accuracy is 88.35%. Preliminary tests showed that a better accuracy of about 92,3% could be obtained with a bigger decision tree; however, the bigger the decision tree, the less intuitive it becomes, and the more difficult it is to visualise.
In another experiment with a big decision tree we set max depth =10 and max leaf nodes =30. The accuracy improved and reached 91.59% (Table 3).
It can be seen that to obtain an accuracy similar to that reported in [
To see this better, assume that in some dataset we have two parameters A and B, and the classification is obvious, but it depends on the A/B ratio that is not explicitly present in the dataset. This can be hard for a decision tree to solve. The addition of an extra column, A/B, will add no new information to the dataset, but it will help the decision tree quite a bit. It can be supposed that in the dry bean case, the 8 removed categories contained little extra information, but they presented essentially the same information in a way more appropriate for the decision tree. 4.4.
The random forest idea is as follows: take many decision trees (employing some randomness, so the trees differ) and let them vote. So the classification decision taken by a random forest is a decision taken by the most numerous group of decision trees in a random set of trees.
Usually a random forest performs better than a single decision tree. However, a random forest is considered a ”black-box” model being very hard to interpret.
A random forest of 150 decision trees was created. No restrictions on trees were applied. The
accuracy obtained was 93.61%, the best so-far, better than the best accuracy reported in [
The idea of ”voting”, which by default is used in random forests, can be applied to any classifiers. There are 2 main ways of voting: ”hard” (straightforward, direct voting) and ”soft” (the votes are weighted depending on how confident the classifier is with its choice). Like in the case of a random forest, there is a good chance that the voting result will be more accurate than for any particular classifier.
The hard voting classifier was implemented using 3 classifiers described above: the radial kernel SVC, the ”big” decision tree, and the random forest.
The obtained accuracy was 92.80%. Thus in this case it is worse than for the random forest. This gives us a clue that voting should be used carefully and preferably with models exhibiting similar performance; otherwise ”stupid” models can outvote ”smart” models. It seems that this flaw of democracy does not only apply to human societies, but is more universal in nature. 4.6.
Besides the (shallow) machine learning/data science methods presented above, a deep learning technique, the so-called dense artificial neural network, has also been tried.
For an artificial neural network the data needs additional treatment. First, the names of bean
species were labelled with numbers and then these numbers 0-6 were codded as so called ”one-hot”.
The reason of using ”one-hot” encoding is well explained for example in [
Two architectures of ANN were tried. The first one is similar to the one described in [
The overall accuracy was 92.58%. In an attempt to improve it, another ”bigger” ANN was tried.
Besides the normal layers, a dropout layer was added. A dropout layer only works during the training
and randomly ”cuts off” (sets to zero) some inputs. It is expected that dropout layers reduce the risk of
overfitting [
The dry beans dataset has been analysed by different machine learning and deep learning techniques. Table 4 shows the summary results.
It can be seen that in general the task of beans classification is a relatively simple task in terms of the necessary computing power. All training times were shorter than 1 minute using a free googlecolab computer.
The accuracy of Naive Bayes is much worse than for the other methods. This is not surprising, as
this method is known to be fast but not very accurate, and is suggested as a preliminary method to
check if there is „something” in the data, rather than to do a full analysis. The other methods give
accuracy in a relatively close range of 88.3-93.6%. This is comparable to [
The authors in [
The only technique, that suffered from this elimination to some extend seems to be decision tree. This is due to the fact that decision tree operates on one feature at a time and cannot ”combine” features. See also the discussion in p. 4.3 devoted to the decision tree section.
One can see that the models vary strongly in the terms of the computing time. SVC and ANN (and also Voting Classifier, because it includes SVC) are the slowest learners. Random forest seems to be the best method in this case, as it perform best and its training time is also reasonable.
Confusion matrices provide a comfortable way to visualise results in more details and compare actual values with predicted ones. The confusion matrix for the random forest classifier (the best performer) will be discussed further. It is presented in figure 5. The most frequent mistakes were between Dermason and Sira (38 + 44). On the other hand, Bombay was classified perfectly which is not surprising. It is easy to notice that Bombay beans are significantly bigger than other species.
The dry beans dataset appeared to be an interesting dataset to demonstrate and compare ML techniques. Two ideas for further research: Deeper insight how and if the elimination of correlated features influences the ML training process. This study shows that there is little, if any, performance decrease. One may try to investigate if the elimination reduces the training time and how much. Despite ”manual” feature reduction, as done in this work, on may try to use PCA (the primary component analysis) to reduce the dimensionality of data and also analyse its influence on model performance (accuracy and training time) 6. References