This article examines popular classifiers such as Bagging Classifier, Nearest Neighbors Classifier, Boosting Classifier, Support Vector Classifier for the highest performance accuracy. Classifiers will be tested for accuracy based on human brain activity data. Brain activity data collected during repetitive mechanical movements. In the process of development of modern technologies, the mechanism of interaction between a person and a computer has become one of the main approaches for interaction with the world. The search and improvement of existing approaches contributes to the further development of the scientific and technological progress of society. Today, there are a large number of approaches for interaction with technical devices. One of the main principles of interaction that is used to restore missing or impaired locomotor parts of the body is the principle of using neural signals. We focused our attention precisely on the signals of the cerebral cortex. The correct interpretation of the appropriate sets of signals into specific mechanical movements is important for helping those who need prosthetic limbs or restoration of their motility. The main goal of this study was to identify the most accurate way of processing brain signals using machine learning, high-precision computing and cloud services to parallelize the calculation process. Creation of an information system that allows determining the will of the user in mechanical movements.
In this paper, we decided to investigate different classifiers for their accuracy in identifying correct predictions based on input data. The topic of neuro signal processing is widely used, but most of the research on this topic is poorly studied. It is important to investigate certain aspects related to the processing of human brain signals, which we will try to highlight in this article.
The data obtained during the physical experiment were used as the basis of the study, during which the subject performed repetitive mechanical movements of the fingers of the hand and turns of the hand. Brain activity data were collected during the performance of a specific mechanical movement. As a result of the experiment, it was possible to obtain 14 unique data sets, each of which was responsible for showing the will to perform one or another movement. After receiving the data, we created several models that performed programmed high-precision manipulations with the data to build a model for predicting likely volitional movements. We created several separate information systems, each of which was based on a specific classifier model. In the process of work, you used the following classifiers: Bagging Classifier, Nearest Neighbors Classifier, Boosting Classifier, Support Vector Classifier.
The purpose of this experiment was to find the best classifier that works with the highest accuracy.
The results of this study are important for further research work related to the recognition of brain signals for motor-motor systems. Figure 1 shows the visualized data of one of the packages.
A person without motility disorders was chosen as the subject. A 16-channel encephalograph was used for data collection. 3. Machine learning and high-precision computing
The Python programming language was chosen for the work, as there is a wide set of libraries and modules for this language, which helps speed up the research process. The main libraries used are pandas, numpy, matplotlib.pyplot, collections, sklearn.
For optimization, cloud services were used to parallelize the calculation process. This is due to the fact that brain activity is a constant process and quite fast, so data collection must occur at a speed close to real time. In the experiment, we used a data collection frequency of 500 Hz. As a result of data collection, we received more than 6,000 units of sets of 16 sensors and we made 14 such sets. As a result, at the time of research, there are more than 84,000 lines, where each line corresponds to the values of the sensors at the corresponding moment in time. The problem of parallelization of the calculation process is due to the fact that we need to get the resulting model as possible in the minimum time. It is cloud services that make it possible to break this process into several streams. It should also be noted that when writing data processing scripts, there is an option for automatic parallelization at the most resource-intensive stage, namely during cross validation when the hyper parameter n_jobs=-1 is specified. 4. The process of creating models
All received packets were read and converted into convenient data structures using open Python libraries. In each package, certain manipulations were carried out, such as removing redundant columns and assigning to each a unique identifier corresponding to a specific mechanical movement.
The next step was to combine the data of all packages into one dataset. The total data set was divided into training and testing data. Training data is needed so that our model is formed on their basis, and test data is needed to check the accuracy of the model's work on data with which it was not familiar until that moment.
It is important to standardize and normalize the data before starting the cross-validation process on the appropriate classifier. This will help reduce the time in the following stages and help bring all the data to a uniform appearance.
For each classifier, we defined accuracy, f1_weighted, roc_auc_ovr_weighted. To determine the overall accuracy of the corresponding classifier, we used formula 1.
accuracy general=
averagaccuracy +averag¿+averag¿
naccuracy
(
For each classifier, 10 cross-validations are indicated, for the possibility of later obtaining a finer accuracy when calculating the overall accuracy of the work. 4.1. Determining the accuracy of the Bagging Classifier
Let's start with the Bagging Classifier. After cross-validating the data using the Bagging Classifier, we obtained the values of 10 cross-validations. As a result, we got a matrix of results for accuracy, f1_weighted, roc_auc_ovr_weighted (Table 1). accuracy 0.819713 0.772561 0.807273 0.814630 0.732456 0.805202 0.807452 fold 1 2 3 4 5 6 7 8 9 10 0.806952 0.779895 As a result, we got the following average values: “f1_weighted” = 0,789581; “accuracy” = 0,791175; “roc_auc_ovr_weighted” =0,972307.
The overall accuracy for the Bagging Classifier is 0.85102174. This classifier showed quite good accuracy and has prospects for its improvement. 4.2. Determining the accuracy of the Nearest Neighbors Classifier After cross-validating the data using the Nearest Neighbors Classifier, we obtained the values of 10 cross-validations. As a result, we got a matrix of results for accuracy, f1_weighted, roc_auc_ovr_weighted (Table 2). 4.3. Determining the accuracy of the Boosting Classifier After cross-validating the data using Boosting Classifier, we obtained the values of 10 crossvalidations. As a result, we got a matrix of results for accuracy, f1_weighted, roc_auc_ovr_weighted (Table 3). accuracy 0,735992 0,817689 0,760538 0,779882 0,79981 0,773502 0,802298 0,804286 0,800893 0,811154 0,728328 0,760152 0,81189 0,799885 0,804551 0,801238 0,778309 0,818221 0,801252 0,773253 0,959569 0,953828 0,964756 0,957359 0,964878 0,96854 0,963378 0,964317 0,966126 0,949287 fold 1 2 3 4 5 6 7 8 9 10 fold 1 2 3 4 5 6 accuracy 0,118189 0,120195 0,097865 0,199554 0,12929 0,131313 0,123457 0,094524 0,105679 0,089197 accuracy 0,750161 0,740468 0,828296 0,828296 0,828296 0,759192 Accuracy Values for the Boosting Classifier Over 10-fold Cross-Validation As a result, we got the following average values: “f1_weighted” = 0,118768; “accuracy” = 0,120926; “roc_auc_ovr_weighted” =0,536872.
The overall accuracy for Boosting Classifier is 0.258855. This classifier showed a poor accuracy result, answering correctly only a quarter of the time. Perhaps in the future, hyperoptimization of the parameters will improve the results. 4.4. Determining the accuracy of the Support Vector Classifier After cross-validating the data with Support VectorClassifier, we obtained the values of 10 cross-validations. As a result, we got a matrix of results for accuracy, f1_weighted, roc_auc_ovr_weighted (Table 4). 0,764573 0,757369 0,807463 0,96527489 0,96208538
As a result, we got the following average values: “f1_weighted” = 0,793667; “accuracy” = 0,780651; “roc_auc_ovr_weighted” = 0,970361897;
The overall accuracy for the Support Vector Classifier is 0.848227. This classifier showed a poor accuracy result, answering correctly only a quarter of the time. Perhaps in the future, hyper-optimization of the parameters will improve the results. 4.5. The results of classifier accuracies The general summaries of the results obtained by the classifiers are as follows: Bagging Classifier 85,1021%; Nearest Neighbors Classifier 84,5838% Boosting Classifier 25,8855%; Support Vector Classifier 84,8227%;
After analyzing the four most popular algorithms, we saw that three of them worked with good accuracy. Bagging Classifier turned out to be the most productive, but it is worth hyperoptimizing the parameters for Nearest Neighbors Classifier and Support Vector Classifier. There is a possibility that with certain parameters, one or another classifier can increase its accuracy for our problem. Boosting Classifier showed the lowest result when working with our problem. Figure 2 shows the normalized confusion matrix for the Bagging Classifier. 5. Conclusions Research on approaches related to cognitive signal processing is important for the development of computer hardware and software technologies. Finding new ways and their research methods helps to find more accurate models for data processing. Given that all data are of a different nature, this raises the issue of finding the right models that will most accurately work in a specific situation.
As a result of our research, we saw that almost all the classifiers we considered showed good accuracy. The Boosting Classifier showed the worst accuracy, so it is not recommended for future research. In the future, it is worth optimizing the parameters to find sets of parameters for each classifier that would give the highest result.