-The last few years have been characterized by a big development of machine learning (ML) techniques, and their application has spread in many fields. The success of their use in a specific problem strongly depends on the approach used, the dataset formatting, and not only on the type of ML algorithm employed. Tools that allows the user to evaluate different classification approaches on the same problem, and their efficacy on different ML algorithms, are therefore becoming crucial. In this paper we present OINOS, a suite written in Python and Bash aimed to the evaluation of performances of different ML algorithms. This tool allows the user to face a classification problem with different classifiers and dataset formatting strategies, and to extract related performance metrics. The tool is presented and then tested on the classification of two diagnostic species from a public electroencephalography (EEG) database. The flexibility and ease of use of this tool allowed us to easily compare the performances of the different classifiers varying the dataset formatting and to determine the best approach, obtaining an accuracy of almost 75%. OINOS is an open source project, therefore its use and sharing are encouraged.
In recent years we have witnessed the development of
new machine learning (ML) techniques and the improvement
of the existing ones, and their application has expanded in
many fields [
In this work we present OINOS, a suite for the evaluation of classifier performances, composed of a set of modules for the comparison of ML algorithms with respect to different dataset partitioning strategies. OINOS is written in Python and ©2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Bash, and implemented as an applicative for the execution of multithreaded benchmark.
In order to give an example of application, we considered electroencephalography (EEG) data related to the problem of alcoholic prediction, i.e., the classification between patients suffering from alcoholism and healthy patients, based on EEG times series of a second of brain activity. Such dataset has been chosen for the high prediction complexity and because the data is publicly available (at https://kdd.ics.uci.edu/databases/ eeg/eeg.html). In this problem the ML tools learn to glean the correlations among the fluctuation of brain signals obtained from the different channels and their dependance on the subject’s pathological state.
The use of a custom dataset partitioning procedure allowed us to find satisfactory performances without the need to overload the data preprocessing. OINOS has simplified us to find alternative approaches to train the classifiers.
The analyzed data belongs to a test concerning 122 subjects. From each of them it has been collected a set of 120 trial. Each trial consists in the measurement of 1 sec. of EEG signals caught from 64 electrodes placed on the subject’s scalp. During the trials, the subjects were exposed to three kind of stimuli: une single image, two matching images or two nonmatching images alternately. Since subjects belongs to the 2 category alcoholic and non-alcoholic and the stimuli to the 3 kind single, matching and non-matching, the EEG data have been labeled through those 2 cohordinates (e.g. if a trial has been caught from an alcoholic patient while he was looking to a non-matching couple of figure, the trial label will be alcnon-matching).
Here we describe the structure of the presented tool and its operation modes.
The algorithms and the logic underlying the classification
processes of OINOS are implemented by the libraries
scikitlearn [
The component modules are: 1) main: the entry point of the suite. This block is responsible for the execution and the orchestration of single modules; 2) OINOS core: the main component. It implements the logic of comparison among different ML algorithms; 3) datalogger: the module for the output management and the experiment reports.
A. Use
In order to start OINOS it is necessary to execute the starter present in the root directory of the project through the command: $ ./start.
In this way the program will return: ================ = OINOS V1.0 = ================ Select an option: 1. learn from the ’Alcoholic’ dataset from UCI Knowledge Discovery in Databases 2. learn from the ’Wrist’ dataset from NeuCube [1,2, quit]:
From this menu it is possible to select on which dataset the prediction algorithms must be tested.
By selecting the first option, OINOS will acquire the datasets of the database of the UCI Knowledge Discovery in Databases. Before to start the execution, OINOS will ask to the user: 1) to specify the dataset among those available; 2) to specify the destination path for the output 3) to specify which portion of the data (i.e., ratio) with respect to the overall dataset, will be used for testing. 4) to specify the number of executions of the same prediction test. The importance of this setting is notable because once fixed the cardinality of the training and test sets (through the ratio), the related elements will be randomly selected; of course at each run the performances will vary depending on the specific training set of each experiment and it may be reasonable to study them in a statistical sense.
At the end of this configuration phase the comparison between the prediction algorithms is executed.
are named with a suffix that indicates the dataset cardinality. For example, if data 100 is selected by the user, it will be executed the prediction on a sample of 100 elements. If a ratio of 0.2 is specified, the prediction will be redistributed using 80 elements as training set and the remaining 20 as test set.
b) Execution: During the execution, messages of four comparison categories are printed on the standard output; at each category corresponds a different classification to be presented to the prediction algorithms: 1) alcoholic-control: the EEG time-series of the dataset are classified as pertaining to alcoholic (alcohol) or healthy (i.e., control) patients; 2) single-matching-non matching images: the EEG timeseries pertain to participants which a single flickering image (single), two identical alternate images (matching) or two different alternate images (non matching) are shown; 3) single-matching-nonmatching images for alcoholic and control: the intersection of the two previous classifications is considered (alcoholic patient watching a single image, alcoholic patient watching two identical images, etc.); 4) alcoholic-control extended: the six classes of the previous step are considered and projected to the two classes alcoholic and control.
The execution goes through these categories in four phases, showing the results of the test on the screen in terms of: classification (ALC-CTRL, SGL-MATCH-NONMATCH,
CTRL EXT, respectively) overall cardinallity of the dataset (for example 100 for data 100) cardinality of the test set (for example 20 for ratio = 0:2) type of classifier under testing accuracy of the classifier, computed as
T P + T N Acc =
T P + T N + F P + F N precision of the classifier, computed as
T P P r =
T P + F P recall of the classifier, computed as
T P Rec =
T P + F N F1 Score of the classifier, computed as (1) (2) (3)
P r Rec F 1 = 2 (4)
P r + Rec where TP stands for true positive, TN for true negative, FP for false positive and FN for false negative. A TP is an outcome where the model correctly predicts the positive class; similarly, a TN is an outcome where the model correctly predicts the negative class. A FP is an outcome where the model incorrectly predicts the positive class, and a FN is an outcome where the model incorrectly predicts the negative class.
c) output: When the execution is finished, the output will be avaible at the path specified during the configuration phase: a microsoft excel file (.xlsx) with the report, as described above; a figure with the comparison between the different accuracy values.
With this option it is possible to directly call the Python relative sources.
This allows the user the execution in unattended mode, useful for the implementaton of custom procedures and benchmarks. The related scripts are :=bin=oinos:py and :=bin=wrist:py respectively; the switch -h enables the help, which returns the following information to the user: $ ./bin/oinos.py -h usage: $ python bin/main.py -d <data path> -r <testing data ratio> -o <output path> example: $ ./bin/oinos.py -d data 100 -r 0.3 -v -o out —————————————————————— $ ./bin/wrist.py -h usage: $ python bin/main.py -r <testing data ratio> -o <output path> example: $ ./bin/oinos.py -r 0.3 -v -o out
This menu makes possibile to select the dataset on which the prediction algorithms have to be tested.
This dataset comes from the database of the Knowledge
Discovery in Databases Archive of the University of California,
Irvine [
In our case, the gathered data comes from an experiment conducted on 122 subjects, each of which underwent 120 trials of the same task.
The task consists in a second of EEG activity recorded while
the subject is asked to watch alternatively:
a single image (case identifiec as single, i.e., SNGL)
two identical images (matching, i.e., MATCH)
two different images (non matching, i.e., NONMATCH)
For each presented stimulus, ten trials of a second of
activity, recorded by 64 electrodes, have been gathered.
Electrodes were located on the head of the subject, to record
fluctuations of postsynaptic activity [
Using the metadata of the experiment, the samples have been subdivided with respect to the type of stimulus given to the subject (SNGL, MATCH, NONMATCH) or on the bases of the type of subject (alcoholic, control). Different tests have been done to evaluate the performances of the classifiers considered, using different configurations (only subject type, only stimulus type, combined).
Unfortunately, neither classifiers who achieved greater success during repeated runs were able to achieve satisfying performances.
Therefore we implemented a different method. In addition to the three types of prediction described above, we implemented - alcohol control extended, i.e., alc-ctrl ext - able to project the classifications obtained by the combined configuration (six classes, one for each combination of pathology and stimulus) in the two classes alcoholic and control; we therefore classified the data in the most stringent way to go back into abstraction and generalize the final solution.
This new way to predict the classes alcoholic and control has significantly improved the performances of the classifiers, in the way we give illustration below.
We have conducted a benchmark of 100 consecutive run to analyze the performances of the following classifiers: K Nearest Neighbors Linear SVM RBF SVM Linear SVC Gaussian Process Decision Tree (with max depth = 5) Decision Tree (with max depth = 10) Random Forest Gradient Boosting Classifier Neural Net Ada Boost Naive Baies Linear Discriminant Analysis
QDA After the execution of the run, an average for each one of the metrics has been done. Although the performances are not very good, it has to be noted that the introduction of the approach acc-ctrl extended has significantly affected the performance of some of the classifiers.
The two types of classifications alcoholic - control (in blue) and alcoholic -control extended (in orange) have been compared, underlining the benefits of the latter approach. The results are summarized in figures 3, 4, 5, 6.
Among the different classifiers tested, it is worth highlighting the cases Linear SVC and Neural Network. Their classification for alcoholic control were just above the average of the other classifiers, with accuracy and precision near the 60%. Such performances have considerably improved with the extended approach.
datasets with the aim of make them usable for Machine Learning research (http://kdd.ics.uci.edu/). In the website different dataset are available, indexed for typology and semantic area. EEG data selected for our study are categorized into the section Time Series - EEG (http://kdd.ics.uci.edu/databases/ eeg/eeg.data.html).
In this work we present OINOS, a suite for the evaluation of classifier performances, composed of a set of modules for the comparison of several ML algorithms with respect to different datasets. We faced a classification problem based on neurophysiology data (i.e., EEG time series), to distinguish alcoholic to non/alcoholic subjects during the execution of a task. Through the performance evaluation of a set of classifiers we found the better configuration among the proposed classifiers and dataset formatting strategies. Although the big cardinality of the dataset, a need of alternative approaches for
Fig. 6. F1 results are summarized, taking in account the considered classifiers. In orange are shown the results obtained by adopting the bottom-up approach whereas the blue ones shows the result of the ”normal” aproach. dataset formatting to facilitate the learning of the classifiers has emerged. The use of a custom procedure allowed us to find a way to improve the classification.
To show how to use OINOS, here we have performed
the evaluation of classifiers for an application related to the
biomedical field. Nevertheless, such kind of tools are of
great help in many other fields [
The work is an opensource project, available at https://gitlab. com/knizontes/oinos.