Feature selection methods, as a preprocessing step to machine learning, are effective in reducing dimensionality, removing irrelevant data, increasing learning accuracy, and improving result comprehensibility. However, the recent increase of dimensionality of data poses a severe challenge to many existing feature selection methods with respect to the efficiency and effectiveness. In this work, we introduce a novel concept, relevant feature selection based on information gathered from decision rule models. A new measure for a feature rank based on the feature frequency and rule quality is additionally defined. The efficiency and effectiveness of our method is demonstrated through exemplary use of five real-world datasets. Six different classification algorithms were used to measure the quality of learning models built on original features and on selected features.
In the era of the acquisition of vast amounts of data, different domain information databases, efficient analysis and retrieval of regularity have become an extremely important task. The issue of classification and object recognition is applied in many fields of human activity. Data mining is fraught with many aspects which hinder it, like a very large number of observations, too many attributes, the insignificance of the part of variables for the classification process, mutual interdependence of conditional variables, the simultaneous presence of variables with different types, the presence of undefined values of variables, the presence of erroneous values of the variables, uneven distribution of categories for the target variable. Thus, the development of efficient methods for significant feature selection is valid.
Feature selection (FS) methods are frequently used as a preprocessing step to machine learning experiments. An FS method can be defined as a process of choosing a subset of original features so that the feature space is optimally reduced according to a certain evaluation criterion. Feature selection has been a fruitful field of research and development since 1970’s and it has been proven to be effective in removing irrelevant features, increasing efficiency in learning tasks, improving learning performance like predictive accuracy, and enhancing comprehensibility of the learned results [1].
The feature selection methods are typically divided into three classes based on how
they combine the selection algorithm and the model building: filter, wrapper and
embedded FS methods. Filter methods select features with respect to the model. They are
based only on general features like the correlation with the variable to be predicted.
These methods select only the most interesting variables. Then, a selected subset will
be a part of the classification model. Such methods are effective in computation time
and robust to overfitting [
Kohavi and John [1] observed that there are several definitions of relevance that may be contradictory and misleading. They proposed two degrees of relevance (strong and weak) that are required to encompass all notions usually associated with this term. In their approach the relevance is defined in the absolute terms, with the help of the ideal Bayes classifier. In this context, a feature X is strongly relevant when removal of X alone from the data always results in deterioration of the prediction accuracy of the ideal Bayes classifier. In turn, a feature X is weakly relevant if it is not strongly relevant and there exists a subset of features S, such that the performance of the ideal Bayes classifier on S is worse than the performance on S ∪ {X}. A feature is irrelevant if it is neither strongly nor weakly relevant.
Nilsson et al. [
There are two important differences between these problems. The first one is
detection of attributes with low importance (ARFS) [
In our research, we used the contrast variable concept to distinguish between
relevant and irrelevant features [
During experiments the following general procedure was applied: 1. Step 1. Selection of dataset and features for investigation.
(a) Application of a set of ranking measures to calculate importance for each feature: i. With set of contrast features.
ii. Without contrast features.
(b) Definition (selection) of the most important feature subset. 2. Step 2. Application of different machine learning algorithms for classification of unseen objects using the 10-fold cross validation method: (a) Using all original features.
(b) Using only selected, important features. 3. Step 3. Comparison of gathered results using different evaluation measures.
In the first step, a dataset as well as a feature for investigation were defined. Then, different ranking measures were applied to estimate importance of each feature. In order to check specificity of the feature selection, the dataset was extended by adding contrast variables. It means that each original variable was duplicated and its values were randomly permuted between all objects. Hence, a set of non-informative by design shadow variables was added to original variables. The variables that were selected as important more significantly than random, were examined further, using different tests. To define the level of feature importance, six well-known ranking measures were applied: ReliefF, Information Gain, Gain Ratio, Gini Index, SVM weight, and RandomForest. Additionally, our new measure, called RQualityFS, was introduced. It is based on the frequency of presence of different feature in a rule model generated from an original dataset and it also takes into consideration the quality of the rules in which this feature occurs. Rank quality of the i-th feature could be presented as follow:
n QAi = X QRj {Ai}
j=1 QRj =
Ecorr
Ecorr + Eincorr where n is a number of rules inside the model, QRj defines the classification quality of the rule Rj and {Ai} describes the presence of the i-th attribute, usually it is equal to 1 (the feature occurred) or to 0 (the feature did not occur).
In turn, the quality of the rule is defined as follows: (1) (2) where Ecorr depicts a number of correctly matched learning objects by the j-th rule and Eincorr depicts a number of incorrectly matched learning objects by this rule.
During the second step, a test probing the importance of variables was performed
by analyzing the influence of variables used for model building on the prediction
quality. Six different machine learning algorithms were applied to build different predictors
for the original set of features and for selected features: Classification Tree (CT),
Random Forest (RF), CN2 decision rules algorithm (CN2), Naive Bayes (NB), k-Nearest
Neighbors (kNN), and Support Vector Machine (SVM). During this step, a 10-fold cross
validation paradigm was used. Ten known evaluation measures were uti-lized in each
predictor: Classification Accuracy (CA), Sensitivity, Specificity, Area Under ROC curve
(AUC), Information Score (IS), F1 score (F1), Precision, Brier measure, Matthew
Coefficient Correlation (MCC) parameter, and finally Informadness (Inform.) ratio [
Our initial investigations focus on applying the developed algorithm on several
re-alworld datasets. Five datasets have been used during experiments. Four of them are
gathered from the UCI ML repository, while the fifth set has been developed earlier by the
authors [
The second step of the experiment was devoted to evaluation of prediction of the quality of utilized machine learning algorithms described in Section 2. During this step, six different algorithms were applied using the 10-fold cross validation method. The average results for the Breast cancer dataset are shown in Figure 1. This procedure was utilized for two specified sets:
Dataset
0.82 0.79 0.93 0.94 1.32 0.81 0.84 0.27 0.75 0.76 0.70 0.90 0.92 1.09 0.72 0.78 0.34 0.64 0.65 0.56 0.80 0.78 0.69 0.59 0.64 0.50 0.42 0.63 0.54 0.79 0.77 0.52 0.62 0.63 0.51 0.43
Dataset
0.81 0.82 0.93 0.94 1.40 0.82 0.81 0.29 0.75 0.77 0.75 0.91 0.92 1.26 0.75 0.76 0.34 0.67 0.64 0.58 0.80 0.79 0.78 0.61 0.59 0.51 0.39 0.62 0.55 0.79 0.76 0.65 0.58 0.57 0.54 0.36