Page 56
In recent years, the increase in the demand for credit leads the nancial institutions to consider arti cial intelligence and machine learning techniques as a solution to make decisions in a reduced time. These decision support systems reach good results in classifying loan applications into good loans and bad loans. Albeit they su er of some limitations, mainly, they consider that the misclassi cation errors have the same nancial impact.
In this work, we study the performance of ensemble cost sensitive algorithms in reducing the most expensive errors. We apply these techniques on German credit data. By comparing the di erent algorithms, we demonstrate the e ectiveness of cost sensitive ensemble algorithms in determining the potential loan defaulters to reduce the nancial cost.
Copyright c by the paper's authors. Copying permitted for private and academic purposes.
In: Proceedings of the 3rd Edition of the International Conference on Advanced Aspects of Software Engineering (ICAASE18), Constantine, Algeria, 1,2-December-2018, published at http://ceur-ws.org 1
Credit scoring is the process of analyzing credit les, to decide the creditworthiness of an individual. Distinguishing a good applicant for a loan from a bad one is important to cut nancial institution's losses [AEW13]. The use of machine learning tools allows auditors to analyze large amounts of information for evaluating the credit risk in a reasonable time [Yu17].
These algorithms tend to decrease the classi cation error and assume that all misclassi cation's have the same cost. However, the cost for labeling a positive example as negative is di erent from the cost for labeling a negative example as positive. Indeed, approving a bad loan is much more costly than rejecting a potentially good loan [KBC16]. Indeed, if a loan can not full ll its loan obligations this may result in negative impacts on bank pro ts and big nancial losses. However, if a good loan is rejected, it causes lower pro ts losses.
These algorithms tend to decrease the classi cation error and assume that all misclassi cation's have the same cost. However, the cost for labeling a positive example as negative is di erent from the cost for labeling a negative example as positive. Indeed, approving a bad loan is much more costly than rejecting a potentially good loan [KBC16]. Indeed, if a loan can not full ll its loan obligations this may result in negative impacts on bank pro ts and big nancial losses. However, if a good loan is rejected, it causes lower pro ts losses.
On the other hand, credit datasets are highly imbalanced which worsens the situation. Traditional machine learning algorithms tend to maximize accuracy by considering most of the cases as good loans (majority class), thus causing signi cant default loss.
Motivated by the non-uniform cost classi cation problem, the data mining researchers propose new cost-sensitive learning approaches for taking into account the misclassi cation costs or other types of costs such as acquisition cost or computer cost [Tur00, Dom99, Elk01, Mar02]. Some studies have been conducted on the use of cost-sensitive (CS) learning in credit scoring as a CS-boosted tree [XLL17], CS-Neuronal network [AGM+13], CS-decision tree [BAO15] and CS-logistic regression [BAO14].
The objective of this study is to compare the e ectiveness of di erent techniques to assist the loan o cer in screening out potential loan defaulters in the credit environment. The rest of paper is framed as follows: Section 2 describes the used algorithms. Experimental results and discussion are presented in Section 3, Finally, we conclude with a summary of results and directions for future works. 2
In this section, we present the cost sensitive learning principle and the selected algorithms for the evaluation. 2.1
There are several methods to deal with unequal misclassi cation costs. The rst one is to use a learning algorithm that takes into account the costs when building the classi er. The second strategy is to use sampling (oversampling and under-sampling) to alter the class distribution of the training data. In costsensitive classi cation, the misclassi cation cost plays an important role in the learning process. A cost matrix is used to encode the penalty of misclassifying an example from one class as another [Dom99]. Table 1 represents a misclassi cation cost matrix, used to obtain the cost of a false positive (FP), false negative (FN), true positive (TP), and true negative (TN).
Positive Negative
The positive class is the most expensive class and C(i; j) denote the cost of predicting an instance from class i as class j. Usually, C(i; i) have a null or a negative cost and the FN cost is more expensive than a FP cost (C(0; 1) > C(1; 0)). The best evaluation metrics in cost sensitive learning is total cost (see equation 1).
T otal Cost = (F N
CF N ) + (F P
CF P ) (1) The cost-sensitive learning methods can be categorized into two categories: direct and indirect.
In the direct method, the learning algorithm is itself cost-sensitive (CS). The CS learning algorithms use the misclassi cation cost during the learning process. There are several works on cost-sensitive learning algorithms such as ICET [Tur95], an evolutionary algorithm using a misclassi cation cost in the tness function. Many cost sensitive decision tree approaches were proposed [MZ12, Tur95, DHR+06, FCPB07, ZL16]. In [KK98], the authors perform a comparative study of di erent cost-sensitive neural networks. Other researches propose cost sensitive ensemble methods [KW14, KWS14, SKWW07, MSV11, Mar99].
On the other hand, the indirect methods, called Cost-sensitive meta-learning, convert existing costinsensitive learning algorithms into cost-sensitive ones without modifying them. The cost-sensitive metalearning technique, propose two major mechanisms: a pre-process instance sampling or weighting of the training dataset and a threshold adjusting of the output of a cost-insensitive algorithm [Zha08]. In this category, we can cite MetaCost [Dom99] which manipulate the training set labels, Costing [ZLA03], Weighting [Tin02]or Empirical Thresholding [SL06]. 2.2
Classi cation and regression trees (CART) Proposed by Breiman et al. [BFOS84], CART is a binary decision tree. This algorithm processes continuous and categorical attributes and target. CART uses the Gini splitting rule to search the best possible variable to split the node into two child nodes and grow the trees to their maximum size until no splits are possible.
Bootstrap Aggregation (Bagging), is one of the earliest and simplest ensemble algorithms [Bre96]. The learners are tted to bootstrap replicates of the training set by sampling randomly from original set with replacement i.e.: an observation xi may appear multiple times in the sample. After the base learners have been t, the aggregated response is the majority vote. Page 57 Hence, Bagging has no memory, it is easily parallelize (as can be seen in Algorithm 1).
Input: S = ((x1; y1); :::; (xm; ym))), P: the number of classi er to train. for p:=1 to P do
Sp = Bootstrap(S), i.i.d. sampling with replacement from S. hp = TrainClassi er(St).
Add hp = to the ensemble.
end for
Proposed by Schapire [SFBL97, SF12], boosting is a technique for sequentially combining multiple base classi ers whose combined performance is signi cantly better than that of any of the base classi ers. Each base classi er is trained on data that is weighted based on the performance of the previous classi er and each classi er votes to obtain a nal decision.
To generate a cost-sensitive CART algorithm Breiman et al. [BFOS84] modify the class probabilities, P (i) used in the information gain measure. Instead of estimating P (i) by Ni=N , it is weighted by the relative cost.
P (i) = Cij (Ni=N )= X cost(j)(Nj =N ) j The cost of misclassifying an example of class j as class i is : cost(j) = P Cij
It learns the di erent individual classi ers then it uses the available classi ers for a better estimation of the posterior probabilities according to a voting scheme. This approach is applicable regardless of the underlying learning method [Shi15].
This algorithm was proposed by [Dom99]. MetaCost estimates the class probabilities then relabel the training instances to minimize the expected cost. Finally, a new classi er is built on the relabeled dataset.
We test di erent combinations of the former algorithms:
The insensitive cost classi ers: CART, BAGGING of CART, BOOSTING of CART.
Input: S = ((x1; y1); :::; (xm; ym))), P: the number of classi er to train. for p:=1 to P do
Sp = Bootstrap(S), i.i.d. sampling with replacement from S. hp = TrainClassi er(St).
Add hp = to the ensemble. end for for p:=1 to P do
Y^p(w) =Set the prediction with hp . end for According to the proportions observed on the P 0s prediction, we have an estimate of P (Y = yk=X(w)).
Make the prediction which minimizes the cost.
Input: S = ((x1; y1); :::; (xm; ym))), L: cost matrix, H : classi er.
Estimate the class probabilities P (yijxi).
Relabel yi = argmin P j = 1kP (jjxi)L( ; j)8i . T = H(x; y).
Output: T .
One phase Cost classi ers : CS-CART, BAGGING CS-CART, CS-BAGGING CART, MULTI COST CART, BOOSTING CS-CART.
Two phases Cost classi er: CS-BAGGING CSCART 3 3.1
The empirical evaluation was made on the German credit scoring dataset from the UCI Machine Learning Repository. This dataset consists of 20 features and 1000 instances including 700 instances of credit-worthy applicants and 300 instances of insolvent customers who should not have been granted credit. This dataset is provided with a cost matrix,
Insolvent Creditworthy Page 58
Table 3 presents the general results of the nine algorithms. Following the recommendation of [TSTL12], we employ the non-parametric Freidman test to compare the classi ers. The Friedman test ranks the algorithms; to the best performing is the rank of 1, the second best is the rank 2, etc. The last column depicts the statistical test.
A number of conclusions emerge from this table. First, it emphasizes the superiority of ensemble methods compared to the individual classi er. When we consider the classi cation error, the best performances are reached by classical bagging and boosting. However, these algorithms focus on improving the classi cation accuracy at the expense of the minority class. So, they obtain a low sensibility which increases the cost.
On the other hand, a CS-bagging of CS-CART obtains the lowest misclassi cation cost followed by the individual classi er CS-CART. In this case, the statistical improvement is not signi cant (just 1.19). We can consider that this little improvement not worth the the computational cost. However, in some cases a small gain in performances represents a great gain in economical bene ts. Albeit this technique obtains the highest classi cation error.
Figure 1 compares the average results of the different methods. In gure 2 and 3, we can see the results error vs cost and speci city vs sensitivity for each classi er. Considering those values, we can suppose 4
In recent years, the number of insolvent loans has increased due to the nancial crisis. It becomes necessary for banks to nd new methods for the evaluation credit application. Machine learning techniques have been used to perform nancial decision making. However, these methods intended to minimize the misclassi cation error and assume that the di erent errors are equals. The cost sensitive techniques are used to handle the misclassi cation cost in many real world problems.
In this paper, we compare the performance of di erent cost-sensitive and cost-insensitive ensemble algorithms in determining the creditworthiness of an individual. The experiments drew the following conclusions (1) the ensemble approaches obtain better results than individual classi er; (2) the insensitive approaches reached the best classi cation accuracy but since the class distribution is highly imbalanced the minority class (insolvent loan) is less well recognized; (3) the cost sensitive approaches intended to reduce the cost at the expense of the accuracy.
Finally, we found that the cost sensitive bagging algorithm o ers the best trade-o between accuracy and misclassi cation cost. For future research, we aim to use techniques to handle imbalanced datasets and experiment with other cost sensitive algorithms. Page 59 Page 60 [AEW13]