Financial institutions use a variety of methodologies to define their commercial and strategic policies, and a significant role is played by credit risk assessment. In recent years, diferent credit risk assessment services arose, providing Social Lending platforms to connect lenders and borrowers in a direct way without assisting of financial institutions. Despite the pros of these platforms in supporting fundraising process, there are diferent stems from multiple factors including lack of experience of lenders, missing or uncertain information about the borrower's credit history. In order to handle these problems, credit risk assessments of financial transactions are usually modeled as a binary problem based on debt repayment, going to apply Machine Learning (ML) techniques. The paper represents an extended abstract of a recent work, where some of the authors performed a benchmarking among the most used credit risk assessment ML models in the field of predicting whether a loan will be repaid in a P2P platform. The experimental analysis is based on a real dataset of Social Lending (Lending Club), going to evaluate several evaluation metrics including AUC, sensitivity, specificity and explainability of the models.
The recent development of digital financial services has led researchers to pay attention to the
management of credit risk, proposing useful models to reduce such a risk but also to obtain
profits from the investment. Banking risks can arise from diferent factors including: operational
risks, market, credit, and the last one represents 60% of problems for banks [
The main cause of credit risk is the spread of Social Lending (SL) platforms, known as Peer-toPeer (P2P) lending. These platforms allow lenders and borrowers to be interconnected without involving financial institutions; they support borrowers in the fundraising process and allow lending entities to participate. One challenge that needs to be addressed in this context is the credit risk analysis, due to possible non-repayment of loans by borrowers, where risk assessment is calculated through credit scoring.
The credit risk assessment of financial transactions on SL platforms is performed through a
binary classification problem, based on debt repayment [
Additionally, it is important to note that P2P platforms produce large amounts of unlabeled
data so additional analysis is required to support real-time decisions [
Predictive models of credit scoring can be classified into two broad categories [
However, the development of credit risk prediction models is dificult due to diferent factors, including high data size and imbalance and high number of missing values. For these reasons, additional approaches have then been proposed, such as Support Vector Machine (SVM) based semi-supervised approach [12], while [13] has introduced an ensemble Decision Tree model for credit risk assessment on 138 Chinese companies with loss-making corporate earnings. Another ensemble method was developed by Feng et al. (2018)[14], in which classifiers are selected based on performance related to credit scoring. While [15] designed a hybrid model that relies on transductive support vector machine (TSVM) and Dempster-Shafer theory to predict social loan defaults. Finally, [16] has described a combination of diferent classifiers using linear weight ensemble to predict SL default, instead Song et al. (2020)[17] an ensemble of classifiers based on distance-model learning method and adaptive multi-view clustering (DM-ACME).
In this paper, which represents an extended abstract of our previous work [18], we propose a benchmarking for credit risk scoring using the most advanced machine learning (ML) techniques used in the literature , to understand whether a loan will be repaid on a P2P platform. The performance was evaluated using diferent scoring metrics such as Sensitivity, AUC, Specificity. In addition, eXplainable Artificial Intelligence (XAI) approaches were used to obtain a high degree of explainability of the models. The goal is to evaluate both in terms of accuracy performance of the classifiers but also to provide results understandable by domain experts, ensuring transparency in decisions, this is particularly required for credit risk assessment.
The proposed benchmark architecture in Fig.1 stems from the need to be able to ofer support to the risk prediction problem, thus an investor can evaluate potential borrowers within social lending platforms. The main challenge to address is that credit risk assessment is a multidimensional and unbalanced issue because it is based on huge amounts of historical data, including, credit history (obtained by filling out a comprehensive application), bank account status, employment status, etc.
In addition, using all these features increases coverage but decreases accuracy, thus it is essential to apply a feature selection approach. In particular, the proposed architecture that is based on three macro-modules:
2. Classification, 3. Explanation.
The ingestion phase aims at crawling the data from the social lending platforms, cleaning
and filtering the obtained data and performing feature selection based on the chosen classifier.
In details, the data are cleaned by removing features with many missing or null values and
attributes with zero variance from the dataset. After cleaning, several transformations are
applied, such as converting categorical features to numeric and changing date attributes to
numeric values. The second macro-block performs credit prediction, here we have to deal with
a problem of imbalance because usually a user of P2P platforms have a high number of rejected
loans compared to those requested. The classifiers chosen in our architecture are:
Logisticregression, Random Forest and Multi-Layer Perceptron , being the most suitable ones for credit
prediction [
The purpose of our experimentation is to compare diferent classification models, evaluating them according to some metrics (for more details see Section 3.1). The dataset is provided by Lending Club1, a P2P lending platform, in particular we focused on loans disbursed between 2016 and 2017, it consists of 877,956 samples and 151 features, where the most important ones are loan_amount and term.
According to [11] and [
Only the labels ”FullyPaid” or ”Charged of” were considered, since we classified the problem as binary, whether the loan will be repaid or not, this leads to unbalanced data, respectively 0.77% of the samples are fully paid , the remaining 0.23% are unpaid. A 10-cross validation was performed in which the dataset was divided into a training set and a test set with a ratio of 75/25.
Finally, the results obtained were compared against those presented in Namvar et al. (2018)[
The following metrics were used to evaluate and compare the efectiveness of the considered models: Sensitivity (TPR), Specificity (TNR), G-mean, Precision, FP-Rate, Area Under Curve (AUC) . Accuracy (ACC) was not used as an evaluation metric because it does not consider that false positives are more important than false negatives, thus it results in an inaccurate evaluation. Instead, TPR and TNR are suitable because they assess the accuracy of positive and negative samples, respectively. While G-mean is an appropriate metric for assessing the balance of
Classifier RF - RUS LR - ROS LR - SmoteToken Logistic Regression Random Forest MLP classification performances for both majority and minority classes.
In turn, Precision and FP-Rate are useful for understanding how well the model predicts positive and negative classes. Finally, AUC determines the area under the ROC curve, thus it is used to assess the trade-of between the rate of true positives and true negatives in the evaluated model.
The aim of this section is to explain the criterion of improving the data through their cleaning and feature selection. Specifically, all features with missing values greater than 55%, and also those with high standard deviation were removed. Finally, the missing values were replaced with the median of the features, furthermore the nominal features were converted to binary data (more details are reported in [18]).
The classifiers used are Random Forest (RF), Logistic Regresion (LR) and Multi Layer Perceptron
and have been evaluated according to diferent sampling strategies: Under-sampling (RUS, IHT),
Over-sampling (ROS, SMOTE, ADASYN), Hybrid-Method (SMOTE-TOKEN, SMOTE-EN). In
Table 1 we report the best combination between the classifiers and the sampling strategies,
comparing them also against the performance obtained by the classifiers without any strategy,
this last comparison highlights the efectiveness of the latter techniques on the prediction
performance. The experiment decrees that RF-RUS turns out to be the best method for predicting
a borrower’s status in a social lending market.
3.3.1. Comparison with state-of-art results
We compared our results against the best results of Namvar et al. (2018)[
Namvar et al.[
Method
RF -RUS
Song et al.[17] Linear discrimination analysis - SMOTE LR - SmoteToken Logistic regression
Random forest
GBDT Random forest
AdaBoost
Decision tree
Logistic regression Multilayer perceptron
GBDT Random forest
AdaBoost
Decision tree
Logistic regression Multilayer perceptron 0.7000 0.643 3.3.2. Explanation results In this last part of the evaluation, we compare the performance of several XAI tools: LIME, Anchors, SHAP, BEEF, and LORE. In particular, the metrics are based on the Accuracy measure, according to the protocol described in Ribeiro et al. (2016)[21], evaluated on the three best classifiers: Random Forest & Random Subsampling, Logistic Regression & Random Oversampling, and Logistic Regression & Smote-Token. Several explanations were generated, using diferent sets of instances computed with diferent random sampling (10 runs) from the dataset. Analyzing the results in Table 3, LORE is the best because it combines local predictions with the use of counterfactuals for explanation generation, while LIME achieves good results for all three classifiers, this is because the prediction is modeled as a weighted sum and this makes it easy to interpret the prediction generation. SHAP, on the other hand, based on the importance of features, ofers statistically more significant results than LIME, this is given by the use of shap values, whose computational complexity, even if dampened by diferent heuristics, can afect the eficiency of the explanation. Finally, regarding BEEF and Anchors, they can be limited in the expressiveness of the explanation, as noted for Logistic Regression, since they are based on axis-aligned hyper-rectangle and specific rules (called anchors).
Determining the risk prediction score is one of the biggest challenges in finance. The aim of the proposed approach is to support people in their investments, proposing a reference model based Anchors Lime SHAP BEEF LORE on Machine Learning approaches for the prediction of credit risk in social lending platforms, going to manage what are the major criticalities in P2P platforms: the high dimension of data to be analyzed and unbalanced data. The evaluation done on a real dataset demonstrates the goodness of the proposed approach, as well as the fact of being able to provide an explanation for the prediction obtained, which is very significant in the financial field to be able to motivate a positive or negative judgment to provide a loan. Developments of future work may be to consider diferent P2P lending platforms and use additional classification approaches such as Deep Learning or ensemble learning techniques in order to achieve better performance. multi-objective evolutionary algorithms, Expert systems with applications 126 (2019) 144–157. [10] R. Emekter, Y. Tu, B. Jirasakuldech, M. Lu, Evaluating credit risk and loan performance in online peer-to-peer (p2p) lending, Applied Economics 47 (2015) 54–70. [11] M. Malekipirbazari, V. Aksakalli, Risk assessment in social lending via random forests,
Expert Systems with Applications 42 (2015) 4621–4631. [12] Z. Li, Y. Tian, K. Li, F. Zhou, W. Yang, Reject inference in credit scoring using semisupervised support vector machines, Expert Systems with Applications 74 (2017) 105–114. [13] J. Sun, J. Lang, H. Fujita, H. Li, Imbalanced enterprise credit evaluation with dte-sbd: Decision tree ensemble based on smote and bagging with diferentiated sampling rates, Information Sciences 425 (2018) 76–91. [14] X. Feng, Z. Xiao, B. Zhong, J. Qiu, Y. Dong, Dynamic ensemble classification for credit scoring using soft probability, Applied Soft Computing 65 (2018) 139–151. [15] A. Kim, S.-B. Cho, An ensemble semi-supervised learning method for predicting defaults in social lending, Engineering Applications of Artificial Intelligence 81 (2019) 193–199. [16] W. Li, S. Ding, H. Wang, Y. Chen, S. Yang, Heterogeneous ensemble learning with feature engineering for default prediction in peer-to-peer lending in china, World Wide Web 23 (2020) 23–45. [17] Y. Song, Y. Wang, X. Ye, D. Wang, Y. Yin, Y. Wang, Multi-view ensemble learning based on distance-to-model and adaptive clustering for imbalanced credit risk assessment in p2p lending, Information Sciences 525 (2020) 182–204. [18] V. Moscato, A. Picariello, G. Sperlí, A benchmark of machine learning approaches for credit score prediction, Expert Systems with Applications 165 (2021) 113986. [19] V. García, A. Marqués, J. S. Sánchez, On the use of data filtering techniques for credit risk prediction with instance-based models, Expert Systems with Applications 39 (2012) 13267–13276. [20] A. Namvar, M. Naderpour, Handling uncertainty in social lending credit risk prediction with a choquet fuzzy integral model, in: 2018 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), IEEE, 2018, pp. 1–8. [21] M. T. Ribeiro, S. Singh, C. Guestrin, " why should i trust you?" explaining the predictions of any classifier, in: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, 2016, pp. 1135–1144. [22] M. T. Ribeiro, S. Singh, C. Guestrin, Anchors: High-precision model-agnostic explanations, in: Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. [23] S. M. Lundberg, S.-I. Lee, A unified approach to interpreting model predictions, Advances in neural information processing systems 30 (2017). [24] S. Grover, C. Pulice, G. I. Simari, V. Subrahmanian, Beef: Balanced english explanations of forecasts, IEEE Transactions on Computational Social Systems 6 (2019) 350–364. [25] R. Guidotti, A. Monreale, S. Ruggieri, D. Pedreschi, F. Turini, F. Giannotti, Local rule-based explanations of black box decision systems, arXiv preprint arXiv:1805.10820 (2018).