Credit cards became one of the most popular payment methods as technology advanced and e-commerce services expanded, resulting in an increase in the volume of banking transactions. Furthermore, the significant increase in the number of frauds necessitates high banking transaction costs. As a result, detecting fraudulent activities has become an intriguing topic, attracting a large number of researchers. We consider the use of class weight-tuning hyperparameters to control the weight of fraudulent and legitimate transactions in this study. We use Bayesian optimization, in particular, to optimize the hyperparameters while taking into account practical issues such as unbalanced data. We propose weighttuning as a pre-process for unbalanced data, as well as using CatBoost and XGBoost to improve the performance of the LightGBM method by taking the voting mechanism into account. Finally, we use deep learning to fine-tune the hyperparameters, particularly our proposed weight-tuning one, in order to improve performance even further. We conducted some experiments to evaluate the proposed methods on real-world data. We use recall-precision metrics in addition to the common ROC-AUC to better cover unbalanced datasets. We use a 5-fold cross-validation method to test CatBoost, LightGBM, and XGBoost separately. Furthermore, the performance of the combined algorithms is evaluated using the majority voting ensemble learning method. According to the results, LightGBM and XGBoost achieve the best level criteria of ROC-AUC = 0.95, precision 0.79, recall 0.80, F1 score 0.79, and MCC 0.79. Also, by using deep learning and the Bayesian optimization method to tune the hyperparameters, we meet the following criteria: ROC-AUC = 0.94, precision = 0.80, recall = 0.82, F1 score = 0.81, and MCC = 0.81. This is a big improvement over the state-of-the-art methods we compared it to.
In recent years, there has been a significant increase in the volume of financial transactions
due to the expansion of financial institutions and the popularity of web-based e-commerce.
Fraudulent transactions have become a growing problem in online banking, and fraud detection
has always been challenging [
Along with credit card development, the pattern of credit card fraud has always been updated.
Fraudsters do their best to make it look legitimate, and credit card fraud has always been
updated. Fraudsters do their best to make it look legitimate. They try to learn how fraud
detection systems work and continue to stimulate these systems, making fraud detection more
complicated. Therefore, researchers are constantly trying to find new ways or improve the
performance of the existing methods [
There are two mechanisms, fraud prevention and fraud detection, that can be exploited
to avoid fraud-related losses. Fraud prevention is a proactive method that stops fraud from
happening in the first place. On the other hand, fraud detection is needed when a fraudster
attempts a fraudulent transaction [
Fraud detection in banking is considered a binary classification problem in which data is
classified as legitimate or fraudulent [
In this paper, we propose an eficient approach for detecting credit card fraud that has been evaluated on publicly available datasets and has used optimized algorithms LightGBM, XGBoost, CatBoost, and logistic regression individually, as well as majority voting combined methods, as well as deep learning and hyperparameter settings. An ideal fraud detection system should detect more fraudulent cases, and the precision of detecting fraudulent cases should be high, i.e., all results should be correctly detected, which will lead to the trust of customers in the bank, and on the other hand, the bank will not sufer losses due to incorrect detection.
The main contributions of this paper are summarized as follows: • We adopt Bayesian optimization for fraud detection and propose to use the weight-tuning hyperparameter to solve the unbalanced data issue as a pre-processing step. • We propose a majority-voting ensemble learning approach to combine CatBoost, XGBoost, and LightGBM and review the efect of the combined methods on the performance of fraud detection on real, unbalanced data. • To better cover the unbalanced datasets, we use recall-precision in addition to the typically used ROC-AUC. We also evaluate the performance using F1_score and MCC metrics. According to the results, the proposed methods outperform the existing and based methods. For evaluations, we use publicly available datasets and also publish the source codes 1 with public access to be used by other researchers.
A number of methods have been proposed by researchers in order to detect and prevent
fraudulent credit card transactions. Halvaiee & Akbari develop a new fraud detection model
called the AIS-based fraud detection model (AFDM). To improve fraud detection accuracy, they
use Immune System Inspired Algorithms (AIRS). According to their paper, their proposed AFDM
improves accuracy by up to 25%, reduces costs by up to 85%, and reduces system response time
by up to 40% compared to basic algorithms [
Randhawa et al., analyze the efectiveness of machine learning algorithms for detecting credit
card fraud. To evaluate the available datasets, they used Naive Bayes, stochastic forest and
decision trees, neural networks, linear regression (LR), and logistic regression, as well as support
vector machine standard models. By applying AdaBoost and majority voting, they propose a
hybrid method. Additionally, they add noise to the data samples in order to evaluate robustness.
On publicly available datasets, they demonstrate that majority voting is efective at detecting
credit card fraud[
In [
To detect fraud in credit card transactions, Altyeb et al., propose an intelligent approach
consisting of a Bayesian-based hyperparameter optimization algorithm for tuning LightGBM
parameters [
Verma and Tyagi investigate machine learning algorithms in order to determine the best
supervised ML-based algorithm for credit card fraud detection in the presence of an imbalanced
1The codes are available at
https://github.com/khadijehHashemi/Fraud-Detection-in-Banking-Data-by-MachineLearning-Techniques
dataset. They evaluate five classification techniques and show that the supervised vector
classifier and logistic regression classifier outperform other algorithms in an imbalanced dataset
[
The proposed framework for credit card fraud detection is presented in Fig.1. We first apply the pre-processing on the dataset and further split the data into two sections: training and testing, followed by performing Bayesian optimization on the training data to find the best hyper-parameters that lead to the improvement of the performance. We use the cross-validation method to obtain performance comparison in an unbalanced set and then examine the algorithms using diferent evaluation metrics, including accuracy, precision, recall, the Matthews correlation coeficient (MCC), the F1-score, and AUC diagrams.
we use a real dataset so that the outcome of the proposed algorithm can be used in practice. We consider a dataset named “credit card” that contains 284,807 records of two days of transactions made by credit card holders in September 2013. There are 492 fraudulent transactions, and the rest of the transactions are legitimate. The positive class (frauds) accounts for 0.172% of all transactions; hence, the dataset is highly imbalanced. the original features and background information about the data are not given due to confidentiality and privacy considerations. PCA yielded the following principal components: 1, 2, 28. The untransformed features with PCA are "time" and "amount." The "Time" column contains the time (in seconds) elapsed between each transaction and the first transaction in the dataset. The feature "Amount" shows the transaction amount. Feature "Class" is the response variable, and it takes the value 1 in case of fraud and 0 otherwise. The summary of the variables and features is presented in Table.1
The total number of fraudulent transactions are significantly lower than the total number of
legitimate transactions, indicating that the data distribution is unbalanced as shown in Table 2.
This data imbalance causes performance issues in machine learning algorithms, and having
a class with the majority of the samples influences the evaluation results[
The “time” feature includes the time (in seconds) elapsed between each transaction and the
ifrst transaction. The features are unknown except for “Time” and “Amount”, and we have
no additional information. Feature selection tries to find a subset of features that improve the
classifier’s performance on efectively detecting credit card fraud [ 17]. The information gain
(IG) method is used to select the most important features that lead to a dimension reduction
of the training data. Information gain functions by extracting similarities between credit card
transactions and then awarding the greatest weight to the most significant features based on
the class of legitimate and fraudulent credit card transactions [
Hyperparameters have a significant efect on the performance of machine learning models. We refer to optimization as the process of finding the best set of hyperparameters that conifgure a machine learning algorithm during its training. In this paper, we use the Bayesian optimization algorithm to tune the hyperparameters that lead to computational time reduction and performance improvement.
This algorithm could not be used for unbalanced data. Therefore, we used hyperparameter class weight to solve the class imbalance prior to applying logistic regression. We show that the ROC-AUC curve cannot be used for the evaluation of unbalanced data and leads to false interpretations.
The LightGBM algorithm is built on the GBDT framework and aims to improve computational
eficiency, particularly on big data prediction problems [ 19]. The high-performance LightGBM
algorithm can quickly handle large amounts of data, and the distributed processing of data
[
Extreme Gradient Boosting (XGBoost) has become a dominant algorithm in the field of applied machine learning. This algorithm is a hybrid technique in which new models are added to fix errors caused by existing models. XGBoost includes parallel computation to construct trees using all the CPUs during training. Instead of traditional stopping criteria (i.e., criterion first), it makes use of the "max depth" parameter and starts tree pruning from the backward direction, which significantly improves the computational performance and speed of XGBoost [ 19]. XGBoost employs a more regularised technique called "formalization" to control over-fitting and achieve better performance [20]. The tuned hyperparameters include learning rate, number of trees, and maximum tree depth, as well as applying weight to classes.
Ensemble learning (EL), which is a type of machine learning, combines several classifiers, minimizes the error of the classifiers, and achieves more reasonable results than a single technique. A voting majority classifier is not a real classifier, but a method that is trained and evaluated in parallel in order to use the diferent features of each algorithm. We can train the data using diferent hybrid algorithms to predict the final output. The final result of the prediction is determined by a majority of votes according to two diferent strategies: hard voting and soft voting. If voting is hard, it uses the predicted class labels to vote for the majority law. Otherwise, if the vote is soft, it predicts the class label based on "Argmax," the sum of the predicted probabilities, which is recommended for a set of well-calibrated classifiers. In this case, the probability vector is calculated on average for each predicted class (for all classifiers). The winning class is the one with the highest value [21, 22].
Deep learning is shown to be a very promising solution to deal with fraud in financial transactions, making the best use of banks’ big data. [23]. In this paper, we use a sequential model, which is a linear stack of layers to construct an artificial neural network model. We use the Relu activation function, and in the last layer, we use “Sigmoid”, since our output is binary. The Sigmoid function generates values in a range of zero and one. The function of the Relu activation function is in many ways similar to the function of our biological neurons. We use kernel-initializer, which defines the method of determining the random weights of the primary Keras layers. To overcome the unbalanced data problem, we consider the ratio of 1 to 4 for the weight of the majority class to the minority class. This causes an increase in the processing speed as well as increasing the eficiency of the model. The size of the input layer is equal to the number of features plus the extracted features. We also remove the "time" feature. To build the Keras model, we optimise the number of layers and neurons, the number of epochs, and the batch size, which leads to an increase in speed. Commonly, batch size is set to 32 or 128. However, our dataset is highly unbalanced, and by choosing the common batch size, there may be no fraud cases in the batch during training. Therefore, our range is chosen so that we can see fraudulent samples in each batch. Also, by choosing a larger batch size, the processing is faster, and we also need less memory. Large epoch sizes can result in either over- or under-fitting. Therefore, selecting the appropriate range for optimization not only increases the eficiency of the algorithm but also reduces the time required to find the optimal points. By performing Bayesian optimization, the number of neurons in the first hidden layer is set to 86, the number of epochs is set to 117, and the batch size is set to 1563.
We use the stratified 5-fold cross-validation method and the boosting algorithms with the Bayesian optimization method to evaluate the performance of the proposed framework. We extract the hyperparameters and evaluate each algorithm individually before using the majority voting method. We examine the algorithms in triple and double precision. The comparison results are presented in Table.3
Most studies in the literature rely on AUC diagrams to evaluate performance. However, as can be seen from the ROC-AUC curve in Fig.3 (a), the value of AUC in severely unbalanced data is not a good evaluation metric. It is influenced by the real positives and considers the negatives irrelevant. According to the ROC-AUC Fig.3, the logistic regression algorithm 0.9583 has the highest number of fraud detection, but it has the lowest value in other criteria. The precision-recall curve is illustrated in Fig.3 (b) and shows the system performance in a more precise manner compared with the ROC-AUC curve. Comparing the precision, recall, and F1-score as well as the MCC, the algorithms used are shown in Fig.3 (c). The evaluation results of the proposed approach using diferent pre-processing and class weight hyperparameter tuning to deal with the problem of data unbalance. In Table 5, it is shown that the proposed methods outperform the intelligence method.
In this paper, we studied the credit card fraud detection problem in real unbalanced datasets. We proposed a machine-learning approach to improve the performance of fraud detection. Our experimental results showed that the proposed LightGBM method improved the fraud detection cases by 50 percent and the F1-score by 20 percent compared with the recently presented method in [17]. For future studies and work, we propose using other hybrid models as well as working specifically in the field of CatBoost by changing more hyperparameters. [16] D. Almhaithawi, A. Jafar, M. Aljnidi, Example-dependent cost-sensitive credit cards fraud detection using smote and bayes minimum risk, SN Applied Sciences 2 (2020) 1–12. [17] J. Cui, C. Yan, C. Wang, Learning transaction cohesiveness for online payment fraud detection, in: The 2nd International Conference on Computing and Data Science, 2021, pp. 1–5. [18] M. Rakhshaninejad, M. Fathian, B. Amiri, N. Yazdanjue, An ensemble-based credit card fraud detection algorithm using an eficient voting strategy, The Computer Journal (2021). [19] W. Liang, S. Luo, G. Zhao, H. Wu, Predicting hard rock pillar stability using gbdt, xgboost, and lightgbm algorithms, Mathematics 8 (2020) 765. [20] S. B. Jabeur, C. Gharib, S. Mefteh-Wali, W. B. Arfi, Catboost model and artificial intelligence techniques for corporate failure prediction, Technological Forecasting and Social Change 166 (2021) 120658. [21] F. N. Khan, A. H. Khan, L. Israt, Credit card fraud prediction and classification using deep neural network and ensemble learning, in: 2020 IEEE Region 10 Symposium (TENSYMP), IEEE, 2020, pp. 114–119. [22] A. Goyal, J. Khiari, Diversity-aware weighted majority vote classifier for imbalanced data, in: 2020 International Joint Conference on Neural Networks (IJCNN), IEEE, 2020, pp. 1–8. [23] A. Roy, J. Sun, R. Mahoney, L. Alonzi, S. Adams, P. Beling, Deep learning detecting fraud in credit card transactions, in: 2018 Systems and Information Engineering Design Symposium (SIEDS), IEEE, 2018, pp. 129–134.