Due to the proliferation of online banking, people are more exposed than ever to attacks. Moreover, frauds are becoming more sophisticated, bypassing the security measures put in place by the financial institutions. In this paper, we propose a novel approach to fraud detection based on Natural Language Processing models. We model the user's spending profile and detect frauds as deviations from it. To do so, we employ the attention mechanism that allows us to model and fully exploit past transactions. Our evaluation on real-world data shows that our model achieves a good balance between precision and recall, outperforming traditional approaches in diferent scenarios.
exploit the advances made in the Natural Language Processing (NLP) field to fully exploit user’s
past transactions to build the user spending profile. This model directly tackles the domain’s
challenges and has shown outstanding performance in many fields [
In this paper, we present a novel approach for fraud detection based on the Transformer
model [
In summary, we make the following contributions: • We present, to the best of our knowledge, the first study on the application of the Transformer model to the fraud detection task. By doing this, we take into account the time dimension and fully exploit the users’ past spending patterns, tackling the concept drift of the user’s profile and the data scarcity issues. • We evaluate our approach on a real-world dataset, showing that it outperforms state-ofthe-art methods in diferent fraudulent scenarios. • We compare the performance of NLP-based generative solutions against NLP-based discriminative approaches deployed in the fraud detection domain.
Most fraud detection approaches define fraud as a deviation from the normal spending pattern. However, frauds are dificult to detect due to the lack of data, the concept drift of spending profiles, and the time dimension. In fact, financial datasets are hard to obtain due to privacy concerns. However, thanks to the collaboration with a large Italian bank, we were able to work on a real-world dataset. Additionally, frauds are rare by definition and mixed with legitimate transactions. The imbalance between classes seriously hurts the performance of detection models. Users, as well as fraudsters, behave diferently as time goes by. Consequently, the temporal dimension is crucial for achieving good performance.
Related Work. Existing approaches can be categorized based on how they model normal user’s behavior [12]: local models are user-centric and aggregate features by users; global models are system-centric and try to model the behavior of the system.
Local models usually rely on neural networks designed to deal with sequences of data.
Among them, LSTM and GRU based solutions are the more popular options [13, 14, 15]. The
main shortcoming of these solutions is the need for a fixed-length input and the LSTM cells’
bottleneck. This problem hinders the processing of both users with very few transactions and
the ones with a large amount of them. Instead, our solution can process users who perform
from 1 transaction up to 1024 thanks to the attention-based mechanism. Fraudmemory [8] adds
attention on top of the LSTM output. The model outperforms state-of-the-art solutions in terms
of Precision, Recall, and AUC. Our solution uses attention-based mechanisms, but it applies
them directly to the input to mitigate the the bottleneck for the flow of information inside the
neural networks [
Global models usually consist of clustering and are based on the probability distribution of the data, which is later used to spot anomalies. Among the most known techniques, there are k-NN [18, 19]. Although simple, k-NN performs well compared to more complex approaches, according to Campos et al. [20]. OC-SVM [21] is a SVM modified to find a separation plane that englobes all the legitimate samples. Frauds will fall outside this plane, and therefore, they will be detected. Another promising unsupervised technique is OC-NN [22, 23], which partially solves the Autoencoders’ problem and can extract a rich representation of the input optimized for fraud detection. Although these models show promising results [12], the training times grow exponentially with the input dimension. Thanks to the use of the attention mechanism, our model does not sufer from this issue.
Banksealer [24, 25, 26] is a semi-supervised model that builds a local, global, and temporal profile using methods with a well-known statistical meaning, which adds explainability to the ifnal result. In a subsequent works [ 27, 28], the temporal profile is improved with the application of signal processing techniques to exploit the end user’s recurrent vs. non-recurrent spending pattern, and the authors exploit analyst feedback to self-tune and improve Banksealer’s detection performance using a multi-objective genetic algorithm.
Research Goal. The objective of this work is to exploit the advances made in the NLP field to fully exploit user’s past transactions to build the user spending profile. To do so, we exploit the transformer model and the attention mechanism to tackle the domain’s challenges directly. The transformed model is designed to deal with sequences of data and time-series, while the attention mechanism over the user’s past transactions has been proven to be robust against concept drift [8].
Historical Transactions
(n,m)
Our approach is based on the Transformer model [
In this work, we exploit the similarities between human language and transactions, focusing
on their temporal dependency. In particular, we exploit this architecture’s modeling capacities
to model users’ behavior in terms of transactions: Instead of words and sentences, we have
transactions and user records. Consequently, instead of predicting the next word, we exploit
the transformer model to predict the next transaction in the sequence belonging to the user’s
spending pattern. Significant changes have been made to adapt the Transformer model to the
fraud detection domain’s particularities. More formally, we train the Transformer model to
predict the next transaction given the user’s past transactions. The model takes the last 1024
(0, . . . , 1023) transactions and will output a representation of the next transaction ′. We select
1024 as input size since it is the value that in our experimental evaluation obtained the best
trade-of between performance and computation requirements. This parameter can be adjusted
as needed, but it must be a power of two to allow eficient computation [
In Figure 1 we present an overview of the proposed solution. The model consists of 47 layers arranged in diferent blocks depending on their function. All the neural network layers sum a total of 1961358 weights that are optimized with the Adam optimizer. The main building blocks are the Embedding, the Attention mechanism (implemented with Multihead attetention and Position-wise feed-forward network), the Predictor, the Dissimilarity Score, and the Classifier.
Natural language models rely on an embedding layer to translate words to numbers, usually called Word2Vec. The objective is to map the one-hot encoded-word, a sparse high dimensional space, to a reduced dense space. The Embedding allows knowing if two words are similar or not. To do so, Transformer models are trained on large vocabularies, which are preprocessed. There are several algorithms like BPE [29], WordPiece [30] or SentencePiece [30]. The idea is to break down words into smaller pieces. In the case of transactions, we build the vocabulary by considering each transaction’s feature as a word of a diferent vocabulary. We avoid building a vocabulary for each transaction since it would have made the Embedding not scalable. We use the concept of Embedding introduced by Tomás Mikolov [31]. These layers transform the sparse one-hot encoded transactions into a dense, fewer dimensional space. The resulting vectors share an interesting property: similar elements in terms of meaning are mapped closer. This property helps the model to focus on the relevant elements. For instance, if the model is interested in transactions with amounts in the 5th bin, the search will also return transactions belonging to bins 4th and 6th as they are "close in meaning". There are diferent types of inputs. Therefore, a diferent embedding is used for each one. The last layer consists of a dense layer that projects the input in a higher space. The concept is similar to the one used in the Electra model [11]. Another issue to keep into consideration is Positional Encoding. The Transformer models need the original position of the encoded input, which is lost during the attention mechanism’s application. Existing techniques involve coding a cosine and sine signal from the input, marking the position of each word in the sentence. This technique assumes that all the input elements are equidistant, as in words in a sentence, but transactions are not. Instead of using positions, timestamps are used. Similar to the usual Positional Encoding, cosines and sines are used to encode the information.
The Attention mechanisms are the core of our approach. They were introduced by Bahdanau [32]
and Luong [33]. They exploit the similarity extracted from the embeddings to compute the
dot product. If the vectors of two similar words are similar, the dot product between them
will be higher. The attention mechanism allows the model to search the input for useful
information. Thanks to attention, the model can deal with long sequences of inputs. The
implementation is similar to the one presented in the original Transformer paper [
Multihead Attention Mechanism. The matrix calculated in Equation 1 indicates which positions of the input are more relevant given the query .
(, ) = (1) ︂( )︂ √ Attention is later multiplied by the value , as shown in Equation 2. The result ′ contains the more relevant input given , , and . It worth noticing that ′ will have the same dimensions (, , ℎ/)
(, , ℎ/) Value dense
Query dense Head join (, ℎ) (, , ℎ/) Attention (, , ℎ/) Head split (, ℎ) of . In fact, in Transformer-alike models, = .
′ = (, ) * (2)
There are diferent types of attention depending on how and are calculated. In this case, all the blocks use self-attention except for the last one. Self-attention is the case in which and are obtained, applying a linear transformation to the input. It is called self-attention because each input position pays attention to the other positions of the input. In this way, each input is enriched with the context around them. The output has the same dimensions as the input. The last layer of the model consists of cross-attention. Instead of getting from all the input, only the last position is used. This forces ′ to have the same dimension of , which is one transaction’s dimension.
Figure 2 shows the attention layer. This layer is called Multihead attention. Instead of using only one head, several are used. Each head performs the Equations 1 and 2 on diferent parts of the input. Therefore, the model can pay attention to several positions at once with the same computational cost.
Position-wise Feed-Forward Network. All the dense layers used in the model are linear except the ones used in this layer. The Point-wise feed-forward network adds non-linearity to the model. This operation is Position-wise since it is a non-linear transformation applied to each input position with the same parameters.
Some adjustments are needed to convert a predictive model to a generative one. The model
outputs the probability vector for each feature. With that, it is easy to get the probability
of a given transaction. Then, the anomaly score is obtained by applying the − (· ) to the
probability [
Dissimilarity Score. This layer receives in input the prediction from the model and the
current transaction and generates as output the probability of the current transaction given the
prediction. The prediction given by the model consists of a vector of probabilities for each feature.
Lastly, − (· ) is computed for each feature, yielding a dissimilarity score. This approach is
similar to the one taken by Brown et al. [
Classifier. The classifier can be seen as a meta-learner. Its function is to weigh the dissimilarity scores of each feature to get a proper classification of frauds. It is a classification problem with two classes. Thus, Binary Crossentronpy is used.
We demonstrate the efectiveness of the model in diferent scenarios and against state-of-the-art approaches. The experiments are divided depending on the attacker’s knowledge [36, 37]. Dataset. The dataset used to validate our model contains the transactions of a large Italian banking group. The transactions belong to two periods. One goes from December 2012 to September 2013, and the other goes from October 2014 to February 2015. This accounts for a total of 1043478, comprising 6195 diferent users. A detailed analysis of the data can be found in [27, 25]. Each transaction has 31 features, of which only 9 can be used due to the anonymization process. The proposed model takes as input 6 of them: the amount, the hour, the day of the month, the telecommunications operator from which the transaction is issued, and information about the destination IBAN.
Experimental Setup. First, we preprocess the dataset. Then, we split the dataset into the training set to train the model, the validation test to assess the model’s performance at each set of the training, and the test set to get the final results of the model. The test set is the only one used to create the diferent scenarios and injected with synthetic frauds. We refer the reader to Appendix A for the description of the metrics used.
In the black box attacks scenarios the attacker does not have prior information of the system
he or she tries to attack. We consider four diferent attacking strategies that model real-life
situations: Stealing, hijacking, persistent, and Mix. The Stealing scenario simulates a phishing
attack in which the user’s credentials are stolen. The amount transferred is very high. The
connection can be originated from a national or foreign IP. The Hijacking scenario simulates
Precision
a Man-in-the-browser attack. The connection details are legitimate. The amount transfer is
high. The transfer happens no later than ten minutes from a legitimate one. The Persistent
scenario simulates the infection of a banking Trojan [
Table 1 compares the proposed model against the baselines algorithms. The proposed model is better in almost all scenarios. The lower performance of baseline algorithms is due to concept drift. The overall low FPs, demonstrate how the proposed approach can correctly model (i.e., it does not negatively impact) the user’s "spending pattern". Traditional algorithms have a hard time detecting frauds that have not been seen before. The baseline algorithms’ performance is higher in the scenarios that are more similar to the training set and lower in the most complex scenarios as the Persistent or the Mix scenario.
Figure 3 shows the ROC curves of each model. All the models have been trained in identical conditions, i.e., using the same dataset. Regarding the persistent (a) and mix (b) scenarios, the proposed model is better and presents an overall lower FPR. The curve is similar in both cases due to the persistent frauds. Also, it is possible to distinguish two groups of frauds in these scenarios. The ones easy to detect, which correspond to the first ramp, and the hard ones belong to the second ramp. In the stealing (c) and hijacking (d) scenarios, almost all the models perform similarly. The proposed model works better than baseline algorithms for very low values of FPR. Because of that, the proposed model has a higher AUROC.
Grey box attacks consist of attacks performed by an attacker with information about the target system, as described in [36]. The attacker has acquired (e.g., thanks to a malware infection or leak) all the transactions from December 2012 to September 2013. The attacker uses the available data to train a Random Forest classifier that is treated as an oracle. The attacker will try the fraud against the oracle before injecting it into our model. If the oracle flags the transaction as fraud, the attacker will change the transaction and will try again. Table 2 summarizes the result obtained for each model in the grey box scenarios. The proposed model outperforms by far the baseline algorithms. As expected, the Random Forest is the worse since it is used as an oracle by the attacker. The results also show the benefits of exploiting diferent approaches to fraud detection. The proposed model sufers less from the grey box attack because it is based on user modeling, while the oracle and baseline algorithms rely on finding a separation plane between frauds and legitimate transactions.
The proposed model is generative: It generates the expected transaction probabilities, which are then used to detect fraud. However, there are also discriminative models: they are trained to discriminate between frauds and legitimate transactions. Random Forest or OC-SVM are examples of discriminative models. Following the architecture of the generator, a discriminative model is proposed here. The diferences between the two models are in the last layers of the model and the training. The discriminative model does not have the Predictor nor the Dissimilarity score shown in Figure 1. Regarding training, the generator is trained for prediction, while the discriminator is trained for classification. The discriminative model uses crossattention instead of self-attention and uses the incoming transaction as a query to search in the past transactions. The results are reported in Table 3. Overall, the generative approach performs better than the discriminative one.
Banking information is not public due to obvious reasons. Due to the lack of standard databases and benchmarks, it is not easy to compare diferent models. Therefore, we have developed an ensemble model composed by LSTM, Random Forest, and XGBoost based on the model proposed by Jurgovsky et al. [13]. To combine the three models’ output, we compute the Cumulative Distribution Function of the exponential distribution applied to each model output y, which gives the probability of the input sample. Then, the ensemble gives the final decision to the model that has the maximum distance between and the mean CDF value seen in training. The ensemble improves the performance of each of the individual models. The improvement over the LSTM model is 22% in terms of AUROC. We used the same dataset to train both models. To provide an accurate comparison, the Random Forest’s performance is set as a baseline, and the models are compared in terms of improvement. Table 4 summarizes the results for the mix scenarios, where all types of frauds are considered. The improvements of both models over the baseline are close. Overall, the proposed model performs slightly better than the ensemble one. It is also important to notice that the proposed model can deal with users with very few transactions, while the LSTM-based ensemble needs at least 50 transactions per user.
This paper presented a novel approach for fraud detection based on the Transformer model,
a state-of-the-art technique in the Natural Language Processing field. Besides demonstrating
the feasibility of applying NLP techniques to model the user’s spending behavior, we showed
how the proposed model overcame the domain’s limitations and achieved better performance
than state-of-the-art algorithms in complex scenarios and against adversarial attacks. Future
works will work in the direction of providing explainability to the proposed framework, which
is of paramount importance in the fraud detection domain. In light of the results obtained, we
deem that the future of banking fraud detection is standardization and transfer learning. The
ifeld needs standard benchmarks like ImageNet [
URL: https://slatestarcodex.com/2020/01/06/a-very-unlikely-chess-game/. [7] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, Language models are unsupervised multitask learners (2019). [8] Y. Kunlin, A memory-enhanced framework for financial fraud detection, in: 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), 2018, pp. 871–874. doi:10.1109/ICMLA.2018.00140. [9] L. Nanni, S. Ghidoni, S. Brahnam, Handcrafted vs. non-handcrafted features for computer vision classification, Pattern Recognition 71 (2017) 158 – 172. URL: http: //www.sciencedirect.com/science/article/pii/S0031320317302224. doi:https://doi.org/ 10.1016/j.patcog.2017.05.025. [10] L. Cai, J. Zhu, H. Zeng, J. Chen, C. Cai, Deep-learned and hand-crafted features fusion network for pedestrian gender recognition, in: J. Cao, E. Cambria, A. Lendasse, Y. Miche, C. M. Vong (Eds.), Proceedings of ELM-2016, Springer International Publishing, Cham, 2018, pp. 207–215. [11] K. Clark, M.-T. Luong, Q. V. Le, C. D. Manning, ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators, arXiv e-prints (2020) arXiv:2003.10555. arXiv:2003.10555. [12] R. Chalapathy, S. Chawla, Deep Learning for Anomaly Detection: A Survey, arXiv e-prints (2019) arXiv:1901.03407. arXiv:1901.03407. [13] J. Jurgovsky, M. Granitzer, K. Ziegler, S. Calabretto, P.-E. Portier, L. He-Guelton, O. Caelen, Sequence classification for credit-card fraud detection, Expert Systems with Applications 100 (2018) 234 – 245. URL: http://www.sciencedirect.com/science/article/pii/ S0957417418300435. doi:https://doi.org/10.1016/j.eswa.2018.01.037. [14] A. Roy, J. Sun, R. Mahoney, L. Alonzi, S. Adams, P. Beling, Deep learning detecting fraud in credit card transactions, in: 2018 Systems and Information Engineering Design Symposium (SIEDS), 2018, pp. 129–134. doi:10.1109/SIEDS.2018.8374722. [15] B. Wiese, C. Omlin, Credit Card Transactions, Fraud Detection, and Machine Learning: Modelling Time with LSTM Recurrent Neural Networks, Springer Berlin Heidelberg, Berlin, Heidelberg, 2009, pp. 231–268. URL: https://doi.org/10.1007/978-3-642-04003-0_10. doi:10.1007/978-3-642-04003-0_10. [16] M. Zamini, G. Montazer, Credit card fraud detection using autoencoder based clustering, in: 2018 9th International Symposium on Telecommunications (IST), 2018, pp. 486–491. doi:10.1109/ISTEL.2018.8661129. [17] K. Veeramachaneni, I. Arnaldo, V. Korrapati, C. Bassias, K. Li, Aiˆ 2: training a big data machine to defend, in: 2016 IEEE 2nd International Conference on Big Data Security on Cloud (BigDataSecurity), IEEE International Conference on High Performance and Smart Computing (HPSC), and IEEE International Conference on Intelligent Data and Security (IDS), IEEE, 2016, pp. 49–54. [18] S. Ramaswamy, R. Rastogi, K. Shim, Eficient algorithms for mining outliers from large data sets, SIGMOD Rec. 29 (2000). URL: https://doi.org/10.1145/335191.335437. doi:10. 1145/335191.335437. [19] F. Angiulli, C. Pizzuti, Fast outlier detection in high dimensional spaces, Proceedings of the Sixth European Conference on the Principles of Data Mining and Knowledge Discovery 2431 (2002) 15–26. doi:10.1007/3-540-45681-3_2. [20] G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent, M. E. Houle, On the evaluation of unsupervised outlier detection: measures, datasets, and an empirical study, 2016. URL: https://doi.org/10.1007/s10618-015-0444-8. doi:10.1007/s10618-015-0444-8. [21] B. Lamrini, A. Gjini, S. Daudin, F. Armando, P. Pratmarty, L. Travé-Massuyès, Anomaly detection using similarity-based one-class svm for network trafic characterization, 2018. [22] R. Chalapathy, A. K. Menon, S. Chawla, Anomaly detection using one-class neural networks, 2019. arXiv:1802.06360. [23] L. Ruf, R. Vandermeulen, N. Görnitz, L. Deecke, S. Siddiqui, A. Binder, E. Müller, M. Kloft,
Deep one-class classification, 2018. [24] M. Carminati, R. Caron, F. Maggi, I. Epifani, S. Zanero, Banksealer: A decision support system for online banking fraud analysis and investigation, Computers & Security 53 (2015) 175 – 186. URL: http://www.sciencedirect.com/science/article/pii/S0167404815000437. doi:https://doi.org/10.1016/j.cose.2015.04.002. [25] M. Carminati, R. Caron, F. Maggi, I. Epifani, S. Zanero, Banksealer: An online banking fraud analysis and decision support system, in: N. Cuppens-Boulahia, F. Cuppens, S. Jajodia, A. Abou El Kalam, T. Sans (Eds.), ICT Systems Security and Privacy Protection, Springer Berlin Heidelberg, Berlin, Heidelberg, 2014, pp. 380–394. [26] M. Carminati, M. Polino, A. Continella, A. Lanzi, F. Maggi, S. Zanero, Security evaluation of a banking fraud analysis system, ACM Transactions on Privacy and Security (TOPS) 21 (2018) 1–31. [27] M. Carminati, A. Baggio, F. Maggi, U. Spagnolini, S. Zanero, Fraudbuster: temporal analysis and detection of advanced financial frauds, in: International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment, Springer, 2018, pp. 211–233. [28] M. Carminati, L. Valentini, S. Zanero, A supervised auto-tuning approach for a banking fraud detection system, in: Cyber Security Cryptography and Machine Learning, CSCML 2017, Springer International Publishing, 2017, pp. 215–233. [29] R. Sennrich, B. Haddow, A. Birch, Neural machine translation of rare words with subword units, 2015. arXiv:1508.07909. [30] M. Schuster, K. Nakajima, Japanese and korean voice search, 2012. [31] T. Mikolov, K. Chen, G. Corrado, J. Dean, Eficient estimation of word representations in vector space, 2013. arXiv:1301.3781. [32] D. Bahdanau, K. Cho, Y. Bengio, Neural machine translation by jointly learning to align and translate, 2014. arXiv:1409.0473. [33] M.-T. Luong, H. Pham, C. D. Manning, Efective approaches to attention-based neural machine translation, 2015. arXiv:1508.04025. [34] J. Rocca, Ensemble methods: bagging, boosting and stacking, https://towardsdatascience.
com/ensemble-methods-bagging-boosting-and-stacking-c9214a10a205, 2019. [35] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, 2014. arXiv:1412.6980. [36] M. Carminati, L. Santini, M. Polino, S. Zanero, Evasion attacks against banking fraud detection systems, in: 23rd International Symposium on Research in Attacks, Intrusions and Defenses ({RAID} 2020), 2020, pp. 285–300. [37] A. Erba, R. Taormina, S. Galelli, M. Pogliani, M. Carminati, S. Zanero, N. O. Tippenhauer,
Fraud Detection is a classification problem in which the classes are very unbalanced. Therefore, the usual classification metrics, such as accuracy, could lead to wrong conclusions. For example, a model predicting that all the transactions are legitimate in a dataset containing 1% of frauds have a 99% of accuracy but will not detect frauds. Hence, we propose the use of the following metrics, more appropriate to assess the quality of a model in an unbalance scenario as fraud detection
Precision measures the number of predicted frauds that are actually frauds. It is similar to the accuracy of the positive class.
Recall measures how many frauds are detected from the total number of frauds. Provides an indication of missed frauds.
=
+ =
+
Both metrics are related. Often, increases in one metric imply decreasing the other.
There are two curves: (3) (4) • Receiver operating characteristic, or ROC, shows the behaviour of the system for diferent thresholds.
• Precision-Recall, or PR, shows the tradeof between precision and recall.
ROC curve relates True Positive Rate with False Positive Rate. Given a desired FPR, the model is better as higher the TPR is. A common metric to measure the quality of the curves is the area under the curve or AUC. Higher values indicate better models.
Also called phi coeficient, Matthews correlation coeficient measures the correlation between the observed and predicted classification.
As shown in Equation A.0.3, MCC takes into account all the metrics given by the confusion
matrix. It is regarded as one of the most reliable statistics for imbalance problems [
* − * = √︀( + )( + )( + )( + ) A.0.4. F-score F-score is a statistical accuracy measure. It is calculated from the Precision and Recall, as shown in Equation 6.
* * ( 2 * ) + is a weighting parameter. In our case, we use the 1 score, which is the same as the harmonic median between Precision and Recall.
The False Positive Rate, also known as fall-out, is the ratio between the number of misclassified negative samples and all negatives samples. Equation 7 shows its calculation, FP is the number of False Positive, and TN the number of True Negatives.
=
+ (5) (6)