The success of Artificial Intelligence (AI) in diferent research and application areas has increased the interest in adopting Deep Learning techniques also in the financial field. Particularly interesting is the case of financial transactional data, which represent one of the most valuable sources of information for banks and other financial institutes. However, the heterogeneity of the data, composed of both numerical and categorical attributes, makes the use of standard Deep Learning methods dificult. In this paper, we present UniTTAB, a Transformer network for transactional time series, which can uniformly represent heterogeneous time-dependent data, and which is trained on a very large scale of real transactional data. As far as we know, the dataset we used for training is the largest real bank transactions dataset used for Deep Learning methods in this field, being all the other common datasets either much smaller or synthetically generated. The use of this very large real training dataset, makes our UniTTAB the first foundation model for transactional data.
Transactional data, Deep Learning for finance, fraud detection, financial predictions
tunities, e.g., in asset management, credit intermediation and finance data analysis, and on the related risks, e.g., lack of explainability, learning bias, etc. Also Europe and
deed “AI for banking, finance and insurance”. This is also
a topic of interest for the new large National research
project on foundational AI “FAIR” [
taxonomy in four main impact areas from the data point of view focuses on: • Client data: Client intermediation and Customer engagement;
used on finance document repositories for classification, search and retrieval.
In the third category, based on finance time-series,
machine learning has been often applied with success,
e.g. for macroeconomic analysis [
Conversely, the adoption of deep learning for
transactional bank data is still under-explored. So far, these data
based expert systems) and/or with traditional machine
learning approaches (e.g., SVMs or gradient boosted de- categorical and temporal. On the other hand, thanks to
cision trees [
The UniTTab Italian project [
The main challenges addressed by UniTTab depend on
the aspects of these data, briefly outlined in the following:
Padhi et al. [
Another recent work is TabAConvBERT, proposed by
Shankaranarayana & Runje (2021) [
The architecture presented by X. Huang et. al. [
Ours proposal difers from the aforementioned works in diferent aspects. On the one hand, we deal with all the variability dimensions of the problem: numerical, • Tabular data, usually collected from diferent sources (e.g., separate databases) thus they require an intensive pre-processing for data cleaning and interoperability. • Time dependence. Transactional data represent a special case of time-series with non-regular frequency: bank customers carry out a variable number of transactions per year, ranging from very few transactions up to several thousand transactions per year. • Heterogeneous data. Transactions have not homogeneous fields: some of them are numerical (e.g. the amount), some categorical (e.g. the type of transaction), some textual (e.g. the bank transfer description) or with a specific structure (e.g.
the date). • Multiple-structure data. The transactions of a client account have a diferent field-structure according with the type of transaction (e.g. a POS, a credit card, an ATM or a bank transfer). • Correlated data. The transaction fields are often correlated to each other in the same time series (e.g. in periodical payments) and among diferent time series: each client can own diferent bank products, diferent accounts, and some accounts have diferent owners (join accounts). Finally, some transactions are correlated with external =
Transactional multi-modal Multi-structure time-series True Transactions t1
but unknown conditions (e.g. holiday times or In the UniTTAB project we explore this trend by rethe lockdown in the pandemic period). designing attentive and generative models, i.e. Transformers. This allowed us to deal with the heterogeneous nature of transactional data: tabular time series with 4. The architecture multiple-structure and multimodal fields. The designed architecture is resumed in Figure 1.
Given the previously discussed challenges of data vari- Borrowing the techniques used in text analysis in BERT
ability, quantity and heterogeneity, deep learning for or GPT models, we used input time series with variable
transactional data has been largely underexplored, with length. We varied the sequence length from 50 to 150
only a few public experiments and a small number of transactions, where each transaction is composed of a
private institutions and banking research centers. One ifxed number of 10 fields. As a result, each time series
interesting recent position paper (J.P Morgan 2021 [
In non-financial AI, state-of-the-art models are usually tecture shown in Figure 1. First, we endow a field-level
based on Transformer architectures, usually trained us- transformer, which encode individual transactions into
ing self-supervised learning (e.g., using “masked word” embeddings. Then these embeddings are fed into the
prediction tasks or through generative or “contrastive second-level transformer, that processes the time-series
learning”). Initially defined as language models in the to encode them as a single element in the latent space.
ifeld of NLP, they are now common also in other AI areas This is the foundation latent space where the
represensuch as Computer Vision, making Transformer networks tation can be potentially exploited for many tasks, such
the basic paradigm for contemporary AI. Specifically, as Classification (e.g., to classify the client behavior),
De“Foundational Models” are typically large Transformers tection (e.g., to detect anomalies, frauds, etc.), Prediction
pre-trained on huge datasets (e.g., the Wikipedia docu- (e.g., to predict product churn in next few months). As
ments, or billions of Web-collected images). Their goal shown in Figure 1, this model can also be used for a
genis to create compact, intermediate representations of the erative task. For instance generating time series data
input in a latent space, useful for diferent tasks, such has the advantage of preserving the content but also the
as classification, generation, recognition, image segmen- privacy of the client.
tation, anomaly detection, etc. Examples are BERT [
detection and prediction, are described in UniTTaB
proposed by Simone Luetto et al. [
6. Conclusions
UNIMORE, presented in this paper, is a first step towards the creation of foundation models for transactional time series data. The empirical results show that our model drastically outperforms both deep learning and standard machine learning based predictive models on diferent benchmarks. We believe that our work and our results can stimulate this research field and the adoption of selfsupervised deep learning in banking data.