Following the spread of digital channels for everyday activities and electronic payments, huge collections of online transactions are available from financial institutions. These transactions are usually organized as time series, i.e., a time-dependent sequence of tabular data, where each element of the series is a collection of heterogeneous fields (e.g., dates, amounts, categories, etc.). Transactions are usually evaluated by automated or semi-automated procedures to address financial tasks and gain insights into customers' behavior. In the last years, many Trees-based Machine Learning methods (e.g., RandomForest, XGBoost) have been proposed for financial tasks, but they do not fully exploit in an end-to-end pipeline all the information richness of individual transactions, neither they fully model the underling temporal patterns. Instead, Deep Learning approaches have proven to be very efective in modeling complex data by representing them in a semantic latent space. In this paper, inspired by the multi-modal Deep Learning approaches used in Computer Vision and NLP, we propose UniTTab, an end-to-end Deep Learning Transformer model for transactional time series which can uniformly represent heterogeneous time-dependent data in a single embedding. Given the availability of large sets of tabular transactions, UniTTab defines a pre-training self-supervised phase to learn useful representations which can be employed to solve financial tasks such as churn prediction and loan default prediction. A strength of UniTTab is its flexibility since it can be adopted to represent time series of arbitrary length and composed of diferent data types in the fields. The flexibility of our model in solving diferent types of tasks (e.g., detection, classification, regression) and the possibility of varying the length of the input time series, from a few to hundreds of transactions, makes UniTTab a general-purpose Transformer architecture for bank transactions.
gorical features using an attribute-specific embedding using natural language-based “interfaces” between the
which is used as a prefix, concatenated with the actual tabular data and a Transformer. For instance, in LUNA
ifeld value. Schäfl et al. [
To overcome these problems, we propose a custom require learning the long-term dependencies of the data.
Deep Learning architecture based on modern Transform- Furthermore, the learning phase is enriched with new
ers [
2. Related works The current great availability of data together with the
advancement of AI research have opened the possibility
The heterogeneous nature of transactional data and the of developing larger models with more general purposes.
lack of large public annotated datasets, due to privacy This is already a reality in almost every application
reand commercial reasons, make these data extremely dif- garding unstructured data, like text, images and video.
ifcult to be handled by deep neural networks. However, Many general-purpose models have gained fame in
rein recent years some works have started to address these cent years: GPT-3 [16], BERT [17], CLIP [18], DALL-E
challenges. For instance, Padhi et al. [
Another recent work is TabAConvBERT proposed by capabilities even without further training, like the
genShankaranarayana & Runje [
The architecture presented by X. Huang et. al. [
A diferent line of work involves directly or indirectly During the pre-training stage, we train our UniTTab model using the Masked Token pretext task [17]. Some UniTTab pre-training
UniTTab fine-tuning ...
...
Sequence Transformer ... ...
MLP [CLASS]
...
Sequence Transformer ... ...
Field Transformer
Field Transformer
Field Transformer
Field Transformer
enTciomdeing eCnacteogdoinrgy enTciomdeing eCnacteogdoinrgy
Date Time Amount [MASK] ... [MASK] [MASK] Amount Merchant
enTciomdeing eCnacteogdoinrgy enTciomdeing eCnacteogdoinrgy
Date Time Amount Merchant ... Date Time Amount Merchant
of the input features are masked to the model, and the in final embeddings, which are concatenated in a single
model is trained to predict the masked features as a func- embedding vector. This embedding is the representation
tion of the “visible” ones. This method makes it possi- of a single transaction. Then a sequence of these
emble to train a model to automatically learn the seman- beddings, each representing a transaction, is fed to the
tics and to extract relevant information contained in the second Transformer (“Sequence Transformer”). The
sequence of transactions. Specifically, for an input se- Sequence Transformer models the statistic dependencies
quence, we randomly replace a field value with the special between diferent transactions in the sequence and
outsymbol [MASK]. We use a standard replacement proba- puts embedding elements in the latent space. This is the
bility value of 0.15 [
It is important to remark that, within the pre-training to predict the field value. Instead, during fine-tuning, we phase, the training of the model solely relies on input add a class token [CLASS] at the beginning of the transacfeatures of transactions, eliminating the need for labels. tion embedding sequence, and we use the corresponding This way we can use the entire transactional dataset, output embedding to solve the financial tasks. This emeven if it is not fully labeled for the specific downstream bedding can attend to all transaction embeddings in the task. This is the case of the Czech dataset [21] used for sequence, allowing it to exploit all field information of loan default prediction, described in Section 4.1. all transactions.
We develop a custom model called UniTTab, designed to Our model can efectively represent heterogeneous data,
be suited for sequences of heterogeneous transactional encoding both numerical fields (e.g., the amount),
catdata. Borrowing the techniques used in text analysis egorical fields (e.g., the type of transaction), and fields
in BERT or GPT models, we use input time series with with a specific structure (e.g., the date).
variable length. We vary the sequence length from 10 The most common approach to tackle this challenge
to 150 transactions, where each transaction is composed is to reduce all the features to a common representation:
of a fixed number of 10 or 6 fields. As a result, each usually numerical for ensemble of trees or categorical for
time series can vary in length from 100 to 900 items, a deep learning architectures like TabBERT [
Given the data structure, we propose the hierarchical results in a loss of information. For example, it could be architecture shown in Figure 2. The architecture is com- important to know if an amount is precisely 20 euros or posed of two diferent Transformers, and it is trained end- 20.50 to distinguish between a withdrawal and a grocery to-end. The first Transformer (“ Field Transformer”) expense. For this reason we develop a custom representakes as input the features describing a single trans- tation to transform numerical values in the input vector. action, like transaction amount, merchant information, In particular, we represent each numerical value as a featransaction date and time. The features are transformed ture vector obtained by the concatenation of a battery of
large size datasets of transactions: the PKDD’99 Financial Dataset [21] and our Real Bank Account Transaction Dataset (in short, RBAT Dataset). The first dataset is public and is used as a benchmark for predicting loan default. Instead, the second dataset is private and is used to assess how well our model predicts customer churn in comparison to standard industry models. The chosen experiments are binary classification tasks with a large level of unbalance in the statistics of the two target classes.
In all the experiments, the model is rfist pre-trained using self-supervision (Section 3.1) and then fine-tuned on the classification task, in a standard supervised way, using the labeled data.
The loan default prediction is a classification task defined on the PKDD’99 Financial Dataset, which is a public dataset of real transactions from a Czech bank [21]. This dataset is composed of 1M of transactions from 4500 clients. It also includes customer information, but we use only the transactions, each composed of 6 fields (timestamp, amount, type and channel of the transaction).
The dataset presents a large fraction of unlabeled data, in fact most of the accounts don’t have any loans, and they cannot be used for the classification task. This is the perfect example of the potentiality of our model: we perform the pre-training on all the accounts present in the dataset (4500) and then we fine-tune the model only on the labeled ones (478 for training and 204 in test). With such a small number of samples UniTTab has been able to obtain good results, way higher than ensemble of trees, and the possibility to exploit all the data in pre-training is its main advantage.
To evaluate our model’s ability to deal with longer
sequences and variable lengths, we test diferent
sequence lengths of transactions. We define a maximum
length value (ranging from 50 to 150), and for each
diferent frequency functions (depicted with a sine wave account we include the entire sequence of transactions if
symbol in Figure 2). Similar representations are used in it is shorter than the maximum. Instead, if the sequence
NeRFs [22] for 3D synthesis. Conversely, we adopt a tra- exceeds the maximum length, we only consider the most
ditional “category encoding” to represent the categorical recent transactions. It’s important to note that,
features, by using simple embedding neural networks during pre-training the average sequence length is 232
(as used in [
One of the main advantages of using Deep Learning methods over traditional Machine Learning approaches is the possibility to pre-train a large network using a large unsupervised dataset, and then fine-tune the same network on the (usually scarcer) available annotated data of a downstream task. In order to quantify the contribution of the pre-training phase, and to show that this is useful also when the unlabeled dataset is not huge, we use the PKDD’99 Financial Dataset, and we pre-train the models with diferent portions of the pre-training dataset. Specifically, in Figure 3 we indicate the fraction of the pre-training dataset used for each experiment, where zero corresponds to training the models from scratch directly on the (labeled) downstream task data. The results in the figure show that both Deep Learning methods (i.e., TabBERT and UniTTab) significantly benefit from the pre-training phase, even using only a small portion of the unlabeled data (e.g., 0.25). Furthermore, when pretraining is performed, our UniTTab gets a significantly higher F1 score than traditional tree-based models.
tree-based pipeline on a churn rate prediction task. The task is defined on the private RBAT dataset, which is provided by an international bank and is composed of several hundred million real transactions of bank customers.
The churn prediction task is defined as whether or not a customer churns in the next 3 months, given a 6-month history sequence of transactions performed by that customer. The history sequences provided to the models are of variable length, with an average of 192 transactions and up to a maximum of 500 transactions. Each sequence is associated with a binary target that represents the presence of the churn event for that customer in 50 100 150
TabBERT [
F1 score the following 3 months. Initially, our model has been pretrained on a random sample of 1M untargeted accounts, corresponding to approximately 300 million transactions. Then, we evaluate the performance of our model and industry standards using fine-tuning datasets of diferent sizes, ranging from 50K transaction sequences up to 1 million sequences.
Figure 4 shows that our UniTTab model significantly outperforms industry standards for every training dataset size. It also demonstrates the scalability of our model through an increased number of fine-tuning samples: increasing the number of training accounts yields considerably improved AUC on the churn prediction task.
The UniTTab project presented in this paper is a step towards the creation of general-purpose architectures for bank transactions. The empirical results show that our model drastically outperforms both deep learning and standard machine learning based predictive models on diferent benchmarks. We believe that our work and our results can stimulate this research field and the adoption of self-supervised deep learning in banking data.