Transactional data are time-dependent collections of financial transactions. For instance, a bank account can be seen as a time series of transactions, each composed of a tabular data entry with fields specifying the transaction amount, the transaction operation and the receiver type (see Figure 1). These data can be used as training data for different Machine Learning approaches, in a variety of tasks. Some examples are:
• Customer Value Management, to support marketing or commercial actions, for instance via the creation of tailored offers; • Credit and Liquidity Risk, to assess the initial risk, and to detect early risk signals, for instance, represented by changes in expense patterns or in the regularity of incomes; • Fraud detection and Anti Money Laundering, to identify potential malicious behaviors.
For these purposes, transactional data have been so far processed with symbolic AI (e.g., rule-based expert systems) and/or with ensembles of trees-based machine learning models. RandomForest [1], LightGBM [2], XG-Boost [3] and CatBoost [4] are the most frequently used. Despite the success of Deep Learning methods in other application areas (e.g., Natural Language Processing and Computer Vision), trees-based models seemed to outperform deep learning models on most of the tabular datasets [5]. These datasets are typically composed of tens to hundreds of features and thousands to hundreds of thousands of samples. However, the size of transactional datasets is growing rapidly, now exceeding millions of transactions in some cases. Since the performance of deeplearning models improves with dataset size, tree-based models are the best choice only for small and mediumsize datasets [6]. In addition, the use of tree-based models for transactional data is limited to constructing simple aggregated features, such as calculating the average spending over recent months or determining the total income for the past year. This approach has clear limitations in fully harnessing the precise and timely information that transactional data encapsulates.
Recently, various deep learning networks have been developed for heterogeneous data, mostly for tabular datasets. Lyu et al. [7] combine different modules and can represent both numerical and categorical features. Borisov et al. [8] use a distillation approach to map decision trees, trained on heterogeneous tabular data, onto homogeneous vectors. Huang et al. [9] represent cate- gorical features using an attribute-specific embedding which is used as a prefix, concatenated with the actual field value. Schäfl et al. [10] use a non-parametric representation of the training data, which reminds the use of external networks in Transformers [11]. However, these approaches do not model the temporal dynamics: each row in the table is an individual sample. A trivial solution could be to concatenate multiple rows into single samples, but none of the previous work has demonstrated that their architecture can model more than hundreds of fields as an individual sample.
To overcome these problems, we propose a custom Deep Learning architecture based on modern Transformers [12], which can uniformly represent heterogeneous time-dependent data, and which is trained on a largescale transactional dataset. We call our model UniT-Tab (Unified Transformer for Time-Dependent Heterogeneous Tabular Data), and we show that it consistently outperforms state-of-the-art approaches based on both Deep Learning and standard Machine Learning techniques.
The heterogeneous nature of transactional data and the lack of large public annotated datasets, due to privacy and commercial reasons, make these data extremely difficult to be handled by deep neural networks. However, in recent years some works have started to address these challenges. For instance, Padhi et al. [13] proposed one of the first deep learning architectures for heterogeneous time series (TabBERT). As a solution to data heterogeneity, the authors quantize continuous attributes so that each field is defined on its finite vocabulary.
Another recent work is TabAConvBERT proposed by Shankaranarayana & Runje [14]. They present an architecture that can deal with both categorical inputs (by using an embedding neural network) and numerical inputs (by using a shallow neural network).
The architecture presented by X. Huang et. al. [9] provides a solution to data heterogeneity, but it cannot handle the temporal component of the data and therefore is unable to solve tasks involving transaction sequences.
A different line of work involves directly or indirectly using natural language-based "interfaces" between the tabular data and a Transformer. For instance, in LUNA [15], numerical values are represented as an atomic natural language string. Our proposal differs from the aforementioned works in different aspects. On the one hand, we deal with all the variability dimensions of the problem: numerical, categorical and temporal. On the other hand, we train our model using arbitrary length and long-range time series, which can include up to 150 transactions per sample. As a result, it is possible to deal with transactional tasks that require learning the long-term dependencies of the data. Furthermore, the learning phase is enriched with new custom masking techniques, which allow all related fields to be masked simultaneously, making the initial generalpurpose training more challenging for our model.
The current great availability of data together with the advancement of AI research have opened the possibility of developing larger models with more general purposes. This is already a reality in almost every application regarding unstructured data, like text, images and video. Many general-purpose models have gained fame in recent years: GPT-3 [16], BERT [17], CLIP [18], DALL-E [19]. All these models have been pre-trained using large datasets jointly with a self-supervised approach.
The goal of the pre-training phase is to learn a good representation of the input data. As a result, models trained within this scheme show great generalization capabilities even without further training, like the generation of text of GPT models. These capabilities can further improve after a second training phase, called fine-tuning over a specific task, for example, ChatGPT's stunning ability to conversate. Taking inspiration from these models, we use a large dataset of transactions to pre-train a Transformer network using self-supervised learning, and then we use a (smaller) labeled dataset to fine-tune the network for a specific task.
During the pre-training stage, we train our UniTTab model using the Masked Token pretext task [17]. Some of the input features are masked to the model, and the model is trained to predict the masked features as a function of the "visible" ones. This method makes it possible to train a model to automatically learn the semantics and to extract relevant information contained in the sequence of transactions. Specifically, for an input sequence, we randomly replace a field value with the special symbol [MASK]. We use a standard replacement probability value of 0.15 [17,13]. Moreover, with probability 0.1, we also mask all the fields in a transaction, while, for the fields representing the time stamp, they are always either jointly masked or jointly unmasked. These additional masking strategies, inspired by the block masking of adjacent image patches used in BEiT [20], make the pretext task more challenging for the network.
It is important to remark that, within the pre-training phase, the training of the model solely relies on input features of transactions, eliminating the need for labels. This way we can use the entire transactional dataset, even if it is not fully labeled for the specific downstream task. This is the case of the Czech dataset [21] used for loan default prediction, described in Section 4.1.
We develop a custom model called UniTTab, designed to be suited for sequences of heterogeneous transactional data. Borrowing the techniques used in text analysis in BERT or GPT models, we use input time series with variable length. We vary the sequence length from 10 to 150 transactions, where each transaction is composed of a fixed number of 10 or 6 fields. As a result, each time series can vary in length from 100 to 900 items, a challenging length to manage even for text sentences.
Given the data structure, we propose the hierarchical architecture shown in Figure 2. The architecture is composed of two different Transformers, and it is trained endto-end. The first Transformer ("Field Transformer") takes as input the 𝑘 features describing a single transaction, like transaction amount, merchant information, transaction date and time. The features are transformed in 𝑘 final embeddings, which are concatenated in a single embedding vector. This embedding is the representation of a single transaction. Then a sequence of these embeddings, each representing a transaction, is fed to the second Transformer ("Sequence Transformer"). The Sequence Transformer models the statistic dependencies between different transactions in the sequence and outputs embedding elements in the latent space. This is the general-purpose latent space where the representation can be potentially exploited for many tasks, such as classification (e.g., to classify the client behavior), detection (e.g., to detect anomalies, frauds), and prediction (e.g., to predict product churn in next few months).
As depicted in Figure 2, during pre-training, for each masked field we use the corresponding output embedding to predict the field value. Instead, during fine-tuning, we add a class token [CLASS] at the beginning of the transaction embedding sequence, and we use the corresponding output embedding to solve the financial tasks. This embedding can attend to all transaction embeddings in the sequence, allowing it to exploit all field information of all transactions.
Our model can effectively represent heterogeneous data, encoding both numerical fields (e.g., the amount), categorical fields (e.g., the type of transaction), and fields with a specific structure (e.g., the date).
The most common approach to tackle this challenge is to reduce all the features to a common representation: usually numerical for ensemble of trees or categorical for deep learning architectures like TabBERT [13]. However, discretizing numerical features into a finite set of values results in a loss of information. For example, it could be important to know if an amount is precisely 20 euros or 20.50 to distinguish between a withdrawal and a grocery expense. For this reason we develop a custom representation to transform numerical values in the input vector. In particular, we represent each numerical value as a feature vector obtained by the concatenation of a battery of different frequency functions (depicted with a sine wave symbol in Figure 2). Similar representations are used in NeRFs [22] for 3D synthesis. Conversely, we adopt a traditional "category encoding" to represent the categorical features, by using simple embedding neural networks (as used in [13]). Finally, we use a custom "time encoding" method for the timestamp attributes. The value of a timestamp is split using a combination of different field values: the year, the month, the day and, if necessary, the hour. Then each such value is represented as a categorical feature (e.g., with 12 elements for the month).
The effectiveness of the model has been tested over two large size datasets of transactions: the PKDD'99 Financial Dataset [21] and our Real Bank Account Transaction Dataset (in short, RBAT Dataset). The first dataset is public and is used as a benchmark for predicting loan default. Instead, the second dataset is private and is used to assess how well our model predicts customer churn in comparison to standard industry models. The chosen experiments are binary classification tasks with a large level of unbalance in the statistics of the two target classes.
In all the experiments, the model is first pre-trained using self-supervision (Section 3.1) and then fine-tuned on the classification task, in a standard supervised way, using the labeled data.
The loan default prediction is a classification task defined on the PKDD'99 Financial Dataset, which is a public dataset of real transactions from a Czech bank [21]. This dataset is composed of 1M of transactions from 4500 clients. It also includes customer information, but we use only the transactions, each composed of 6 fields (timestamp, amount, type and channel of the transaction).
The dataset presents a large fraction of unlabeled data, in fact most of the accounts don't have any loans, and they cannot be used for the classification task. This is the perfect example of the potentiality of our model: we perform the pre-training on all the accounts present in the dataset (4500) and then we fine-tune the model only on the labeled ones (478 for training and 204 in test). With such a small number of samples UniTTab has been able to obtain good results, way higher than ensemble of trees, and the possibility to exploit all the data in pre-training is its main advantage.
To evaluate our model's ability to deal with longer sequences and variable lengths, we test different sequence lengths of transactions. We define a maximum length value 𝑡𝑚𝑎𝑥 (ranging from 50 to 150), and for each account we include the entire sequence of transactions if it is shorter than the maximum. Instead, if the sequence exceeds the maximum length, we only consider the most recent 𝑡𝑚𝑎𝑥 transactions. It's important to note that, during pre-training the average sequence length is 232 transactions, whereas during fine-tuning the average sequence length is 80 transactions. This happens because, for fine-tuning, we only take transactions made before the loan begins. For this reason, if we set 𝑡𝑚𝑎𝑥 to 150, during fine-tuning almost all transaction sequences are of variable length. It's also interesting to observe that, increasing the length of the sequence, the result of the model improves. This is likely due to the information increase in the input sequence, but it demonstrates that the model is able to deal with long sequences of transactions.
One of the main advantages of using Deep Learning methods over traditional Machine Learning approaches is the possibility to pre-train a large network using a large unsupervised dataset, and then fine-tune the same network on the (usually scarcer) available annotated data of a downstream task. In order to quantify the contribution of the pre-training phase, and to show that this is useful also when the unlabeled dataset is not huge, we use the PKDD'99 Financial Dataset, and we pre-train the models with different portions of the pre-training dataset. Specifically, in Figure 3 we indicate the fraction of the pre-training dataset used for each experiment, where zero corresponds to training the models from scratch directly on the (labeled) downstream task data. The results in the figure show that both Deep Learning methods (i.e., TabBERT and UniTTab) significantly benefit from the pre-training phase, even using only a small portion of the unlabeled data (e.g., 0.25). Furthermore, when pretraining is performed, our UniTTab gets a significantly higher F1 score than traditional tree-based models.
We also compare our model with a custom transactional tree-based pipeline on a churn rate prediction task. The task is defined on the private RBAT dataset, which is provided by an international bank and is composed of several hundred million real transactions of bank customers. The churn prediction task is defined as whether or not a customer churns in the next 3 months, given a 6-month history sequence of transactions performed by that customer. The history sequences provided to the models are of variable length, with an average of 192 transactions and up to a maximum of 500 transactions. Each sequence is associated with a binary target that represents the presence of the churn event for that customer in the following 3 months. Initially, our model has been pretrained on a random sample of 1M untargeted accounts, corresponding to approximately 300 million transactions. Then, we evaluate the performance of our model and industry standards using fine-tuning datasets of different sizes, ranging from 50K transaction sequences up to 1 million sequences. Figure 4 shows that our UniTTab model significantly outperforms industry standards for every training dataset size. It also demonstrates the scalability of our model through an increased number of fine-tuning samples: increasing the number of training accounts yields considerably improved AUC on the churn prediction task.
The UniTTab project presented in this paper is a step towards the creation of general-purpose architectures for bank transactions. The empirical results show that our model drastically outperforms both deep learning and standard machine learning based predictive models on different benchmarks. We believe that our work and our results can stimulate this research field and the adoption of self-supervised deep learning in banking data.