There has always been a close connection between banks and the collection of different kinds of empirical data: banks, just like any other company, have always poured large amounts of resources into understanding numbers, and how to deal with them. Numerical data, being closely related to the financial performances of companies, has always taken the spotlight.
On the other hand, linguistic data has always been much less considered, due to the difficulties of analysis and underwhelming performances.
But things are changing. More and more companies are understanding the value of language, which contains information that no number can convey. Different Natural Language Processing (NLP) tasks, language resources, and computational linguistics practices have now become a staple in many realities, like sentiment analysis [1] and word embeddings [2].
In fact, there is a wide variety of linguistic data that banks can exploit: emails, bank transfers descriptions, internal communications, and customer feedback. Some peculiar issues arise, when dealing with linguistic data in the banking sector, like the usage of acronyms, abbreviations and technical terminology. These data are often proprietary, meaning that the bank owns them, and the access is forbidden to externals. While the quantity of information they contain is massive, a downside is that the impossibility of sharing it with other banks hinders the possibility of a more global analysis.
In this context, this paper wants to explore the application of topic modeling techniques to the auditing process, in particular regarding the analysis of reviews of anti-money laundering (AML) alerts. Topic modeling can, in fact, be an incredibly helpful tool for auditors who want to perform an in-depth analysis on large amounts of data.
An overview of topic modeling algorithms and applications in the banking sector, both documented in scientific research and in concrete applications within banks, will be presented. Then, we will provide a comprehensive description of the data employed, followed by the preprocessing operations. We will
Topic Modeling is an unsupervised task of NLP, consisting in the extraction of latent themes in a given corpus. Latent Dirichlet Allocation, or LDA [3] is a probabilistic generical model, which became the most widely used and expandedupon topic model. However, LDA faces several limitations, like scalability, low performances with large datasets, and the struggle against polysemy and homonymy [4].
To overcome the limitations of LDA, a lot of effort has been put into developing models that rely on word embeddings and neural networks, like ETM [5] and ProdLDA [6]. These models have been proved to provide better performances than LDA, at the cost of a higher computational effort [7].
In the last decade, topic Modeling has already been largely employed in the banking sector, and in auditing as well. [8] focused on the assessment and handling of frauds, while [9] analyzed financial misreportings. Another popular subject of analysis is accounting (for example [10]).
The data employed is a collection of reviews of anti-money laundering alerts, that are automatically detected by a rulebased detection tool, whose name cannot be disclosed due to a specific request. This tool is widely employed across all Italian banks, and is aimed at tackling potential money laundering and terrorism financing schemes. It uses advanced algorithms to identify patterns that deviate from standard behavior.
An activity is considered suspicious whenever it exceeds certain risk thresholds. These activities are then reviewed by a human operator, who will evaluate whether the movement is actually tied to illegal operations or not. If the operation is not considered dangerous, or if there is not enough evidence to decide whether the activity is actually a threat or not, the operator will write a brief review, consisting of two sections. The first one is a description of the analyzed activity, The second section is either an explanation for why it was not considered dangerous; or a statement about the lack of evidence and the need to keep monitoring. This latter kind Case of greengrocer active in the square of ***, only greengrocer in the square. Active bank account, that collects income and charges relative to the activity. No particular anomalies at the moment. We keep monitoring.
Examples of sentences from the dataset with translations of reviews usually ends with expressions such as 'monitoriamo' and 'continuiamo a monitorare'. The dataset employed consists of such reviews.
In Table 1 we provide two examples of documents, with their corresponding English translation. The English translations have been cleaned of abbreviations and spelling mistakes.
Due to hardware limitations, we worked using a selection of 35,000 documents, chosen randomly. The data is owned by Credem and is not publicly available, due to legal constraints. It is not possible to reveal the time period in which these documents where collected, nor the whole dataset size.
Each document has an average of 20.94 tokens per document.
It is important to note that the documents feature an abundance of spelling errors, abbreviations, acronyms, and missing blanks spaces between words. This in part due to a 300characters limit. By comparing the tokens in the dataset with a dictionary of 4 millions Italian words 1 , we obtain the results shown in Table 2:
Total number of tokens
OOVs in the complete dataset
The dictionary has been further enhanced in a data-driven approach, by including a list of Italian names 2 and surnames 3 , and a list of the most frequent acronyms featured in the dataset, so that they are not incorrectly considered OOVs. In order to find the acronyms, we created a list of all OOVs in the dataset, in descending order, based on frequency. The 20 most frequent acronyms were added to dictionary, such as PEP (Persona Politicamente Esposta) and CC (Conto Corrente).
The table shows that about 13% of the dataset is made of OOVs. In comparison, the UD_Italian-ISDT treebank 4 , tested 1 https://github.com/sigmasaur/AnagramSolver/blob/main/dictionary.txt 2 https://gist.github.com/pdesterlich/2562329 3 https://github.com/PaoloSarti/lista_cognomi_italiani/blob/master/ cognomi.txt 4 https://github.com/UniversalDependencies/UD_Italian-ISDT against the same enhanced dictionary, contains only 6% of OOVs. For this comparison, the treebank in its entirety has been employed, consisting of training, testing and developing set.
The result shows a peculiar dataset, containing a considerable amount of OOVs, which will require robust methods of analysis.
Before processing the data, we performed data cleaning through stopwords removal and lemmatization.
Stopwords removal includes prepositions, articles, and conjunctions. This operation is helpful in reducing the number of tokens to be processed, gaining in efficiency, while also excluding data without semantic content. This operation was performed using the stopwords removal tool for Italian provided by Natural Language Toolkit5 (NLTK).
After performing stopwords removal, the number of tokens in the complete dataset is reduced to 972,019, with an average of 13.47 tokens per document. Since we are using 35,000 rows, about half of the dataset, the number of tokens is 471,293.
Secondly, we performed lemmatization. The model employed is it_core_news_lg, provided by spaCy 6 , which is made by 500.000, 300-dimensions-shaped vectors. Lemmatization is helpful in maintaining consistency through the whole dataset, as well as improving text understanding and efficiency. The spaCy model employed has a lemmatization accuracy of 97%, which is a satisfactory performance 7 . However, the model's performance on the dataset was tested. We created a sample of 100, randomly selected documents, who were then manually lemmatized, acting as the gold standard. The model's lemmas were then compared to the gold standard. The model's accuracy score was 79%, which is much lower than its usual accuracy. This underwhelming result further indicates how challenging to analyze the dataset is.
Before preprocessing, the TTR (Type/Token Ratio) was 0.0541; after this operation, the Lemma/Token Ratio is attested at 0.0428. The score is lower, indicating that we managed to reduce dispersion. Reducing dispersion is helpful in improving the performance of the algorithms, since word forms that used to be different are now considered to be the same.
We have chosen three models for our analysis: LDA, ETM, and ProdLDA. These models were selected due to their different natures: the first is generative, the second is embedding-based, and the third is neural-network-based.
LDA assumes that each document is a mixture of topics and that each topic is a distribution over words. It uses Dirichlet priors to model the distribution of topics within documents and words within topics.
ETM represents words as vectors in a continuous space (word embeddings) and models topics as distributions over these embeddings, enabling it to capture more semantic relationships between words compared to traditional models like LDA.
ProdLDA is a neural-network based variant of LDA that uses a variational autoencoder (VAE) framework. ProdLDA models document-topic and topic-word distributions using neural networks, and it represents a "product of experts" model, focusing on improving topic coherence and overcoming the limitations of LDA.
The tool used for optimizing, training and comparing these models is the OCTIS (Optimizing and Comparing Topic Models is Simple!) library, developed by [11]. It allows users to compare the performance of various models with respect to different metrics, like Topic Diversity and Coherence Score.
Before training, a fundamental step is hyperparameters optimization, which controls the behavior of the algorithm, and therefore, its performance.
OCTIS allows to perform Multi-Objective Bayesian Optimization [12], a method that searches for the best hyperparameters configuration considering more evaluation metrics at once; in particular, the evaluation metrics we employ are:
• the Coherence Score, measuring how interpretable the topics are [13]; • the NPMI (Normalized Pointwise Mutual Information, measuring the statistical similarity of words inside a topic [14]; • Topic Diversity, measuring how different topics are from one another [15].
However, certain limitations need to be considered. In particular, the hardware employed was uncapable of handling such computational efforts; and, since the data is protected by privacy laws, using another, more powerful machine, is out of question.
To overcome this problem, we relied on SOBO (Single-Objective Bayesian Optimization) [16] which finds the best hyperparameters configuration with respect to only one metric. In particular, we chose the Coherence Score as the target evaluation metric. This metric was chosen due to its nature of measuring semantic coherence and, therefore, it can be considered a good indicator of topic quality. SOBO works by training the model n times, each with different hyperparameters. The output of this process is the configuration that provides the best result.
Algorithms were optimized and trained in four different configurations:
• without the enhancement of word embeddings; • enhanced by 1-gram Word2Vec [17] embeddings; • enhanced by 2-grams Word2Vec embeddings; • enhanced by pre-trained embeddings.
The Word2Vec embeddings are created from our dataset. Table 4 shows the composition of these word embeddings.
We can check the quality of the created embeddings by employing the library Bokeh8 . Bokeh allows us to perform interactive visualization, creating a representation of the vectorial space that can be easily examined. As we can see in Figure 1, the word embeddings create a plot where the different semantic fields are nicely divided and distinct from the others.
The pre-trained embeddings, instead, are trained on Common Crawl and Wikipedia 9 . The pre-trained embeddings composition can be seen in Table 5.
In Table 6 we can find an average of the scores of the evaluation metrics for each model run, either enhanced or not enhanced by the aforementioned embeddings.
We can clearly see that ProdLDA provided the best performances across all runs. In particular, the dataset enhanced by 1-grams embeddings yielded the best overall performance, with an average score of 0.564. Much worse is the performance of both LDA and ETM, which failed at creating distinct and interpretable topics. In the reminder of this section, in Table 7 we show some of the topics created by 1-grams-ProdLDA, together with examples of the most relevant words associated.
The topics of 1-gram-ProdLDA were examined by seven bank employees, working in the auditing sector. They were then asked how interpretable the topics were, and to give a label, indicating what that topic was about. The chosen label for each topic was the most frequent one, assigned to that topic, by the employees. Out of the 12 topics created, only one was considered to be non-interpretable, confirming the excellent performance provided by ProdLDA. However, this non-interpretable topic was also the most frequent, as shown in Figure2.
We can clearly see the even distribution of the documents associated to each topic. The most frequent topic, labeled as "X", is the aforementioned non-interpretable topic, containing miscellaneous or difficult to categorize documents. Most of the topics refer to specific clients' activities, like bank transfers, payments, or activities related to the bank account.
There are also some more specific topics. An entire topic is dedicated to tobacconists and gambling. This kind of activity typically makes wide use of cash, which can potentially be tied to money laundering schemes. This level of specificity in auditing could indicate either regulatory requirements for these sectors or the bank's recognition of unique risks associated with these business types.
There is also a specific topic for suspicious activities with foreign countries or carried on by foreign users. Dealing with cross-borders regulations on transfers can be difficult for the bank, suggesting that particular effort should be put into developing efficient strategies for auditing cross-border activities.
Using 2-grams word embeddings was the best option for both LDA and ETM. However, in ProdLDA, 1-grams word embeddings provided a slightly better performance. Nonetheless, 2-grams were generally the better option, especially considering the sharp difference in ETM. On the other hand, enhancing the dataset with pre-trained embeddings did not result in a significant impact: the performance improvement of LDA was
NLP is now an essential component of the banking sector, and any company that wants to be competitive should make use of linguistic data science. In particular, in this paper we presented a NLP task, topic modeling, and how it can be imple-
Pre-trained embeddings model parameters mented in the daily job of bank employees, in order to perform more detailed investigations. In particular, topic modeling can be a key component in the understanding and identification of money laundering schemes, as it allows auditors to perform more in-depth and focused analyses. For example, auditors could investigate patterns from the recent years, in order to have a better understanding on whether an activity is part of a larger trend, or an anomaly that deserves attention. After citing other implementations of topic modeling in banking, we described the data employed, and its preprocessing, consisting in stopwords removal and lemmatization. Examples were provided, showing the peculiarities of the documents in the dataset. Then, the data was processed using three algorithms: LDA, ETM and ProdLDA. These algorithms were evaluated using three metrics: coherence score, NPMI score, and topic diversity. The optimal hyperparameters were found using SOBO. Optimization and processing were performed using four different configurations: without additional word embeddings, enhanced by 1-gram word embeddings created from our dataset, enhanced by 2-grams word embeddings created from our dataset, and enhanced by pre-trained word embeddings. The results show that ProdLDA's performance was far superior than its competition, especially when employing 1-gram Word2Vec embeddings. The algorithm outputted distinct and interpretable topics, which can provide a great insight into the data.
This experiment also has a large potential of being expanded. In particular, future works could employ a more computationally performing machine, in order to make use of the whole dataset, as well as performing MOBO, and obtain more precise hyperparameters. Finally, it is also possible to perform the same analysis on different kinds of data, in order to notice more clearly the differences and similarities from one kind of linguistic data to another, and their similarities. There are also new techniques that could have a great impact on this research, such as LLMs, Attention-based topic modeling, and Contrastive topic modeling.