Our tries to learn machines how to reconstruct journal entries with the aim of nding anomalies lead us to deep learning (DL) technologies. Nowadays Variational autoencoder and Long short- term memory architectures as well as other deep learning architectures solves wide range of problems, yet they are not enough implemented in a eld of accounting information systems (AIS). Inside AIS, accounting data follows accounting logic and makes speci c datasets constructed by different type of columns - categorical and continuous variables. Our aim is reconstruction of these variables. Development of the model capable for precise reconstruction is not an easy task. This paper describes our research for anomaly detection model architecture which will be capable to reconstruct dataset with categorical features mixed with continuous monetary value feature. We developed basic models trained on accounting journals from 2007 to 2018 and then tested in the scal year 2019. Still, lots of hyperparameters need to be checked if we want to improve accuracy. Deep learning research is an engineering task leaded by experience so there is no linearity in the model improvement. Consequently, this paper is our contribution to collection of experience in developing accurate, useful and intelligent accounting control system.
As in every information system, human e orts, as well as interaction between modules, can cause errors. Anomalies in accounting books occur on a daily basis, and unintentional human errors, attempted fraud, and continuous legislative changes are some of the critical causes. Anomalies occur despite the fact that the most existing controls integrated into accounting modules of modern ERP Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). This volume is published and copyrighted by its editors. SEBD 2020, June 21-24, 2020, Villasimius, Italy. systems are created in compliance with bookkeeping rules. Because small and medium enterprises (SMEs) do not have audit obligations regulated by law, manual tax inspections are the only mechanism of their accounting and tax control. In general, detection of errors, made intentionally or not, consumes a large portion of a bookkeeper's or tax inspector's time, and correction of errors is not an easy part of their job, particularly owing to the architecture and functioning of the accounting software modules. Namely, most of today's ERP systems have specialized documents (digital forms) for an every speci c business event. An every digital form is connected with one or more journal entry schemes created by senior accountants. Junior accountants or non-accountant employees do not have to be familiar with journal schemes because they communicate only through forms. As long as the modern accounting modules inside ERP systems are functioning based on the described principle, a single error in the only one journal entry scheme can cause an incorrect accounting entry for the whole set of connected digital forms.
When statistic methodology had become the part of a nancial audit process, life of employees involved in auditing became a lot easier. Now it is the time for improving and make more easier auditing, accounting and tax inspection processes by utilization of deep learning algorithms. 2
When the idea for this research began to form, we found a previous study dealing
with the same challenge and methodology written by Schreyer, M et. al. [
Shultz, M. et. al. [
Another study that inspired our research [
Anomaly detection accounting systems are developed by applying di erent
technologies as well as inputs. Instead of accounting data, the authors of the
study [
Other available papers considered did not exploit a deep learning technology
in the accountancy when solving a speci c real-world problem. Still they were
extremely helpful to us because they contained general thoughts regarding the
application of arti cial intelligence in the eld of accounting [
Inspired by Schreyer, M. et. al. [
Keras deep learning library has provided a signi cant contribution to our understanding of neural networks and arti cial intelligence(AI). This high-level neural network application programming interface (API) is capable of running Google's TensorFlow, Microsoft's Cognitive Toolkit (CNTK), or the Theano deep learning library. Deep learning libraries can be used for supervised learning, in which a network is trained on labeled datasets. A supervised model can be well optimized but may be useless at the same time when new data are applied. However, neural networks can also be used for semi-supervised learning, through autoencoders and with unlabeled datasets. Semi-supervised learning is more suitable for anomaly detection problems and the nature of our particular data, namely, unlabeled and without errors or fraud entries, according to an audit by an experienced accountant. 3.1
Semi-supervised learning can be realized through an autoencoder architecture. Autoencoders reconstruct an input given the same input, and are usually trained and tested on separate data. If they are trained and tested on the same dataset, the generalization of the model is sacri ced.
Based on the friendliness of Keras and the power of TensorFlow, a neural
network model was built as a result of this research. More precisely, instead of
a classical autoencoder, the model is a variational autoencoder (VAE), which
was simultaneously discovered by Kingma and Welling in December 2013, and
Rezende, Mohamed, and Wierstra in January 2014 [
In contrast to a classical autoencoder (AE), a VAE is capable of sampling
from a latent space to create an entirely new output, which is possible because
it turns the input into the parameters of a statistical distribution, namely, the
mean and variance, instead of compressing its input into a xed code in the latent
space, which is how a AE operates. Our decision to use VAE was in uenced
by An, J. and Cho, S. [
So the main advantage of the variational autoencoder is its probabilistic manner for describing an observation in latent space. VAEs can be used to develop latent spaces of sound, music, or even text; however, in practice, the most interesting results have been obtained with pictures. Accordingly, a VAE is potentially capable of generating new journal entries from a latent space as well as reconstructing existing entries.
The VAE model ( gure 2a). was trained through 1,000 epochs ( gure 1a.), and our algorithm saved the best model according to the binary cross entropy loss function. We tried to set the number of epochs as high as it is possible with the available hardware. The model checkpoint technique allowed us to set high number of epochs but only the best model is saved according to the binary cross entropy metrics. Binary cross entropy measures how far away from the true value (y, which is either 0 or 1) the prediction ^y is for each of the features and then averages these class-wise errors to obtain the nal loss according to equation 1.
L(y; y^) = 1 N
X[yi log(y^i) + (1 N i=1 yi) log(1 y^i)] (1)
We used relu activation in all layers except the last one in decoder where
we used sigmoid activation. Our model is compiled with rmsprop optimizer.
According to Chollet ([
In addition to an autoencoder, this research exploited the capabilities of a long
short-term memory (LSTM) architecture. LSTM is applied to a deep
learning model with LSTM layers. According to Chollet [
The LSTM model was trained through 500 epochs ( gure 1b.), across 3 time steps. In a wide pallet of loss functions packed in Keras, we chose the mean squared error metrics. According to equation 2, the best LSTM model was saved and used to predict the amounts in the test datasets.
M SE = 1 Xn (Yi n i=1
Yi)2 10 20 30 40 50 60 70 80 90 (2)
10 20 30 40 ; 20 30 40 50 ; 30 40 50 60 ;
40 50 60 70 ; 50 60 70 80 ; 60 70 80 90
Suppose we have MSE = 0.02. According to equation 2, the error of the LSTM
model is p0:02=0.14. Looking at our toy example target values ranging from 40
to 90, we can say that, on average, the error of the model was 0.14. This means
that, when predicting a value of 40, we guessed 39.86, and when predicting a
value of 50, we may have guessed 50.14, and when predicting a value of 60, we
may have estimated 59.86, and so on. However, these are average values. Each
individual error could have been negative or positive, making their sum zero but
not their squared sum. Thus, this is simply the average of the squared di erence
between the predicted and actual data points.
Transactions are classi ed through a pre-de ned chart of accounts based on a
prescribed structure of the balance sheet, the income statement, and o -balance
sheet items. Accounts are divided into ten classes according to the speci c needs
of the entrepreneur. Most enterprises use a chart of accounts from some of the
most popular Croatian accounting magazine publishers and adjusted to their
particular needs because the classes are not provided by law. Because the number
of account digits is also not regulated by law, but based on the needs of the
entrepreneur, in our research, we decided to use the rst three digits.
t-Distributed Stochastic Neighbor Embedding (t-SNE)
technique
In the year 2008, Laurens van der Maaten and Geo rey Hinton presented
technique t-Distributed Stochastic Neighbor Embedding (t-SNE) that visualizes
highdimensional data by giving each datapoint a location in a two or three-dimensional
map [
Both models were compiled with rmsprop optimizer. Layers of the both models
were used relu and sigmoid activation. The model VAE was trained and saved
when the loss function reached the minimum (Figure 1.). During the training, we
used 10% of the dataset for testing which is presented by the orange line in the
gure. It is evident that the loss function curve has some spikes. High
learningrate, batch size (number of samples processed before the model is updated)
and the number of epochs (the number of complete passes through the
training dataset) are the hyperparameters which demand ne tuning. Loss spikes are
often encountered when training with high learning rates, high order loss
functions or small batch sizes, according to the authors [
(a) VAE model
The gure 2a. shows types and connections between layers of the VAE model. First and last layer must have the same number of inputs (103) which is the same as the number of dataset columns. The model needs to learn 13,405 learnable parameters, such as weights and biases.
The models were trained on journals from 2007 to 2018 using 84,759 rows and tested on the scal year 2019 with 3,864 rows. The test dataset is 4.5% of the training size. Considering the document type, the account number from the chart of accounts and the debit/credit side of account, the VAE model incorrectly reconstructed 33 (table 1.) single journals entries and 7 journal entries pairs (table 2.) of 3.864 journal entries of the test dataset, that is 0.8% of the journal entries including 25 entries of the test dataset that have never happened in the trained dataset.
The LSTM-VAE model needed to learn signi cantly larger number of learnable parameters (162,824) with the monetary column as the extra feature in contrast to the VAE model inputs. That is 104 inputs (LSTM-VAE) in opposite to 103 inputs (VAE). As it can be seen in the Figure 2b., encoder of the model was included LSTM layer as well as the decoder, but decoder's LSTM (a) VAE (b) LSTM-VAE layer is under Sequential layer. Although the model was trained on the same journals from 2007 to 2018 using 84,759 rows and tested on the same scal year 2019 with 3,864 rows, we do not have predictions for the rst 4 rows due to the hyperparameter timesteps = 3. That gave us 3,860 predictions.
Both model needed to learn the document type, the journal account and the debit/credit side of the account. Additionally, LSTM-VAE model needed to reconstruct monetary values. We tried the model with di erent loss functions, number of layers and the best architecture had 138 (1.65% of the test dataset) reconstruction errors with monetary value included. It is important to mention that the LSTM-VAE model had di culties with the very rst 88 journal entries.
Despite the fact that the researched models had di erent inputs (monetary value included in the LSTM VAE model), decision has been led by the nal aim of our research, that is development of the most accurate autoencoder. As a bonus we got the opportunity to analyze predicted versus real value.
Although the common characteristics of the journal entry clusters given by tSNE methodology and their relationship were not researched, we used tSNE to visualize dataset and the given results. tSNE reduced latent dimensions of the autoencoders to 2-dimensional space. Points in the gures 3a. and 3b. show journal entries of the tested dataset. Marked points represents entries that the VAE and the LSTM-VAE model reconstructed incorrectly. It can be seen that the both models have reconstruction errors in the same area of the 2-dimensional space. Further, the models did not have a problem to reconstruct the most obvious outliers though it was not capable to reconstruct outliers closed to the well de ned clusters of the journal entries.
(a) VAE model Fig. 3: tSNE visualization of the models: Real journal entries with marked anomalies
With research described within this paper we are trying to collect experience in development of an arti cial intelligence models by using real datasets generated in real world accounting books and with the nal aim of developing intelligence module of accounting information system capable to assist to accountants, auditors and tax o cers in nding anomalies. What we were considering for anomalies are errors, intentionally made or not. Precondition for intelligent error detection task is the model which is capable to learn the whole entries in a training process and then to reconstruct every single entry from the tested dataset except these that were seen for a very rst time. To reached satis ed precision as well as to help us to understand the model functioning, we used the real-world dataset which is well known to us. It does not include any kind of errors or bookkeeping rules violation, but it includes entries that were happened in the test scal year for a very rst time.
In this research we tested two models based on autoencoder architecture. The rst model is variational encoder (VAE) model and the second one is variational encoder with Long short memory layers (LSTM-VAE) model. The inputs in these models were accounting journals for 12 scal years of the small-sized enterprise. Journals did not include any kind of anomalies or errors.
To reach satis ed accuracy of the model, we included document types and journal accounts as unavoidable drivers of business events. Except the most necessary variables, common for every accounting system, we added the monetary value column as the input into the second model (LSTM-VAE). We will also try to add more variables as an input to our models in our future studies.
In general, we can conclude that semi-supervised methods, autoencoders, are promising technology for developing anomaly detection modules inside AIS. Our research also showed potentials of the journal entry anomalies control system development with the help of t-SNE architecture. Visualization helped us to better understand the nature of our data as well as the models functioning. We saw that both models have reconstruction errors of the journal entries in the same area of the 2-dimensional space. Errors of the LSTM-VAE model are the errors of the VAE model at same time. Maybe world of accounting could have bene ts from visualization techniques as a supporting technology, so we will use t-SNE for latent dimension visualization in our future studies.
In this paper we did not analyze predicted monetary values generated with our LSTM-VAE model, because we tried to research autoencoders' learning capabilities of the bookkeeping rules and they are not dependable on monetary values. That will be researched in our future studies too.
Generally, VAE models have one more characteristic still unresearched in a eld of accounting. They have capability of generating new data from the latent space. It will be interesting to see results of our model generator of ctive entries in our future researches.
Both models we developed with respect of deep learning rules are promising. Still, accuracy in an accounting, especially in anomaly detection problems is requested condition, so the hyperparameters of the models need to be tuned according to the collected knowledge. Lots of experimenting with architecture and hyperparameters are waiting for us to get the prototype for the real-world module of an account control system.