Software system behaviour analysis is a challenging research problem in software engineering. The main reason for this is the lack of real data from large industrial systems. Softtech Inc. is a subsidiary of a large private bank in Turkey and this study is aimed to analyse mapping the services architecture and the software system health of a particular department at Softtech using speci c software web service logs. The services that are the subject of this study consist of 196 web services related to credit and credit card application transactions from various channels. While these processes are related to similar applications, they call various web services that perform di erent operations in the background. Related services account for 2 million logs daily. We have conducted empirical and statistical analysis on the data, in order to infer the correlations and dependencies among the observed web services. Hypothetically, we think there are 3 types of dependencies between the web services. In our experiments, we used average response times and the number of times web services are called at speci c time intervals as input data. The results suggest that they can be used for inferring that there is a dependency between two web services. In this preliminary work for dependency inference from unstructured web services' log data, we have utilized simple statistical analysis tools to derive important insight about the collection of services under our observation. The results have encouraged us to carry on with a more detailed analysis approach to further advance our research e orts.
Ethem Utku Aktas1;2, Mehmet Cagr Calpur1;2, Umit Ulkem Y ld r m1, and Emrah Y ld r m1
1 Softtech A.S.
Research and Development Center Tuzla Piri Reis Cad. 62, 34947 Istanbul, Turkey
2 Sabanci University
Orta Mahalle, 34956 Tuzla, Istanbul, Turkey futku.aktas, cagri.calpur, umit.yildirim, emrah.yildirimg@softtech.com.tr O zet. Yaz l m davran s analizi onemli yaz l m muhendisligi konular ndand r ve endustriyel sistemlerden elde edilen gercek veri eksikligi nedeniyle incelenmesi zor bir problemdir. Bu cal smada Turkiye'nin en buyuk ozel bankalar ndan I_s Bankas 'n n yaz l m gelistirme ve bilgi islem alanlar nda hizmet veren istiraki Softtech A.S.'nin bir departman n n gelistirdigi ve yonettigi web servislerinin loglar ile web servisleri aras bag ml l k analizi ve servis mimarisi c kar m yapmay hede iyoruz. Cal smam za konu olan servisler cesitli kanallardan gelen kredi ve kredi kart basvuru islemleri ile ilgili 196 web servisinden olusmaktad r. Bu islemler benzer uygulamalarla ilgili olmakla birlikte, arka planda farkl islemler yapan cesitli web servisleri cag rmaktad rlar. I_lgili servisler gunluk 2 milyon kutuk kayd olusturmaktad r. Elde ettigimiz veriyi ureten servisler aras ndaki korelasyon iliskisi c kar mlar nda bulunabilmek icin ampirik ve istatisliksel analiz yontemleri kullan lm st r. Varsay msal olarak, web servisleri aras nda 3 tur bag ml l k oldugunu dusunuyoruz. Deneylerimizde, belli zaman aral klar ndaki ortalama yan t sureleri ve web servislerin kac kere cagr ld klar girdi olarak kullan lm st r. Sonuclar bu iki bilginin, web servisleri aras nda bir bag ml l k oldugu sonucuna varmak icin kullan labilecegi ongorusunu ortaya koymaktad r. Bu oncu cal smam zda istatistiksel analiz yontemleri kullanarak toplad g m z web servisi verisinden degerli bilgiler c karmay basard k, bu sonuclar bizi arast rmam z daha ileri goturmek konusunda da tesvik edicidir.
Anahtar Kelimeler: I_statistiksel Analiz Tahminleme Analitigi Bag ml l k C kar m Kutuk Analizi Yaz l m Servis Yonetimi Yaz l m Guvenilirligi
Yaz l m Mimarisi
Software systems evolve and as they evolve, their complexity increases. As a
result it gets harder to monitor them. However, reliability of such systems is
a key factor for customer satisfaction. One of the fundamental assumptions in
monitoring such complex software systems is that there exist patterns in the
behavior of executions. Many previous studies support this assumption [
Recent trends in software architecture such as microservices, magni es the problem of maintaining large number of services. The maintenance problem is really complex for large companies that rely on hundreds or even thousands of web services. Documentation and records for ownership of the architecture may be neglected. In this study, we propose web service dependency inference techniques to map large scale software systems.
One of the problems in studying this research area is the lack of real data from
such complex systems. Execution data such as system logs need to be collected
and analysed in these systems and we conjecture that it is possible to nd out
dependency relations between services by observing the log data [
In this preliminary study, our aim is to conduct a statistical analysis using system logs of speci c web services and nd dependency relations among these services. First, the architecture of the system under observation is presented. Then, structure of the log data and the procedure to collect and preprocess this data is explained. The methodology for the analysis of the data and the results are given next. Finally, comments and conclusions on the study are presented. 2
Lin and Siewiorek collected error log data from 13 le servers for a 22 month
period [
Hellerstein, Zhang and Shahabuddin proposed a predictive approach to
detect the probability of a threshold value to be violated for a production web
server and the occurrence time for the violation [
Weiss modelled a genetic based machine learning system to predict rare
events by identifying predictive temporal and sequential patterns within data
[
Vilalta et al. reported three case studies for failure prediction [
Vilalta and Ma described an approach to detect patterns in event sequences
[
Levy and Chillarege conducted a case study to develop an approach for early
warnings of failure [
Liang et al. investigated the characteristics of failure events, the correlation
between these events and non-fatal events [
Fu and Zu developed a model to cluster failure events based on their
correlations and predict their future occurrences [
The system architecture consists of four layers: Top layer is the user interface layer. Second layer is for load balancing the transaction requests from the interface layer (IIS layer). Third layer contains the web services processing requests. Final layer is the database layer (Figure 1).
Tellers at branches send requests using Desktop/web applications or customers send requests via alternative distribution channels (such as internet, ATM, etc.). The data is gathered from the round trip times of second layer and the third layer, in other words, load balancing layer and web-service layer. It includes the time stamp, physical server name (IIS server), name of the web-service called, response time and the contents of the request and the response.
Data preparation is the crucial part of statistics and machine learning processes. In the following section, the structure of the log data, how they are selected and pre-processed are explained. 4.1
The research data is a subset of the original log output that is gathered from the system. The information which had been used for our research was extracted from the original logs, which are web-service name, time stamp, number of calls, response time, system and business faults. 4.2
The features for machine learning methods are prepared from the number of calls data of the logs for speci c time intervals. The number of calls of a web service is an important indicator for showing the healthiness of the service. Increasing or decreasing number of calls may be indicating an upcoming problem in the system. Each row of the data set contains the number of calls for the same time slot of previous weeks, starting from 8 weeks prior to the prediction date. Since banking operations are mostly conducted on weekdays and the web services under observation are mostly used by the workforce in the main operations headquarters and branches, the dataset is formed using only weekday information.
The features for statistical analysis are also prepared by aggregating the log data. Both the number of calls and the response times for speci c time intervals are used for statistical analysis. The number of times the web service is called at speci c time intervals may show the dependency between the web services. For example, if the number of times a service is called increases and at the same time that of another web service also increases, this may be the result of a dependency between the services. The details of the statistical analysis are given section 5.2. 4.3
Log Data Aggregation. The amount of data generated for all the banking transactions would overwhelm any data processing system. The solution to this problem is by applying data aggregation techniques to the log data. Five minute periods of log data is aggregated into count of calls for that web-service, mean and standard deviation of the response time, and count of system faults and business faults.
If no service call occurs in that time period, no data aggregation can be done and so there would be missing data for those time periods. For such periods, data having zero counts, averages, etc. are inserted afterwards, so as to ensure the continuity of the time series.
The research consists of application of machine learning techniques on the aggregate log data to predict future response times and statistical analysis of the data for inferring the web service dependency mapping. 5.1
Linear Regression, Support Vector Machines (SVM) and Random Forest techniques have been employed for the prediction of number of calls at speci c time intervals from the aggregated data previously explained in Feature Selection section. Average error, which is the average percentage of the di erence between the true value and the predicted number of calls of the web service for the related time period to the true value, is used as the performance metric. Random Forest application is the best performing machine learning technique, SVM come in second place with the average error value almost doubled and Linear Regression is the worst performing algorithm. The average error results are in the Table 1. The data consists of response times and number of calls for 196 web services, our aim is to uncover the relationship between these web services. In order to gain insight on the subject we have employed statistical methods and analysed correlations between the services' behaviour.
Empirical Analysis The purpose of empirical analysis of the data was to conduct fast analysis of the data to lter out obvious unrelated services. We have created pairs from each service and calculated Mean Percentage Similarity(MPS) for each pair. MPS values are calculated using the number of counts of the web services during speci c time periods and these values are expected to show us the similarities between the web services. The error is based solely on web service activity, which means if there is a call for the web service there should be log data generated for that time window in our log les. Since the data is from one of the biggest banks and approximately 2 million log records are generated, there is a very high chance many of the services get calls in the same aggregate window. Therefore we have ltered the MPS values to search for 99% and above similarity for the empirical correlation observation. There are 196 pairs with 99% and higher MPS value, Table 2 shows some examples for the MPS based correlation. The evaluation shows that we can observe the dependency relations for these services as they represent the 3 dependency types that we have derived from the actual system.
Correlation Analysis The second correlation type is based on the Pearson Correlation. The variables are the average response times for each aggregation window for the services under observation. In time series analysis, a correlated event may occur some time after the original event, such a delay is called lag and application of lag to the correlation analysis may yield interesting outcomes, which are normally hidden.
Pearson analysis was conducted on the data with 0 lag value. Because, in this work, the nature of our data collection strategy (aggregation of log data in 5 minute intervals) introduces a pseudo-lag property to our data set. The correlation is about the response times and the resulting R values suggest that both positive and negative correlations can be observed for the response times of the web services. We have ltered the Pearson Correlation results to observe only the strongest correlations with 95% or higher for positive correlation and negative correlation results.
Table 3 includes sample values out of 1020 correlation rules surpassing the 95% lter. Information in Table 2 and 3 summarizes the results and shows that empirical analysis produced rules are compatible to the Pearson Correlation. Dependency Types Finding dependencies between web services is considered to be useful to nd the possible sources of problems and the e ected services when these problems occur. The architect of the systems-under-evaluation analysed the correlation rules and we have conjectured 3 types of dependency relations. The rst type of relation is Dependency on the Data Source, where two unrelated services competing for the same database resource (Figure 2).
The second type of relation is the Hierarchical Dependency, where a web service is called in another executing web service. Apart from the rest of the execution time for calling service, the response time is increased by the response time of the inner web service (Figure 3).
The third type of relation is the Serial Execution Dependency, where a web service call follows another web service call in a procedure. We conjecture that negative Pearson Correlations of service calls occur based on this kind of dependency. If the rst called web service becomes unresponsive and the response time increases, the second web service would not execute (Figure 4).
The dependencies given above are the hypothesis that we think there are between the web services. And the selected correlation results of R values most possibly verify our hypothesis with our current knowledge of the related web services. However, the results should be veri ed by increasing the granularity of the data used. 6
The results of the research demonstrates the inadequacy of the data aggregation method, because it is very likely that the real behaviour of the web services are lost. For further research e ort, we will try to improve the granularity of the data to be able to re-apply machine learning methods to the response time predictions. Response time prediction and anomaly detection methods are very important for large organizations utilizing numerous system for their operations. Such research is important for companies with strict service level agreements and availability goals.
The statistical analysis methods generated some rules, which provides insight about the system-under-observation. By observing the system architecture, we think there are 3 types of dependencies between the web services. The results suggest that "average response times" and "the number of times web services are called at speci c time intervals" can be used for inferring that there is a dependency between two web services. As a future research direction, we will use the correlation results as a foundation and we will conduct software system dependency mapping research.