In this paper, we describe an application of a recently proposed comprehensive UML-based (Uni ed Modeling Language) approach to the conceptual modelling of log les. On the real example, we built an ad hoc UML-based (class) diagram to represent the key features of the logs nested structure and generated an artifact (a template in JSON) based on ECS (Elastic Common Schema). We also describe plans for designing a specialized tool through a conjunction of the already developed artifacts. Presented work is a part of a broader study on the proposed initiative for the general concept of log les standardization. A clear structure of log data would allow more systematic development and more straightforward implementation and employment of the latest information systems, minimize anomalies, errors, and time delays.
The stable work of information systems, with the constantly increasing
complexity, and security of a tremendous amount of data, they are processing, profoundly
rely on log les management. A log message is a piece of information produced
during the work of the computer system or software, generated as a response to a
running process or an action. The information pulled out of the log message
provides an idea of the log message meaning and the reason for it being generated.
Despite that modern log les management systems are powerful mechanisms
for resolving issues of the IT industry generally, a growing number of custom
solutions make every case rather particular [
Nowadays the practice of fast development and customization of applications
leads to the situation when logs semantics is not always clear. Such messages do
? Our work is performed with the support and in the interests of the company SIA
s.r.l., the provider of information technology solutions in the banking domain. The
authors are particularly thankful to Daniele Spinelli (daniele.spinelli@sia.eu).
Copyright © 2021 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
not give a distinct perception of processes and interfere with further analysis,
hence, are not quite valuable. Even more challenging to keep track of di erent
log le formats in large heterogeneous systems in which software and devices are
dynamic in nature [
We declare an intention to limit heterogeneity in log les management by
proposing a standardization of the log les through developing a conceptual
modelling approach and a suitable tool. Conceptual data modelling provides
analysts and designers with a high-level representation of the real world and
an e cient way to communicate with each other. Such data models promote
understanding of the real-world domain and enhance the ability to meet users'
requirements [
Based on the study and scienti c literature review, we have determined a lack
of studies on conceptual log les modelling [
In this paper we describe a comprehensive general approach to conceptual
modelling of log data with a UML-like [
Applied Approach and Features to Conceptual Modelling Log Files Our choice to establish log les conceptual modelling on extended UML-based (class) diagrams and on JSON is based on the following motivations: { The UML graphical notation is commonly used over decades; it is structured and easily understandable by various users. { Recently the JSON format has widely emerged as the most convenient standardized format for structuring data such as log les. { JSON is relatively (with respect to other structured formats) compact, exible, as almost every programming language can parse it, and human-readable. 2.1
Since log les are mostly created automatically, as a minimum log data must include date/time stamp, description of the event and information unique to that event, in order to provide information to bene t further analysis, troubleshooting processes or data breach investigation. Information must be structured and suitable for running data analysis with the use of various tools.
JSON logging provides more exibility to the current logging system, especially when migration from the text logging format (as most common and unsettled format) to JSON can be simply performed. There is currently a vast number of frameworks and programming language drivers that support the translation of log data in JSON format if it was not initially the case. This shows the tendency of the industry to a standardized format for structuring such kinds of data.
One of the advantages - JSON is simple to implement in languages
without built-in JSON functionality. It is important to highlight that on the
metadata level ECS (Elastic Common Schema) [
An application of the recently proposed approach [
A log record is composed of attributes and eld sets, which group in their turn attributes related to the same feature the eld set is representing. This gives extensive and clear to any user representation with a possibility to implement both top-down and bottom-up strategies on system development and/or adjustments.
Model Features The concepts of the class diagram model were taken as a basis
[
Composition associations are used to represent a proper nesting, where the nested parts may appear also within other parts, i.e., they are reusable. Field sets are represented through a class-like shape, where we distinguish three di erent sub-boxes, for core, extended and custom elds, respectively. An ad-hoc notation is also introduced for local nesting of eld set. Other aspects considered in the conceptual data model for log les are types for attributes, associations between attributes, enumeration types, ECS metadata as categorization events, and an array of values.
For a demonstration, we show our extended diagram model obtained from the common log le record. It helps understanding complex data transformations. Fig. 1 shows a UML-based diagram for a single batch log of the custom application in the banking domain created with the implementation of the proposed conceptual model.
It is fair to highlight that the application is in use and currently acquiring log les are presented in text form, not structured accordingly, have numerous issues and completely unsuitable for proper monitoring and especially analysis. Applying our approach, together with an application owner, we succeeded to design a set of suitable log les records for the batch processes. 3
As for the usability of the conceptual approach, we started by considering some real-world domains, from bank applications. Indeed, such kind of application covers various general event logging. 3.1
The raw data were taken from the ECS repository opened for contribution at https://github.com/elastic/ecs. Originally ECS provides excessively many elds for log records, and only a few of them are needed to be populated for a certain case. Repository collects various les and tool templates, yet they do not provide universal applicability to any system.
We have chosen to maintain customizations by taking into consideration the tools provided by ECS and creating our own generator (python script, input and output les) to create relevant artefacts for the unique set of data sources. The script is running through the command line. Here are the main steps of the working process: { As an input le, the current version of the ECS log elds set in YAML format is converted into JSON. { Users can select the log elds from the set and include custom elds relevant to the project if it is needed. { As an output, the artifact in the format of JSON le is obtained; it represents a sample template for a log record for the particular system.
Notwithstanding that the script is still in active improvement, it is already has been in use for several test cases of batch log les modelling. Fig. 2 provides an example of the case used as well for the UML-based diagram demonstration.
This script is a preliminary development for the future tool and has not been published in open source yet. It is one of the parts with the following that must relate to the UML-based graphical representation part. 3.2
At this point work not only propose the tool and step for its development but provides preliminary solutions. All together the organizational ow and conceptual model of possible architecture are showed on Fig. 3.
The tool is aimed at artifacts creation: log les structural templates (in JSON format according to de ned elds from the YAML doc) and related documentation (which includes extended UML-based (class) diagrams).
The tool is aimed to provide for the conceptual modelling of log les from the beginning of system modelling or act as a supportive solution for rede ning system logs. Yet in the second case, it is necessary to integrate loggers i.e. plugins for the application (system) logging library to format logs into compatible JSON format. 4
As the result, we provided a comprehensive overview of our work on log- les modelling schema (in form of extended UML-based (class) diagrams) and developed preliminary instruments (python script) for log le modelling in JSON format (according to the prede ned structure). In addition, we demonstrated our intention and actual steps for developing a comprehensive tool build based on the proposed conceptual log le models, which will provide both ad hoc UML-based diagrams as documentation and JSON formatted templates for log records.