Data matching is widely used in different applications. However, very often we do not know data schema completely of the matching objects, because it might be changed during application use. As a result, objects require different processing methods, depending on their structure. To provide this, a special way of storing data should be used. For building an effective application, we need to choose DBMS suitable for selected storage data structure by measuring time spent on different operations with a high load database. The purpose of this article is to demonstrate a benchmark development which is performing these measurements and estimations.
Nowadays, due to a huge amount of information, describing real-life objects, searching, comparison and choosing the most suitable object requests are becoming more and more actual. Data matching is widely used in different applications. It might be processing of social data and real-time similarity calculation between persons, recommending systems with computation of scores for different goods for a specific person and many other domain areas. The data matching issue can be easily solved when the data structure is fixed and simple, or data can be stored in memory only, but there are many issues when an object has complex, dynamically changed structure and it becomes impossible. In that case, we have to speak about using some storages and databases. Nevertheless, matching of data is a different task comparing with data fetching. Moreover, very often we do not know data schema completely of the matching objects because it may be changed during application use. As a result, the answer to which data structure should be used for matching of some objects is not so obvious. Besides, those objects require different processing methods for different substructures.
The main purpose of this work is to do assessment of different implementations of data storing for solving the task of data matching, on big data databases with complex objects.
To achieve this, we need to conduct a performance analysis of the number of database systems and provide systematic comparison of them. Among analytical modeling methods for are applicable for DBMS’s are: 1. Queuing Models: Queuing models are effective to study the dynamics of a database system when it is modeled as a multi-component system with resource allocation constraints and jobs moving around from one component to another. Examples of such dynamic studies are concurrent transaction control algorithms, data allocation and management in distributed database systems etc; 2. Cost Models: Cost Models are useful in studying the cost in terms of Physical storage and query processing time. The cost model gives some real insight into the actual physical structure and performance of a database system; 3. Simulation Modeling: A simulation Modeling is more effective for obtaining better estimates since it not only analyses the database system in isolation but also can effectively analyze the database system with the application program running on top of it and the database system itself operating within the constrained environment of an operating system on real physical hardware; 4. Benchmarking: Benchmarking is the best method when multiple database systems need to be evaluated against each other but suffer from the inherent setback that it assumes all systems to be fully installed and operational. Benchmarking relies on the effectiveness of the synthetic workloads. Real workloads are nonrepeatable and hence not good for effective benchmarking.
The most suitable is benchmarking method, because we need to measure a similar operation on a set of DBMS. 2
There are standardized benchmarks called as TPC or “Transaction Processing
Performance Council” [
TPC is the largest and most popular benchmarking authority, but, still there are
some other benchmarks among them [
It provides knobs to vary the system parameters for a single scenario across different
sets of rules or environments. The standard run parameters for synthetic replication
include a number of users, number of tables, number of rows per table, number of
rows returned by queries or size and complexity of queries, etc.
• CIS benchmark [
Nevertheless, previously mentioned benchmarks are not suitable for data matching measurement. The main problem is that we need to measure big data and decision operations on both SQL and NoSQL DBMS’s it one program. Another problem is that we do not know data schemas of the matching objects completely because it may change during application using. In addition, those objects require different processing methods for different substructures. These facts make the development of your own benchmark relevant. The definition of data storage structures is needed for this purpose. 3
While implementing a benchmark, we consider Friend of a friend (FOAF) models as a
base for our complex object representation. FOAF describes the world using simple
ideas inspired by the Web [
Complex object operations and requests are ones that can be made to specific objects with high probability. For the benchmark, we define the following list of operations based on FOAF object. ─ Find object by a set of FOAF properties; ─ Find objects by FOAF property with restrictions; ─ Find an object by special property and value; ─ Find similar objects.
Three groups can generalize these use cases: 1. Finding an object by certain property or group of properties; 2. Finding an object by word or phrase occurrence in property; 3. Finding a similar objects, according to all properties.
For the first group it is advisable to use simple property search with property value selection by conditions, united by logical “and”, which is integrated into DBSM toolkit.
For the second group fuzzy search is the most suitable. This search type intended to preserve mistakes that are made because of the user’s misprint. The search is proceeded by the entered request first, and then by several similarly written requests.
For the third group comparison on calculating generalized estimates is the most suitable. Working principle of this method is in calculating general estimate for the whole object based on local properties’ estimates. Then the most appropriate objects are selected by comparison of objects’ general estimates.
To make a proper program that is worked with complex objects it is needed to analyze the requests execution speed on different database sizes and models. Benchmark, measuring time spent on a single operation should be programmed to get this information.
As we have different DBMSs’ with different logical and physical data models we need to determine most general data models but with their specifics. As FOAF models are the base for our complex objects, it is suitable to use data models containing full information about storing them. The developed data model should describe the set of object fields for each object in the database.
Three main data models have to be implemented in each test in order to compare DBMS’s performance in similar situations. Nevertheless, there can be additional structures to show DBMSs’ unique features, according to their specific.
1. Field data model – each object property has its own database object, containing the property value and root object identifier (see Fig. 1). 2. Key-value data model – each record in database is key-value pair, where the key is the name of object property and value is the value of this property for a particular object. Also, the identifier of the root object is added to this record to provide connection of the key-value pair with the root object. The example of this structure can be seen in Fig. 2. 3. Tuple data model – each object has its tuple, contains key-value pair for each object property. This tuple also stored with a root object identifier (see Fig. 3).
Performance measurement has to be done for three different database storage model
they are relation, graph and document-oriented. The most common DBMS in their
segments are PostgreSQL [
Two more data storage structures were added to the test according to selected DBMS specific.
4. Json data model – PostgreSQL has a special jsonb type for work with json objects. It stores it in the parsed binary format that helps to speed processing up. Resulting json in the database consists of key-value pairs for each FOAF property. Fig. 4 is an example of how data stored in json
Fig. 4. Data stored in json 5. Connected graph data model – connection to root project is provided by edges with property key-value pairs, making connected tree to particular property value. How data is stored in a graph is shown on Fig. 5. The table below shows data models distribution across the selected DBMSs. Three modules “Test PostgreSQL”, “Test Neo4j”, “Test MongoDB” were implemented for performance measurement of DBMS. According to program specific, each test should implement three main object operations. The special interface should be created for this purpose, to make operation implementation in each test’s class necessary. Class subject area diagram is shown on Fig. 6.
The diagram represents following classes: ─ Test – interface, containing main operation for performance measurement description; ─ App – test launcher class; ─ TestPostgreSQL – performance test implementation for PostgreSQL DBMS; ─ TestMongoDB – performance test implementation for MongoDB DBMS; ─ TestNeo4j – performance test implementation for Neo4j DBMS.
All test classes contain three general methods that were declared in Test. These methods are: seed – is used for generating random information and save it to selected database; search – is used for calling all search-methods in class; clean – is used to delete all test data from the database after a test run.
Each class also has specific methods used for searching data. Search queries are depended on data model presented in testing database. For example, search methods are: testSearchByTuple, testSearchByKeyAndValueInRelations, testSearchSingleRepo. 4
Performance test starts after launching the benchmark’s code on all described DBMS and includes tests with 1000, 10000, 100000 records in the database. During each iteration, a random data, generated by Java Faker library, added to the database. Each added object has seven main fields of FOAF representation. They are: name, surname, address, email, gender, interests, education. The example of the database object is on Fig. 7. - tests: Array - context: context + startTests()
TestNeo4j - driver: Driver - rand: Random + clean () + search (db: String?) + seed (db: String?) - testSearchByValue(n: int) testSearchByKeyAndValu e(n: int) - testSearchByTuple(n: int) testSearchByKeyValueInR elations(n: int) bulkGenerate(timeMeasu rer: timeMeasurer, n: int)
TestMongo - repo: RecordRepository? - KeyRepo: KeyRecordRepository? - tupleRepo: TupleRepository? - rand: Random + clean () + search (db: String?) + seed (db: String?) - testSearchSingleRepo(n: int) testSearchSingleKeyRepo (n: int) - testTupleRepo(n: int) testSearchDoubleRepo(n: int) bulkGenerate(timeMeasu rer: timeMeasurer, n: int)
TestPostgreSQL - data_session: Session - rand: Random lates to described data models. In Table 2 this relation are set.
As it is seen from the table, PostgreSQL requires more time to seed information in the database, but it has the best results in searching scenarios on every tested database size. The PostgreSQL loss on seeding information is not critical, as in real application it is impossible to get such load in a small period of time, the database size will grow constantly.
According to this research PostgreSQL and tuple data storage structure are the most suitable for developing complex object matching system. This work is in progress. In the future, we will extend a number of operations with data and will change model structures. In addition, some additional matching specifics operations will be added. One of the issues, which is not considered in this paper, is benchmarking of big volumes of data in a computational cluster. In that case, huge difference in performance results might be found. This part we also will consider in further work.