This article presents analyze of using MongoDB database to storing aned ective data mining from open networskources. This paper attempts to use NoSQL insteoafd traditional SQL database in systems with strongly related information, comparing to relational and non relational approach in the performance and architecture.
Modern data storage technologies provided a practical opportunity to accumulate huge amounts of information, which allowed a qualitative change in the attitude to the results of analysis of stored information. It became possible to move from a descriptive process of analyzing the results obtained over a certain period of time to predictive data processing technologies that make it possible to o er valid recommendations for the future. The using of relational databases (MySQL, PostgreSQL, Oracle Database and others) to solve large data storage problems becomes problematic. The main advantage of relational databases is the availability of techniques for maintaining data integrity, achieved by storing links between data elements. However, the storage and validation of these links require additional time resources, which, with signi cant amounts and poor data structure, makes the use of relational databases di cult in some real-time systems.
As an alternative, there is a NoSQL database. One of the advantages of these databases in alternative storage formats and the links between them.
With the development of metaprogramming, the concept of re ection developed { the ability
of the program to use and modify its structure[
This led to the task of ltering such data across di erent data elds of text formats, including
using standard lters (for example, XPath[
A common problem is the problem of data storage, the nal form of which is some uni ed
format (hereinafter referred to as UF)[
In classical relational databases, storing such formats is associated with the creation of several
linked tables and subsequent cross queries [
Consider an example of a web service that provides access to the UF type JSON for some class "book":
Consider de nition of scheme for UF:
We assume that most often the book is ltered by title. And we assume that sometimes we just want to nd a book, for example, for a search function on a web site, and sometimes download a page with a book where besides the book and its description we also want to display all the comments.
It is common practice in the relational database to compile the following schema 1.
MongoDB uses JSON-like format to keep data and can keep data in the output view.
ORM or Object-Relational mapping is a programming technique designed to map database relations with object-oriented programming languages entities. It creates virtual objects database inside speci c language representation.
The main the goal of technique is get rid of the need to write SQL queries to access database data. The dual way of data representation, relation and object-oriented, usually requires from programmers to write code for getting data from database in relation way, after transforming it to object-oriented, and transform back to relation data to safe changes. Relational databases operate over sets of tables with simple data representations, it leads to use SQL "JOIN" operation to get full object information. Since relational database management systems usually do not implement a relational representation of the physical link layer, the execution of several consecutive queries (referring to one "object-oriented" data structure) can be too expensive.
Relational database management systems work with good performance with global queries,
a ecting a large area of memory, but object-oriented access is more e ective in the work
with small amount of data, as it reduces semantic gap between the object and relational data
representation[
Two way of data representation increases the complexity of object-oriented code to work with relational databases, it becomes more prone to errors.
The most common problem with ORM as an abstraction over SQL is that it can't fully abstract realization details. Some of the program realizations of ORM works as SQL code generations tools, some of them do not use SQL equivalents at the external level.
The reason why abstraction make sense is simpli cation of code writing, but if you use ORM framework with knowing SQL as a requirement, it doubles programmers e ort, for example the popular ORM framework Hibernate use HQL language SQL for complex requests, which is very semantically close to SQL. It brakes uniformity of the code abstraction when programmer needs a speci c union of data processing.
Ine ciency is another common problem of ORM. If programmer need extract object data from relation database, ORM cannot know which of the object property are going to be used or changed, so it forced to extract all, it cause many requests instead few. The lack of context sensitivity means that ORM can't consolidate requests, which leads to the impossibility of data caching or other compensation mechanisms.
ODM or Object-Document mapping is an alternative for ORM in document-oriented databases. The basic idea of ODM frameworks is the same, match data to the object, but we have few di erences here.
Firstly, in ORM we should complete data for the object, and complete data for backward mapping inside database. In ODM there is no requirement for fully completeness of data. The document can be mapped partly to a database, without multiple table changing.
Secondly, ODM can make data mapping independently from the data source. It makes easier to operate over data in a program.
Relation databases usually based on ACID { Atomicity, Consistency, Isolation, Durability[
NoSQL databases usually based on BASE: basic availability { every request will be completed (successfully or not) soft state { The system state can be changed without any data changes due to data consistency eventual consistency { The data can be inconsistent for some time but will be consistent after a while.
It's obviously that NoSQL databases can not be used in any application. Some application are requires for transaction systems (like banking, ecommerce, etc), but at the same time usual ACID system doesn't suit for systems based on large data storages, like amazon.com and other. So NoSQL databases sacri ce data consistent to make more scalable system, to operate over large amount of data.
Also, NoSQL databases represent following features:
Unlike relation databases, NoSQL databases have various data schemas, implemented through the use di erent data structures.
Depending on the data schema and the approaches to distribution and replication, four types of storage can be distinguished: key-value store, document store, column database stores, graph databases.
Key-value storage
The key-value store is the simplest data store that uses the key to access the value. Such
repositories are used to store media images, create speci c le systems, as caches for objects,
as systems well scalable by design. Examples of such storage facilities are Berkeley DB,
MemcacheDB, Redis, Riak, Amazon DynamoDB [
Bigtable-like databases (column database stores)
In this store, data is stored as a sparse matrix, the rows and columns of which are used as keys. A typical application of this type of database is web indexing, as well as tasks related to large data, with reduced requirements for data consistency. Examples of databases of this type are: Apache HBase, Apache Cassandra, Apache Accumulo, Hypertable, SimpleDB.
Column family stores and document-based repositories have similar usage scenarios: content management systems, blogs, event logging. The use of timestamp allows using this type of storage for the organization of counters, as well as the registration and processing of various data related to time.
The family column stores should not be confused with column stores. The latter are relational
databases with separate storage of columns (in contrast to the more traditional line-by-line data
storage)[
Document-based database management system
Document-oriented databases serve to store hierarchical data structures. They are used in content management systems, publishing, document search, and so on. Examples of this type of database are CouchDB, Couchbase, MarkLogic, MongoDB, eXist, Berkeley DB XML.
Databases based on graphs Graph databases are used for tasks in which data has a large
number of links, for example, social networks, fraud detection. Examples: Neo4j, OrientDB,
AllegroGraph, Blazegraph, In niteGraph, FlockDB, Titan [
Since the edges of a graph are materialized, that is, they are stored, traversing the graph does not require additional computation (like JOIN in SQL), but to nd the initial vertex of the traversal requires the presence of indices. Graphical databases generally support ACID, and also have di erent query languages, like Gremlin and Cypher (Neo4j).
As an alternative to the classical approach, there are NoSQL databases. For tasks in which UF is involved, the most common use of document-based one.
It has following advantages:
It works with unstructured data, which make possible to add new data eld with no additional cost.
It available to compromise nding in performance-reliability.
Working with ODM frameworks (Object-Document mapping) { alternative for ORM
[
The JavaScript support on server side.
As a working example, the authors used MongoDB. This choice is due to prevalence and
testing in high load projects[
Based on benchmarking top NoSQL databases performance tests conducted by "End Point
Corporation", the authors systematized MongoDB performance indicators f or various hardware
con gurations, which were summarized in the table 1[
The performance comparison experiment was conducted in the cloud services of "Amazon Web Services EC2", which provides an industrial platform for systems that require a horizontal extension of the architecture, such as distributed non-relational databases. In order to minimize the errors in measurements related to the current load of "Amazon Web Services EC2" services, each set of test scenarios was played three times, at least 24 hours apart, using newly created clusters with hardware con gurations described in the table 2 of hardware con gurations of cloud services "Amazon Web Services EC2":
As an operating system in the nodes used Ubuntu 14.04 LTS AMI in the HVM mode (virtual
hardware virtual machine) virtualization, customized with Oracle Java 7. For each test, an
empty database was used as the starting point. The client applications were programmed
to enter randomly generated information into the database. After the database was nally
populated, each of the test scenarios was executed sequentially. All clients performed requests
in parallel, and then waited for all operations to be performed and the corresponding results
obtained. The client software was supplemented by the installation of the YCSB free software
package designed to analyze the performance of NoSQL databases[
The SQL database with 5 millions entries for book instance and 10 millions for comments (two for each book entry) was used. And 5 millions entries for full (with comments inside view, also two comments on each book) book instance was used in MongoDB collection.
Three main scenarios were examined:
Find by comment
Find comments for the book
Since MongoDB stores the data immediately in the nal UF view, it will be absolutely identical for searching the book and requesting an extended output.
Averaged values over 10 experiments are used to except the non-deterministic in uence of external factors.
As can be seen from the results(Figure 3 and Table 3) of the research, the search query in the collection of the MongoBD is more e ective than the search in the MySQL. The most e ective scenario is to search for comments on the book, that is, additional information related to the main entity. The speed of work of MongoDB in this case surpasses MySQL almost 5 times. Nevertheless, it is important to understand that if the search target is additional information(comments here), the velocity of the query for MongoDB will be greatly worse then traditional relational databases.
The conducted research allowed to draw a conclusion about the expediency of using documentoriented databases for storing large amounts of data for indexing purposes with a small number of supported links or their absence. Feasibility is con rmed by the fact that the use of documentoriented databases for storing large amounts of data for indexing with a small number of supported links or their absence allows you to select the optimal con gurations of computing systems in the framework of current and future business tasks, with the possibility of horizontal and vertical scaling, in conditions better performance than relational equivalents.