Unlike traditional databases, Big Data stored as NoSQL data resources. Therefore such resources are not ready for efficient use in its original form in most cases. It is due to the availability of various kinds of data anomalies. Most of these anomalies are such as data duplication, ambiguity, inaccuracy, contradiction, absence, the incompleteness of data, etc. To eliminate such incorrectness, data source special cleanup procedures are needed. Data cleanup process requires additional information about the composition, content, meaning, and function of this Big Data resource. Using the special knowledge base can provide a resolving of such problem.
The problem of data quality remains to be topically for a long time in various areas of
data processing. Nowadays it is especially considerable in Big Data technologies and
analytics [
Data anomalies in the Big Data resources are presented by such phenomena as absence,
duplication, ambiguity, lack of meaning, inaccuracy, incompleteness, unreliability,
inconsistency, etc. [
Classification allows us to choose the best way to eliminate it of data anomalies.
Each data anomaly would be detected for a data item by checking its correspondence
to some predefined requirements. Such requirements have to be described as conditions
of data value comparison. For detection of described above types of anomalies, such
conditions may be used [
Type of data anomalies value not present value is unknown value is invalid value is unreliable value is duplicated value is ambiguous value is not accurate enough value is an incomplete
Data item check condition data item Is Null data item Is not in <interval> data item Is not in <set> Count(data item)>1 data item != data item data item != value Number(data item)<value This list may be continued or changed according to specifics of processed data.
NoSQL Big Data Resource
Invalid Data values Classification opf Data
Anomalies
Methods of Data Anomaly elimination Knowledge-Based Data cleanup process Ready to use Big Data Resource
Knowledge: How to eliminate certain Data Anomaly
for certain Data Items
Fig. 1. General schema of producing and use of Big Data resource cleanup knowledge
Usually, to eliminate data anomalies, the most commonly used techniques are the
removal, ignoring or re-defining of an appropriate data element, using the average, most
likely, estimated or surrogate value, duplicate values remove etc. [
The principal problem of efficient Big Data resource cleanup is how to build exact and complete descriptions of the methods and rules of elimination of data anomalies. This may be presented as a specific knowledge set includes: 1. a description of Big Data resource, its properties, and all included data items, 2. description of data anomaly types, 3. rules of search and recognition of invalid data values, 4. description of the methods of fixing corrupted or invalid data.
General schema of producing and use of such knowledge presented on Fig. 1.
The process of anomalies of data elimination in the Big Data resource performs as a replacement of the incorrect value by the new value, which define by a special procedure. In the general case, the value vij of some data unit Vi, which is formed to eliminate its anomaly, depends on the nature (category) of data anomalies – Uk and the method of its elimination – Sl. The procedure for defining a new data value can be describe as a sequence of steps of kind
Vi → Uk → Sl → vi . (1) That mean: invalid data value Vi of category Uk by using of method Sl have to be replaced by value vi for elimination of one case of data anomalies in some resource. The same transformation can be presented as a mapping
vi =Ф(Vi , Uk, Sl), where Ф is a function for define new value for invalid data item using its category and corresponding method. These actions are perform during general data cleanup process of Big Data source. Using ontologies for Big Data cleanup Because it is necessary to have exact and complete descriptions of the correspondence between anomalies in the data resource, their classification is needed. As well a formal description of the ways to eliminate such anomalies is needed. Knowledge base may be uses for such purpose in the set of tools for Big Data sources cleanup. The core of this knowledge base may be formed by an ontology of type
OO=< CO, RO, FO>, where CO={CV, CU, CS} is the set of concepts (classes), which include such subclasses: CV is entity set (subclass) for a presentation of data units in the Big Data resource to be processed, CU is the set of entities that describe the types and nature of each of the data anomalies presented in the Big Data resource, CS is the set of entities for the description of data anomalies elimination methods (data anomaly problems solving); RO ={RVU, RUS, RVUS} is a set of relations between the above-defined concepts that include three subsets: (2) (3) RVU is a subset of binary relations between data values of CV and types of anomalies in Big Data resource CU, RUS is a subset of relation between types of data anomalies of CU and methods of anomaly elimination CS, RVUS is a subset of ternary relations between data values and methods of data anomaly problem solving; FO ={ FV, FC } is a set of axioms (rules). Two kinds of rules are needed for data cleanup process. Each one describes the steps which must be performed for solving of data anomaly problem, The first kind of rules – FV include rules for data validation in presented Big Data resource. The second subset – FC consist of rules defines the method (from the set CS) of data cleanup. This method directly depends on certain data values (presented by the set CV) and a certain type of anomalies of this data item (described by the set CU).
The resource of Big Data needs to be deeply investigated to solve the problem of data
cleanup. The main goal of this investigation is to get the answer to the questions:
what data items and values are needed to solve some defined task;
what is the structure and content of the data source used for this purpose;
what kinds of problems with data are possible in the presented resource;
what factors have an influence on data quality in the resource;
what are data quality criteria and requirements;
what are the methods to fix corrupted or invalid data items;
how some methods of bad data fixing would be used to correct some data items.
Whether obtained all necessary information about input Big Data resource, we can
formalize such knowledge as units of certain special ontology [
Data Set Data Item Data Value
CV
Big Data resource Data Anomalies
Types CU
RUS
RVUS Data Anomalies Elimination Rules
FC Data Validation
Rules FV
Data Anomalies
Elimination Methods CS Fig. 2. Structural model of ontology for Big Data resource cleanup Generalized structural model of ontology for Big Data resources cleanup presented on Fig. 2. So, Big Data resource would be described by three concepts of the ontology according to the developed model. The first describes certain data as a hierarchy: "Data set" (table, collection, etc) –"Data item" (column, field, element) – "Data value". The second concept describes the types of potential anomalies of data for processed data resource. And the third one - the ways to fix corrupted or invalid data in the resource. Two types of axioms describe the rules of data check to find certain anomalies and the rules of various data anomalies elimination. The arches describe possible relations between defined concepts and the rules. As a remark, the relation RVUS seems redundant, because of its matching to RVU and RUS. Bet such overage allows defining of unambiguous correspondence between the data set and possible methods of data anomalies elimination.
According to proposed principles of the generalized model, the ontology for any Big Data resource is developed. It is necessary to determine the specific values of concepts, relationships, and rules in accordance with the content of the resource data when performing this action. As a result, the primary version of ontology for Big Data resource cleanup will be obtained. As the next, we can create an appropriate knowledge base for the data cleanup tools based on created ontology.
So, in the above-described way we can obtain the set of tools needed for efficient solving of Big Data resource anomalies elimination. Therefore, an ontology created in accordance with the principles described above can be used as the basis of the knowledge base for intelligent tools of the Big Data resources cleanup. 3
The use of ontology as the core of the knowledge base of the Big Data Cleanup tools determines the peculiarities of the process of solving data problems (Fig. 3). So, construction and tuning of basic ontology is a principal prerequisite for any Big Data source cleanup process. This stage requires a number of actions based on expert knowledge. Expert knowledge provide definition and creation of basic ontology parts just like that.
Description of the set of concepts, which corresponds to data values and data units CV, according to the set of requirements of ontology construction. The concept CV for the presentation of data value may be defined as a result of hierarchic taxonomy – "Data Set -> Data Item -> Data Value" according to the proposed structural model of ontology.
Construction of the set of definitions of data anomalies CU, which are characteristic of the given Big Data source. The list of most common data anomalies is presented above.
Definition of the set of methods CS for data anomalies elimination. Most of
wellknown and often used are such methods of solution of data anomaly problem [
Definition of relations between concepts - data items, data anomalies and methods
of data anomalies elimination – RVU, RUS, RVUS. The most suitable format for
definition relations between concepts is RDF triplet [
So, these relations may be formed in such a way. For relation RVU the triplets may be constructed by the scheme "Data Value (CV)–
Have the Anomaly – Anomaly Type (CU)". For relation RUS – by the scheme "Anomaly Type(CU) – Eliminated by – Method (CS)". For relation RVUS the triplets have to be constructed by the scheme "Data Value (CV)– Anomaly Type (CU) – Method (CS)". Here, the concept CU execute a function of the predicate. 5.
Rules of data validation for its anomaly detection – FV. This rules also may be
presented as RDF triplets [
Rules of data anomalies elimination method using for various data values and various data anomalies types – FC. These rules unambiguously correspond to the relation RVUS between data items, data anomalies, and methods of anomalies elimination.
Ontology constructed in this way may be used as primary ontology for knowledge base of Big Data source cleanup Framework.
When the base ontology is constructed algorithm of Big Data source cleanup can be applied for the first time. The first application may not give an effective result of data cleanup in the general case. It is because the knowledge base need the addition of new knowledge. General steps sequence of this algorithm is like the next. 1. For each data item with anomaly from class CV , the type of anomaly has to be qualified. Qualification of anomaly means determining anomaly nature and the way of its interpretation. As a result, any concept from category "kind of anomaly"– CU would correspond to any data item of CV. If this operation can't be executed complement of ontology is necessary (see step 4 of algorithm).
Using data item determination and anomaly type qualification, corresponding relation, and the rule of the anomaly of certain type elimination for certain data item would be defined. Generally, data anomaly elimination rule is the expression of type (Vi ^ Uk )→ Sl, (4) where, Vi is data item, Uk is a kind of data anomaly, Sl is method of data anomaly elimination (new data value definition). 3. At the nest step replacement if invalid data value has to be executed according to defined above relations and rules. When whole data resource is processed algorithm to be complete if not – return to step 1. 4. This step is need to recognize and fix the problem situation, appeared during the attempt of data resource cleanup. These problem situations can be categorized according to its origin: no description of data item or data value in the ontology; no description of anomaly type in the ontology; no description of method to eliminate some type of data anomaly; no description of rule to eliminate anomaly for particular data value; using of rule to eliminate uncertainty did not effect.
Big Data source cleanup system Input Big Data
Resource
Data Value Data anomaly qualification Data anomaly elimination
rule New data value
Knowledge Base/
Ontology
New knowledge generating/Ontology learning
If the situation is recognized and categorized, we can fix it by the specified way.
These ways are of define the new concepts for the data value, for data anomaly type or for data anomaly elimination method; define the new item in relation set for definition of correspondence between data value, data anomaly type, and anomaly elimination method; define the new rule of data anomaly elimination.
Step 1 of the algorithm needs to be repeated again after the execution of described over operations. 4
This example has been developed to show how to describe a piece of knowledge about
data, data anomalies, and data cleanup methods using ontology tools. RDF-OWL
technology [
The data set for exploring has been modeled as a NoSQL document-oriented database
using MongoDB JSON [
a) Data Values Class a) Data Anomalies and Data Cleanup Methods Classes The set of rules to describe how certain data values with certain data anomalies has to be fixed by a certain method (during Big data resource cleanup process) has been developed as well. RDF/XML format has been used for these rules presentation (Fig.6). By this way, we have constructed the primary version of the special ontology described above. Further, it is ready to be improved and developed for efficient support of Big Data resource cleanup processes. 5
An approach to solve the problem of quality of Big Data sources by their cleanup has been considered in the paper. We propose to solve the problem of such type by the development of knowledge-based intelligent tools. The peculiarity of the solution is using an ontology as a core of knowledge base for the Big Data resource cleanup framework. It is the principal difference between the proposed approach and the traditional methods of data cleanup. We consider the ontology as a special type of metadata, which describes Big Data resource, anomalies of data, corresponding methods of data cleanup and the relations between theirs. The developed approach gives us the such possibilities as to design special tools for intelligent Big Data resource cleanup; to make better procedures of data clearing of a Big Data resource; to accumulate for further use of knowledge and experience to solve data quality and data cleanup problem.
The principles and methods developed in the paper may be useful for data scientists at the processes of preparation of Big Data resources to analysis.