Data quality is an unavoidable issue in Data Warehouse systems due to their nature. Quality of data is compromised throughout the Data Warehouse (DW) lifecycle. This work addresses the completeness data quality dimension applied to DW hierarchies, which refers to the existence of a parent for each data value in a hierarchy. An incomplete hierarchy can result in data errors when applying aggregations and navigating cubes. The detection of incomplete hierarchies makes the user aware of the problem, avoiding biased decisions. As data quality is context-dependent, proper metrics should be defined for each domain and use case. We propose a generic data quality rule to assess hierarchy completeness, using OWL domain ontologies as context. With this approach, we obtain a data quality metric that adapts itself to any application domain.
Data Warehouses (DW) are populated with heterogeneous sources, which are re-structured to be queried from a multidimensional point of view. Multidimensional operations allow navigating the DW dimensions hierarchies, through the aggregation of data. As the quality of external sources is unknown, the user may obtain dirty data as the result of multidimensional operations.
Many aspects can be taken into account when trying to define and measure the quality of data. These aspects are called DQ dimensions, while the mechanisms for measuring DQ dimensions are called DQ metrics [1].
This work is part of an ongoing project that proposes a mechanism for detecting DQ problems in DW systems, considering context [2]. The general scenario consists of a final user who wants to measure DQ over her existing DW. Our approach is to use one or many domain ontologies as context to define possible DQ metrics. One of the main challenges is that the DW and the domain ontologies do not necessarily have the same structure, and this mismatch must be solved in order to use the ontology as context for the DW. Well known knowledge graphs such as Yago [3], DBPedia [4], etc. can be used as domain ontologies to get available contextual data. In our approach DQ metrics are defined through DQ rules. teristic of the context we use in our approach. In the literature review presented in [11] about context dependant DQ, authors find a lack of formalization for context in DQ, pointing out that many of the proposals do not even define the context. [ 12]. Hurtado-Mendelzon’s multidimensional model [13] is the foundation for many other existing specifications. It comprehends the main DW concepts. One particularity of this model is that enables the representation of heterogeneous dimensions. As a consequence of heterogeneous dimensions, using this model generates the need to handle the problem of incomplete hierarchies [14, 15]. Our approach tries to detect whether a hierarchy incompleteness is due to a DQ problem whereas these works focus on completing the hierarchies without considering the DQ point of view.
The main contributions of this work are: (i) a solution for detecting the problem of incompleteness in DW hierarchies, based on the context, and (ii) the specification of a DQ rule for hierarchy incompleteness detection (completeness rule), together with the specifications of the needed system components and relationships.
The rest of the document is structured as follows: in Section 2 we present some definitions that are necessary to introduce our completeness rule in Section 3, in Section 4 we present an example and in Section 5 its implementation. Finally, in Section 6 we conclude.
We introduce the DW and context formal specifications and the correspondences between their elements.
DW Formal Specification The specification of the DW is based on Hurtado-Meldenzon’s multidimensional model [13]. We made some minor adaptations to their model for it to meet our necessities: in this paper, we introduce the use of tuples instead of atomic elements as members of categories. We only introduce the notation needed for understanding this work, details about the formal model are available in [13, 16].
A dimension schema is a tuple = ⟨, ↗⟩ where: • = {1, . . . } is a finite set of categories. • Each = ⟨1 , . . . , ⟩ is a list of attributes,
representing the characteristics of the category. • ↗: × s a binary relation between elements of , which represents the hierarchies of dimension , and ↗* its transitive closure.
A dimension instance for , is a tuple = ⟨ , <, ⟩, where: • : → is a total function that for each
value of returns its category. • <: × is a binary relation between elements of that represents hierarchy instances of , and is consistent with ↗. <* is its transitive closure.
In this work it is not necessary for relation < to be homogeneous nor strict (every member of a category has exactly one parent in each of the categories above). Both conditions are required in [13] but not in [16].
Context We use OWL domain ontologies to represent context. Given an OWL ontology , we consider its classes , its object properties , and its data properties .
• = {1, . . . } • = {1 . . . }, where ( ) and
( ) are its domain and range. • = {1, . . . }, where ( ) is
a class and ( ) is a data type.
An instance of is composed of its classes, data properties, and object properties instances.
• Class instance. An instance of is a set of individuals = {1 , . . . , }. Therefore, = 1 ∪ . . . ∪ is an instance of . • Data Property instance. An instance of is a relation between an instance of () = and a value of type (), ⊆ × (). Therefore, ⊆ 1 ∪ . . . ∪ is an instance of . • Object Property instance. An instance of is a relation between an instance of () = and at least one instance of () = , ⊆ × 2 . Therefore, ⊆ 1 ∪ . . . ∪ is an instance of .
DW - Context Mappings The correspondences between DW and ontology elements are binary relations, called mappings (only the mappings needed for the completeness rule are presented). The first argument is the DW element and the second one is the ontology element.
Consider a with dimension schemas, its corresponding dimension instances, an ontology and its instance , as presented in 2.
A value of an attribute of a DW tuple is mapped to an instance of a data property.
⊆ (11 ∪ . . . ∪ || ) ×
Where = [1]∪. . .∪ [] and is the num• a non empty finite set of tuples called members. ber of attributes of category ( ). Precondition: • = ⟨1, . . . ⟩ is a tuple of the universe, there must exist a mapping at schema level between the each is its value for attribute and is the corresponding attribute and the data property schema. length of the list of attributes of ().
A pair of categories of a DW hierarchy is mapped to an object property.
⊆ ((1 × 1) . . . ∪ ( × )) × Precondition: there must exist a mapping between each of the categories and ( ) and ( ) respectively. Furthermore, mapped categories must be related through ↗* .
An element of the DW can be mapped to more than one element of the ontologies.
Our formal DQ rule for hierarchy completeness is based on the definitions presented in section 2.
According to [5] an incomplete hierarchy is one in which some of its levels have missing values in one or more instances. These values can be nonexistent or unknown. The proposed completeness rule aims to detect values in a category that do not have a corresponding value in the immediate superior category and to check whether or not this situation is consistent with the context. To do so, we detect the tuples that do not have a parent in the immediate superior category and check their mapped values in the ontologies, used as referential, to see if there exists a corresponding value there.
Considering two categories and ′ of dimension such that ↗ ′ , we define the set of tuples of category that do not have a corresponding tuple in category ′ as: Δ ,′ = { ∈|() = ∧ ¬(∃′)((′) = ′ ∧ < ′)} (1) To define a completeness rule based on the context, two preconditions must hold: These preconditions are shown in Figure 2. The Figure shows a mapping between the pair of categories ( 1 ( ), 2 (′ )) and the object property : (( , ′ ), ). In addition, attribute value 4 of tuple = ⟨4, 5, 6⟩, which belongs to category , is mapped to instance of data property , whose domain is the class 1 = ( ) : ([1], ). If all of the previous conditions hold, then the completeness rule We consider a Cinema DW that stores the quantity of tickets sold and contains a dimension called Movie. The hierarchy goes as Movie → Subgenre → Genre → All. Figure 3 shows an instance of Movie dimension. An ontology populated with data from Yago knowledge graph for movies, shown in Figure 4, gives context to Movie dimension. The ontology contains information about Movies, Subgenres, and Genres and their relations. between category and ′ can be defined as shown in equation 2.
∈ Δ ,′ ∧ (, _) ∈ → ,′ (, 0) ̸∈ Δ ,′ ∨ (, _) ̸∈ → ,′ (, 1) (2) The presented rule may be aggregated to measure completeness for a complete hierarchy and not only between a pair of levels. ⟨2, Toy Story⟩ ∈ ∆ , and (mov3,subg2) ∈ : hold, hierarchy completeness between and categories for this movie is 0.
⟨3, Toy Story⟩ ∈ ∆ , ∧
(mov3,subg2) ∈ : → (4)
In this work we proposed a general mechanism for DW , (⟨3, Toy Story⟩, 0) hierarchy completeness assessment, considering context For movies titled Coraline and The Lion King the as domain ontologies. We presented the specification of a result of the application of the completeness rule completeness rule, as well as the specifications of the conis 1, because ⟨1, Coraline⟩ ̸∈ ∆ , and cepts needed for supporting the rule, which are the DW, the domain ontology and the mappings between them. ⟨2, The Lion King⟩ ̸∈ ∆ , . We also presented an example and an implementation that show how the proposed mechanism is applied. 5. Implementation and Validation We believe that our proposal has a level of abstraction in the formalizations that enables evaluating hierarchy We use an extended Datalog language with aggregation completeness for any DW in any context. functions in order to allow the expressivity of diferent This work is being extended building solutions for DW elements. Even though we used pyDatalog in our other DQ dimensions. Future work includes applying implementation [17], we present our solution using stan- our proposal to a real world case. dard Datalog syntax in the sake of clarity.