Graph databases allow direct data insertion without predefining a schema. While flexible, this often leads to a relaxed schema, especially without the usage of schema checking component, such as constraints. Consequently, heterogeneous structures emerge, requiring identification during data exploration. Furthermore, evolving such a relaxed schema can increase this heterogeneity even more. In this paper, we present Graph Data Profiles (GDPs) which capture initial heterogeneity by extracting statistical metadata (e.g., the number of labels and property keys for each node and edge). Additionally, these include uniqueness constraints (i.e., unique column combinations), existence constraints (mandatory or optional properties), missing edges, type constraints and valid inclusion dependencies. To support schema evolution and transformation, we introduce the GDP Dif, which underlines changes in heterogeneity. Thereby, inclusion dependencies help to detect in-version redundancies and evolution operations such as renaming or moving between versions. In contrast to other works, we address both schema evolution - essential for aligning a schema with changing requirements - and schema transformation, an frequent task in database monitoring. Our approach includes a cost calculation based on (1) a formalization of evolution or transformation operations and (2) heterogeneity metrics derived from the GDPs. Subsequently, GDPs ensure the detection of heterogeneity during data exploration, while the GDP Dif highlights the impact of an operation on the schema. This ensures user-informed decision making, supporting users to efectively clean unintended heterogeneity, such as typos.
Schema evolution is essential to retain the functionality of a system with changing external and internal requirements. Thereby, even simple evolution operations can cause complex underlying schema changes, especially when multiple software engineers operate on a single graph database. Furthermore, users cannot estimate whether an evolution operation (add, rename, delete, copy, move, split, merge) afects all selected data, a subset of the data, or none at all. Since schema-less databases allow direct population without a predefined schema, issues regarding heterogeneous structures such as typos frequently arise.
Both schema evolution and transformation share identical components but are distinguished by which component needs to be derived. Evolutionary tasks take a source schema and evolution operations to generate a target schema. In contrast, schema transformation compares a source and a target schema to identify schema modifications. We envision a modular framework that integrates both scenarios, aiming to detect, visualize, and estimate the impact of schema evolution and transformation on heterogeneity.
To address the issue of relaxed schemata, for instance, in collaborative environments, we present Graph Data Profiles (GDPs) to capture heterogeneity in an initial data exploration phase by extracting statistical metadata (e.g. number of occurrences for labels) and diferent constraints (e.g., uniqueness, inclusion dependencies). Besides illustrating and quantifying heterogeneity, the GDPs enable userinformed decisions during schema evolution or version comparison (schema transformation). More precisely, they underline potential syntactic and semantic errors. These insights allow users to align the schema with their intentions by cleaning accidental errors with our tool Nautilus. Thus, our approach ofers a modular solution for data exploration, schema evolution, and transformation.
To compare changes in irregularity a Graph Data Profile Dif (GDP Dif) is created (see Table 1), enabling users to track how the degree of heterogeneity changed through a preview option during schema evolution. In terms of transformation, the Difs give insights into how heterogeneity was modified between versions. Additionally, a formalization for each evolution operation (see Table 2) is provided. These formal definitions are combined with the heterogeneity metrics stored in the GDPs to estimate an operation’s complexity. This is essential, as heterogeneity substantially impacts the costs of schema changes.
Contribution We propose GDPs to display heterogeneity in schema-less graph databases by extracting statistical metadata to make implicit structures explicit during data exploration. Moreover, the GDPs incorporate constraints, namely uniqueness, existence, type constraints, and inclusion dependencies. Users benefit from GDP Difs when monitoring pipelines, as they depict changes in the degree of heterogeneity between versions. During schema evolution, a preview of the emerging GDP Dif alongside the operation’s cost is displayed, illustrating its impact. Inclusion dependencies, hereby, support an automatized detection of adjustments, caused by rename or move operations, and in-version redundancies emerging when copying. Due to the modularity of our framework, it is applicable for both schema evolution and transformation.
Schema Evolution Schema Evolution is defined as the process of modifying a schema to align
with new requirements. Due to its importance, it is subject in a variety of works [
Schema Transformation Schema transformation is a generalization of schema evolution, and can
take place between two arbitrary schemata or databases. It requires schema matching (to identify
diferences between a source and target schema) and mapping strategies (to transform a source database
into a target databases). Various matching techniques are categorized in [
Graph Data Profiling Data profiling aims to gain insights into a dataset. In [
Data Quality Marchesin et al. [
Source Graph Target Graph
Execute the
Defined BE Operation 1
Profile Extraction Algorithm
User
Cost-Aware Schema Evolution & Transformation 2 Source Graph Data Profiles
AT Target Graph Data Profiles
Define
Evolution AE Operation
Profile
Comparison 3 Algorithm
Identify
Evolution
BT Operations 4 Graph Data Profile Diffs Visualized for User-informed decision making Automatized Processes Mandaory Interaction
Calculate
Costs per 5 Operation
Evolution
Operations 6b Target Graph
Schema & 6aProfile Diffs
Use Case 1: In-Version Heterogeneity In a medical company, multiple software engineers simultaneously manage a graph database without schema constraints, leading to irregularities. To detect these variations, a data analyst uses our GDPs to clean unintentionally created variations by defining evolution operations in our evolution language Geo. Subsequently, data quality is improved by homogenizing the schema in alignment with user needs.
Use Case 2: Heterogeneity Between Versions In the second scenario, a data analyst compares diferent database versions. Our framework thereby, automatically generates GDP Difs to identify transformation operations including their costs. This allows the analyst to quantify whether heterogeneity increased e.g. by adding new data variants or decreased through homogenization. Understanding schema changes when monitoring or optimizing pipelines is essential, for instance to draw conclusions for similar pipelines in the manner of a knowledge base.
Interim Conclusion Both scenarios outline how error-prone schemata in collaborative environments are. Thus, illustrating the need to detect heterogeneous structures through our GDPs. Moreover, GDP Difs ensure the estimation of an evolution operation’s impact and costs. During monitoring tasks, the framework assists users by automatically detecting transformation operations, reducing manual adjustments and overcoming a missing documentation of the evolutionary process.
Figure 1 shows the workflow of our envisioned, modular framework. Blue elements indexed with E are schema evolution specific, turquoise elements marked with T illustrate schema transformation, whereby gray components are part of both tasks. In Step 1, a source graph database serves as input, followed by the extraction of the schema components required to generate GDPs in Step 2. For schema transformation, the GDPs are extracted and generated for both the source and target graph data (Step A ). In contrast, for schema evolution, the GDPs are visualized to the user – indicated by the eye icon – to depict in-version heterogeneity, i.e., synonyms or typos. Based on the information on heterogeneous structures in the GDPs, the user can define an evolution operation (Step A ), for instance, to decrease heterogeneity by renaming labels with typos. Thereafter, the GDP Difs are generated in Step 3 and displayed in Step 4. In Step 5, the metrics stored in the GDPs and the formalized operations (see Table 2) are incorporated in the cost functions to illustrate the complexity of evolution or transformation operations. The output consists of an interactive schema visualization and the GDP Difs for each entity, i.e., nodes and edges, (Step 6a) together with an evolution or transformation operation log-file (Step 6b). During schema evolution, the user then can decide whether the operation is executed (Step B ) or not. Moreover, the GDP Difs incorporate inclusion dependencies to automatically resolve ambiguities when comparing versions in Step B .
We present a modular framework that extends the graph evolution tool Nautilus – implementing our evolution language Geo (Graph Evolution Operation) [34] – with Graph Data Profiles (GDPs). These depict heterogeneous graph data and form the basis for predicting an operation’s efort. Heterogeneity Presumptions As a result of the modeling freedom ofered by schema-less databases, we distinguish between three ways of storing graph data. (1) Analogous to relational databases, users can predefine a non-relaxed schema in a schema-first approach, utilizing constraints. (2) Just like in Use Case 1, the data can be directly inserted without using schema constraints. (3) The third scenario is a partially schema controlled usage, where homogeneous structures are forced by constraints, while heterogeneous parts are kept schema-free. If schema-first approaches allow relaxed structures like optional properties, they also belong to this class. Our approach focuses on the third case, as it is often unclear whether observed heterogeneity is intentional. Hereby, our GDPs support users in homogenization – for instance, by enforcing existence on formerly optional properties – and increasing heterogeneity – e.g., by adding variants or evolving subsets. This allows users to choose between homogeneity of data and structural variations in the data when evolving graph data. Additionally, the GDP Difs give insight into heterogeneity modifications during version comparison like in Use Case 2.
Graph Data Profiles (GDPs) To make heterogeneous structures explicit as needed in Use Case 1, we present GDPs incorporating statistical and structural metadata – extracted by the algorithm used in Step 1 of Figure 1 – for each label of a given graph database. The term label is used for both node labels and edge types. Since relaxed schemata are common in collaborative environments, detecting irregularities during data exploration provides deep insights into the dataset’s structure. Emphasizing heterogeneity additionally depicts candidates for data cleaning, thereby enabling the analyst to optimize the schema in alignment with data quality aspects. Distance metrics display syntactic and semantic in-version candidates, whereby valid inclusion dependencies aid identifying identical information. Furthermore, constraints highlight homogeneous structures that could be schema-controlled. GDP Dif Table 1 depicts an exemplary GDP Dif, serving as a preview before executing an evolution operation. Consequently, supporting user-driven decision making by depicting schema changes in the context of heterogeneity. Such a preview is also beneficial to perform data cleaning (Use Case 1). When monitoring pipelines, the GDP Difs facilitate a deeper understanding of schema modification and each operation’s efect on the dataset’s heterogeneity, as described in Use Case 2.
The GDP Dif in Table 1 illustratively shows Client nodes, formerly labeled Customer. The p in the header shows that the label Customer was deleted. Changes in structure metadata such as occurrence are icon-coded by ( for increasing and " for decreasing numbers. 25.56% of the Client nodes are additionally labeled KeyAccount, i.e., depicting multi-labeling. The Dif shows a decreasing number of multi-labeled Clients, caused by increasing their number from 65 to 90. Furthermore, syntactic and semantic heterogeneity metrics – namely Levenshtein to detect typos and Wu-Palmer for synonyms – are shown to detect structural irregularities. Precisely, the variation customer was identified for the Client label.
In order to analyze properties, constraints need to be made explicit. This includes deriving uniqueness constraints, resembling unique column combinations, together with existence and type constraints from the data. Moreover, valid inclusion dependencies serve an in-version detection of identical data, caused by copy operations, as it is the case for . _ ⊆ . _ . Here, the GDP Dif illustrates that this error was later cleaned by p under Levenshtein. A relaxed candidate generation gives insights into renamed or moved elements (syntactic and semantic metrics). In terms of inclusion dependencies, false positives might occur, e.g., ._ ⊆ . . Nevertheless, user-centric cleaning decisions ensure robustness, while depicting potential overlaps. Another key aspect is illustrated in the super-type cell. Here the associated label for each property is depicted. Solely for node entities all associated edges are displayed, with the objective of identifying syntactic and semantic errors, as it is the case for p BOUGHT, as well as to display missing and required edges (= existence constraint).
̂︀
Based on the GDP Dif, the following representative conclusions can be drawn: The deletion of the associated edge labeled BOUGHT, formerly connecting Customer and produtc, a variant of the (Customer)-[BOUGHT]→(Product) pattern, represents a homogenization action. Consequently, a GDP Dif-driven analysis reveals a renaming process. In the properties section the key firRst_name from Cusotmer was added, resembling a move operation.
Cost Functions Besides GDPs, we envision a cost model to capture the impact of each operation executed during schema evolution or transformation. In this context, heterogeneity can cause severe structural changes; consequently, the cost model accounts for both an operation’s complexity and its efect on data irregularities.
For the complexity estimation, each evolution operation is formalized in an implementation near manner, as exemplified in Table 2 for add. This allows users to predict the complexity of the Cypher code. In alignment with our evolution language Geo, every operation is partitioned by entity types (nodes and edge) and features (label, property). In our notation, represents the aligning Cypher command, in this context CREATE or MERGE. is the finite set of available labels, subdivided into node labels and edge labels , whereby and resemble a precise instance. Properties are defined as key-value pairs × , whereby is a Cypher variable to refer to a defined pattern, like in CREATE(n:Client). Since Geo translates directly into Graph Manipulation Operations (GMOs), the formalization is based on the aligning Geo statement.
To ensure a cost-aware schema evolution and schema transformation, the cost function are further
refined by the following aspects derived from the metadata stored in the GDPs and categorized according
to [
• Structure metadata: The number of afected entities to estimate pattern complexity. • Cardinalities and value distribution: Analysis of whether constraints become invalid as heterogeneity increases or whether new constraints emerge by homogenizing former irregularities. • Paths metadata: Identification of important nodes with numerous edges, prone to cause a structural break upon their modification. • Patterns, data types & domains: Use of syntactic distance and semantic similarity to detect potential heterogeniouse structures. • Functional: Deriving graph inclusion dependencies to detect renamed, copied or moved structures in and between versions or databases.
For each aspect, a weight is defined in association of its efect on heterogeneous structures, whereby a value of one resembles homogeneity. Costs are sensitive to irregularities, thus, requiring the composition of GDP metadata and operation complexity. Our approach considers cost-estimations for monitoring data engineering pipelines and supports user decisions via a GDP Dif preview together with the emerging target schema in terms of schema evolution. The tool displays the resulting costs approximation to the user.
Syntactic Accuracy The integration of GDPs as well as the GDP Dif preview aim to support data cleansing by improving syntactic accuracy. This is accomplished by depicting candidates for typical errors occurring in relaxed schemata, such as typos, by measuring the syntactic distance. Subsequently, users can utilize the evolution form to clean the data accordingly (see Use Case 1). Since users cannot estimate the impact of an evolution operation on the dataset, we propose a GDP Dif preview serving as control mechanism before actually conducting the operation on the original dataset. Thus, the GDPs depict heterogeneity during data exploration caused by the schema-less environment. In addition, the GDP Dif allows estimating the impact of schema evolution and gives insight into irregularity changes during schema transformation (see Use Case 2).
Semantic Consistency The GDPs specialize in detecting semantic errors to facilitate user-driven decision making, thereby, increasing data consistency. To achieve this, the semantic similarity between entity types and properties is extracted, allowing refinements of structures that emerged unintentionally during schema evolution. On top of that, constraints are included to illustrate homogeneous parts suitable to be put under schema control. Regarding data dependencies, GDPs store information on valid inclusion dependencies, which provide insights into (1) identical in-version information as well as (2) evolution or transformation operations like rename, move or copy. Each GDP represents the left side of an inclusion dependency. Moreover, the GDP Difs make increases and decreases in semantic irregularities during monitoring tasks visible.
Nautilus implements our evolution language Geo, closing the gap for an evolution language for property graphs. Geo aims to widen the range of users by intuitively defining each operation, requiring no prior knowledge of Cypher. The implementation is available on GitHub1. The tool includes a form to define the renown evolution operations add, rename, delete, copy, move, spit, merge and the graph-specific transform operation (i.e., converting a node into an edge and vice versa). Moreover, a ifrst version of the GDPs is integrated in a data exploration option. This includes information on label occurrences; partitioned by nodes and edges. For each label all associated properties of the initial and latest schema versions are depicted, allowing a manual comparison.
We plan to extend the current data profiles in Nautilus by integrating the GDPs. This will highlight irregularities in the source graph and enable users to clean variations by evolving the schema with our evolution form. After defining the evolution operation, the resulting GDP Dif (see Table 2) is displayed in a preview along with the estimated costs. This supports users in estimating the impact of an operation on the schema and its underlying data before executing it on the original data. To evaluate the accuracy of the presented GDPs, we will generate a dataset with heterogeneous structures, such as optional properties, missing edges, and variations, serving as gold standard. Initial tests revealed a high accuracy of the current data profile generation. In the future, we will extend the evaluation by determining precision, recall and F1 score derived from true positives, false positives, and false negatives.
Nautilus-Graph-Schema-Evolution of a graph evolution language: https://github.com/DominiqueHausler/
node add edge label property
Formalized Graph Manipulation Operation ( : { }) with ∈ {CREATE, MERGE}, ⊆ × , ∈ () − [ : { }] → () with ∈ {CREATE, MERGE}, ⊆ × , ∈ SET :
outputting ′ () = () ∪ {} SET . = with ′() = () ∪ {(, )}
"add node with label" label "add relationship with type" label "add label" label "to node with label" labels "add" ("unique")? ("mandatory")? "property" property "with datatype" datatype "to node with label" label
Our presented framework facilitates user-informed decision making in the terms of schema evolution and transformation – two closely intertwined approaches. For schema transformation, the framework outlines heterogeneity changes in form of Graph Data Profile Difs (GDP Difs), additionally calculating the costs for each transformation operation. Thus, the framework can be seamlessly integrated into monitoring components of data engineering pipelines. Evolutionary tasks benefit from a GDP Dif preview, allowing users to estimate an operation’s impact and costs before executing it on the original data. Additionally, GDPs highlight potential errors through heterogeneity metrics, enabling the user to improve data quality by defining evolution operations to clean them. To determine whether heterogeneity in or between versions increased, decreased or remained consistent, the cost functions incorporate the information stored in the GDPs alongside the formalized operation. Users, therefore, are provided a precise estimation of an operation’s impact together with a comprehensive analysis of relaxed schemata in schema-less graph databases.
The work of Dominique Hausler has been funded by Deutsche Forschungsgemeinschaft (German Research Foundation) – 552691270.
The logo was generated with DALL-E 3. ChatGPT and AI Studio were used for rephrasing and to find synonyms. in Neo4j, in: ER (Companion), volume 3618 of CEUR Workshop Proceedings, CEUR-WS.org, 2023. [34] D. Hausler, M. Klettke, Nautilus: Implementation of an evolution approach for graph databases, in: MoDELS (Companion), ACM, 2024, pp. 11–15.