Database schemata are not static artifacts but subject to continuous change as new requirements daily occur and the modeling choices of the past should be updated or adapted. To this direction, multiple approaches available already try to keep multiple co-existing schema versions in parallel or model schema evolution through Schema Modification Operations, known as SMOs. However, to the best of our knowledge, in the era of big data, where thousands of SMOs might appear, it is really hard for developers to identify the modeling choices of the past and to explore how a specific column or table has been evolved. In this demo, we present VESEL, the first system enabling the VisuaL Exploration of Schema Evolution using provenance queries. Our approach relies on a state of the art database evolution language, and can eficiently answer queries about when a specific table or column has been introduced, how - with which SMO operation and why - which is the sequence of changes that led to the creation of the specific table/column. In the demonstration we will present the architecture of our system and the various algorithms implemented, enabling end-users to visually explore schema evolution. Then we will allow conference participants to interact directly with the system to test its capabilities.
As recent databases do not proactively support schema
evolution, developers have to migrate data from one database version
to another using error-prone and complex
Extract-TransformLoad scripts [
Independent of the database evolution language used,
complex evolution scenarios can easily lead to hundreds of schema
modification operations, e.g. the evolution of Wikipedia’s schema
through 171 schema versions [
In this demonstration, we present the VisuaL Exploration of Schema Evolution (VESEL) system, enabling developers to issue provenance queries, in order to actively explore the evolution of the schema. The database evolution language on top of which our system works is BiDEL, but the modular nature of the system makes it possible to be extended with other languages in the near future. It supports how (which schema modification operation), when (which version) and why (sequence of schema modification operation) provenance queries for a specific table or column and enables the graphical visualization of the results. As the selected database evolution language guarantees bidirectionality, queries can be issued using a recent or a past schema version and we can track the evolution of the specific table/column through multiple schema versions.
To the best of our knowledge, our approach is the only visual
approach enabling active exploration of schema evolution. A
preliminary work to the same direction is [
The remaining of this paper is structured as follows: In section 2 we describe system’s architecture, the various components used and the implemented algorithms. Then, in Section 3 we present an outline of our demonstration. Finally, Section 5 concludes this paper and presents directions for future work. 2
VESEL has been implemented using python and flask
microframework [
On the top layer, we find the graphical user interface (GUI), in the middle the query engine and at the bottom the persistence layer. In the sequel we introduce the various components used and we describe their functionality. 2.1
The whole process is started as soon as a file with BiDEL SMOs is uploaded to the system or an existing one is selected. The BiDEL database evolution language includes SMOs for creating a
As soon as the file is uploaded, it is parsed and stored to the internal database. Instead of the database, the uploaded file could also be directly parsed, but for reasons of eficiency we selected to store the SMOs in a database. Then the system is able to answer provenance queries. The queries available to the end-users are the following: (1) How. Which was the operation that introduced a specific table or a specific column? (2) When. At which version the specific table/column was introduced? (3) Why. Which was the sequence of operations that led to the introduction of a specific table/column?
When the answers are returned, both a graphical and a textual overview of the answer are presented to the user. A screenshot of the system is shown in Figure 2. Each node in the graph represents an SMO in its short form. When the mouse hovers on top of a node, it additional information is presented as a tooltip, whereas the whole list with the retrieved SMOs is shown on the right panel. A user can click on a specific SMO and refine the exploration based on this specific SMO. This will redraw the canvas and the 1https://db47122.inf.tu-dresden.de/ textual overview, enabling the user to actively move between versions and SMOs.
In the example, shown in Figure 2, we demonstrate the evolution of the "revision" table from the wikimedia dataset. The "revision" table has been the result of merging two tables, i.e. the "revisiona" and the "revision_old" table. Before that the table "revisiona" had been created out of the decomposision of the table "revisionb" into "revisiona" and "text". On the other hand the table "revision_old" was the result of a decomposition of the table "old". The graph continues until the tables "cur" and "old" are created for the first time. 2.2
The query engine receives requests by the GUI and subsequently answers the issued queries. To answer queries about the evolution of a specific table/column we define first the notion of an afecting schema modification operation .
Definition 2.1. Let t be a table/column of a schema version m, namely the Sm , and ∆ Sk,Sm , (k < m) be the sequence of schema modification operations between Sk and Sm . In addition, let δa (s) be the new schema information introduced with the SMO s and δd (s) the schema information that is being deleted using s. An SMO s ∈ ∆ Sk,Sm afects the table/column t , denoted by a f (t ), if t ∈ Sm and t ∈ δa (s) .
An afecting schema modification operation captures the way a table/column was introduced between two schema versions. The intuition behind this is that, as we usually use the latest schema version, we base our exploration on this version and we would like to track how a specific table/field has been introduced. 2.2.1
Assuming that we have the ∆ Sk,Sm available, it is trivial to identify the afecting SMO by just scanning the delta log once. This afecting SMO will be the answer to how provenance queries. As BiDEL guarantees that neither the rules for a single SMO, nor the version genealogy have cycles, the afecting SMO if it exists is unique.
2.2.2
Next, for answering when provenance queries we are interested in identifying the version at which the afecting SMO appeared. This is again trivial as for each evolution step, we have the complete list of SMOs and we can easily identify the specific evolution step that included the specific afecting SMO. 2.2.3
Presenting only answers to when and how provenance queries is not enough when complex evolution has occurred and we would like to see the sequence of events that led to the creation of a specific table or column. This sequence can be identiefid by answering why provenance queries. To be able to answer such queries we first define the change sequence of a specific SMO.
Definition 2.2. A change sequence for a schema modification operation s ∈ ∆ Sk,Sm , denoted by CSs , is the minimal sequence of schema modification operations in ∆ Sk,Sm such that s ∈ CSs and that when CSs is applied to version k we get the fields/tables participating in s.
As in essence the whole ∆ Sk,Sm is a set of changes that lead to the fields/tables participating in s, we are looking for the minimal sequence of SMOs, in the sense that one cannot remove any of the SMOs in it, and still when applied to the version k you get the fields/tables participating in s. As such, a change sequence is providing the minimal information in order to understand how the specific table or column has been evolved to reach version m.
Similarly to an SMO , δa (CS) is the new schema information introduced when CS is being applied to Sk and δd (CS) the schema information that is being deleted from Sk using CS.
It can be easily proved that a change sequence for the BiDEL language if it exists, is unique.
Next we present an algorithm that given a sequence of SMOs capturing the ∆ Sk,Sm , it produces the change sequence for an SMO s. The algorithm is shown in Algorithm 1. The idea is that the algorithm starts by an input SMO and identifies the tables/columns that are also added to the schema, possibly by deleting other schema parts. Then it searches for the SMOs that delete the added information in order to add new information and so on.
Algorithm 1 Compute Change Sequence
Output: CSs 1: procedure ComputeChangeSeq(s, ∆ Sk,Sm ) 2: CSs ← [] 3: CSs .push(s) 4: for each s ′ ∈ ∆ Sk,Sm do 5: 6: if δa (s ′) ∈ δd (CSs ) then
CSs .push(s ′) return CSs
The algorithm is immediately proved by construction and has a complexity of O(N ) where N is the number of SMOs in ∆ Sk,Sm . As such we argue that exploring the evolution of big schemata with multiple changes is feasible and eficient.
It is quite easy to adapt the aforementioned algorithm for presenting the change sequence for a specific table/column instead of a specific SMO. The idea is first to look for the afecting SMO and then to use Algorithm 1 for identifying the corresponding change sequence.
The aforementioned algorithm is not only applicable for visualizing the change sequence of a table/column of the latest schema version, but can be also used to visualize the change sequence of a table/column from an arbitrary schema version of the past. In that case, instead of providing as input to our algorithms the ∆ Sk,Sm we could provide the ∆ Sk,Sn where n < m. 2.2.4
In addition, since BiDEL guarantees bidirectionality and inversibility (e.g. the inverse of the merge SMO is the decompose SMO), we could get the evolution steps for a table/column of a past schema version till we reach the most recent version by providing as input the ∆ Sm,Sk to our algorithms. This can be directly constructed from the ∆ Sk,Sm using the inverse SMOs of those in the ∆ Sk,Sn . Although bidirectionality and inversibility are not properties of all database evolution languages, if available, they contribute to a richer exploration experience on the available schema versions. 3
For demonstrating the functionality of the system, the wikimedia dataset will be used. The dataset contains more than 170 schema versions with the corresponding SMOs and ofers a rich real dataset for experimentation. The demonstration will proceed in four phases. (1) Exploring wikimedia’s schema and raw evolution log. The demonstration will start by visualizing the latest version of the schema and exploring the various tables and columns. Then the raw evolution log will be opened and the dificulty to identify the actions that led to the generation of specific column/tables will be explained. (2) System overview. Then an overview of the system will be presented along with the corresponding components. The ideas behind how, when and why provenance queries will be explained and the algorithms will be briefly described as well. (3) Small tutorial. Next the users will be informed about the functionalities provided through the graphical user interface and some examples will be executed demonstrating various how, when and why provenance queries. More specifically, we will demonstrate how and when queries for selected tables and fields and then we will focus on answering why queries on tables that have been constructed with multiple SMOs. This will enable users to better depict the tool’s usage and usefulness. (4) "Hands-on" Phase. In this phase the conference participants will be invited to directly interact with the system, having the chance to explore the evolution of tables/columns through all versions available for the wikimedia dataset.
We also intend to release the system online before the conference, so that the conference participants to have a chance of exploring system’s functionalities at their own pace and location. 4
In this demonstration we present a novel system enabling the VisuaL Exploration of multiple Schema Versions (VESEL) using provenance queries. The system gets as input the change log with the BiDEL SMO’s over multiple schema versions and is then able to visually answer queries about how a specific table/field was introduced, in which schema version and which was the sequence of operations that led to the creation of the specific table/field.
As future work, several challenging issues need to be further investigated. For example, we already see BiDEL as just a language describing SMO’s, with nice features. It is worth investigated the other proposed languages as well. This would obviously lead to a redefinition of some of our algorithms. In addition we intend to combine our approach with statistical information on the evolution, further guiding the user understanding on the schema evolution through multiple versions. Finally, it would be interesting to to study the efects of visualizing schema evolution in collaborative/multi-user or distributed environments that usually are prone to schema conflicts. 5
Work presented in the paper is part of the BOUNCE project (H2020 #777167) that has received funding from the European Union’s Horizon 2020 Research and Innovation Programme. Any opinions, results, conclusions, and recommendations expressed in this material are those of the authors and do not necessarily reflect the views of BOUNCE or the European Commission. 6