The lack of an explicit data schema (schemaless) is one of the most attractive NoSQL database features for developers. Being schemaless, these databases provide a greater flexibility, as data with different structure can be stored for the same entity type, which in turn eases data evolution. This flexibility, however, should not be obtained at the expense of losing the benefits provided by having schemas: When writing code that deals with NoSQL databases, developers need to keep in mind at any moment some kind of schema. Also, database tools usually require the knowledge of a schema to implement their functionality. Several approaches to infer an implicit schema from NoSQL data have been proposed recently, and some utilities that take advantage of inferred schemas are emerging. In this article we focus on the requisites for the visualization of schemas for aggregate-oriented NoSQL Databases. Schema diagrams have proven useful in designing and understanding databases. Plenty of tools are available to visualize relational schemas, but the visualization of NoSQL schemas (and the variation that they allow) is still in an immature state, and a great R&D effort is required to achieve tools with the desired capabilities. Here, we study the main challenges to be addressed, and propose some visual representations. Moreover, we outline the desired features to be supported by visualization tools.
The NoSQL term (Not SQL/Not only SQL) is used to denote a new
generation of database systems that overcomes the limitations of relational systems
to satisfy the requirements demanded by modern applications. A large
number of companies have already embraced NoSQL databases, and this adoption
will rise considerably in next years, as reported in [
The absence of an explicit schema (schemaless approach) is a convenient
feature in most NoSQL systems, because it provides the required flexibility when the
data structure varies often. Being schemaless, the database can store data with
different structure (i.e. schema) for the same entity type (non-uniform data),
and data evolution is favoured due to the lack of restrictions imposed on the
data structure. However, removing the need of declaring explicit schemas does
not have to be confused with the absence of a schema, as a schema is implicit
into stored data and in the application code that deals with the database. The
developers must always keep in mind the schema when they write or maintain
code that is connected to the database. This is an error-prone task, more so when
entities have a degree of variability. Moreover, some NoSQL database tools and
utilities need to know the schema to offer even basic functionality. Therefore, the
flexibility gained by being schemaless should not be at expense of the benefits
provided by having explicit schemas. A growing interest in managing implicit
NoSQL schemas is therefore arising, as evidenced in the schema inference
approaches recently proposed [
The recent Dataversity report [
Visualizing database schemas in form of diagrams is useful both to designers and developers. Designers can express the database structure at a high level of abstraction and better reason about efficiency and physical layout, and developers can write better code if they have a model that properly represents the database schema. Moreover, schema diagrams provide very useful documentation that facilitates the understanding of an existing database in performing evolution tasks. In fact, plenty of commercial and open-source modeling tools exist for relational databases.
While a relational database schema is formed by a set of entities and the
relationships between them, in NoSQL databases several schemas can exist for
the same entity. We refer to this feature as versioned schemas. This complicates
the visualization of the global schema that includes all the entity version schemas
and relationships between them. This issue is normally tackled by (i) showing
entities whose schema is the union of all of its schemas, and (ii) ignoring the
reference relationships between entities [
This article is focused on the visualization of NoSQL schemas, more
specifically, we address the challenges that visualizing versioned schemas raise in the
case of aggregate-oriented NoSQL systems. For this, we will work on schemas
inferred by applying the model-driven reverse engineering approach described
in [
The remainder of this article is organized as follows. First, we briefly introduce the concept of aggregate-oriented data models and a running example. Then, we define the set of schemas identified and the diagrams proposed. Next, we comment the related work. Finally, we present the conclusions drawn from our work, which will guide our future work. 2
Object references and aggregate objects are constructs specifically conceived to
represent complex data, but they are not part of the relational model. Aggregate
objects are usually preferred to object references in the case of NoSQL databases,
because the data is distributed through clusters to achieve scalability, and object
references may involve contacting remote nodes. Thus, aggregate-orientation has
been identified as a characteristic shared by the data models of the three most
widely used NoSQL systems: key-value, document, and wide-column [
An aggregate-oriented NoSQL database can be seen as storing a set of semistructured objects. Semi-structured data is mainly characterized by its schemaless nature. A semi-structured object O is composed of one or more fields (attributes or properties ) pi: O = fp0; p1; : : : ; pmg. Each field pi is specified by a pair < ni; vi >, where ni and vi denote the name and value of the field, respectively. The value of a field can be: (i) an atomic value (a number, string, boolean, . . . ); (ii) another object, i.e. an embedded object inside the object which the field belongs to; (iii) a reference to another object: This is usually a string or integer that matches the value of a field in the referenced object; or (iv) an array of values, which can be homogeneous or heterogeneous. Therefore semi-structured data has a hierarchical (nested) structure with a root object which can recursively embed other objects and arrays.
A database system stores data that relate to entities of real the world (i.e any physical or conceptual thing that exists). Here, an entity labels all the objects that refer to the same concept (e.g. movie, director, or prize). As indicated above, the schemaless nature of NoSQL databases allows for different stored objects of the same entity type to have variations in their schemas. Therefore, we will introduce the notion of entity version to denote each of the sets of objects that, sharing the same entity label, have a different schema. Each entity will have one or more entity versions. Versions exist both for root and nested entities.
JSON [
Our database example in Figure 1 is represented as an array that stores the collections of the Movie and Director entities. There are 3 versions of Movie, 2 versions of Director and Criticism, and one version of Price and Rating. Movie and Director are root entities; Criticism, Rating, and Prize are embedded into Movie. We refer to versions of an entity by means of the name of the entity joined to an unique version number by an underscore (e.g. Movie_3 and Director_1 )
We have supposed that a Movie entity has four common fields: title, year, director, and genre, and four optional fields: prizes, rating, criticisms, and running_time. The Director entity has two common fields: name and directed_movies, and the acted_movies optional field. The Criticism embedded entity has three common fields: color, journalist, and media, and one optional field, url. The Prize embedded entity has three fields: year, event, and names. Finally the Rating embedded entity has just two fields: score and voters.
Figure 2 shows the metamodel described in [
In this section, we shall define the types of schemas that we have identified for aggregate-oriented NoSQL databases. For each object in the database, an Object Schema (or type) is obtained by replacing, recursively, the atomic values of a semi-structured object by an identifier that denote its type (i.e. String, Number). Therefore, it has the same structure as the described object with respect to fields, nested objects, and arrays. The schema inference process starts with this set of object schemas to infer the set of entities and relationships.
In relational databases, an entity has just one schema, and the database schema is formed by the entities (its only schema) and the relationships between them. However, in NoSQL databases, the existence of entity versions, each with variations in their schema, adds a set of new dimensions to the schema definition.
An entity version schema (or simply version schema) is obtained from the object schema of an entity version by replacing each embedded and referenced { " rows ": [ { " type ": " movie ", " title ": " Citizen Kane ", " year ": 1941 , " director_id ": "123451" , " genre ": " Drama ", " _id ": "1" , " prizes ": [ { " year ": 1941 , " event ": " Oscar ", " names ": [ " Best screenplay ", " Best Writing " ]}, { " year ": 1941 , " event ": " National Board of
Review , USA ", " names ": [ " Best Film ", " Top Ten Films " ]} ], " criticisms ": [ { " journalist ": " Roger Ebert ", " media ": " Chicago Sun - Times ", " url ": " http : // chicago .
suntimes . com /" , " color ": " green " }, { " journalist ": " Richard Brody "
, " media ": " The New Yorker ", " color ": " green " } ]}, { " type ": " movie ", " title ": " The Man Who Would Be
King ", " year ": 1975 , " director_id ": "928672" , " genre ": " Adventures ", " _id ": "2" , " running_time ": 129 }, { " _id ": "3" , " type ": " movie ", " title ": " Truth ", " year ": 2014 , " director_id ": "345679" , " genre ": " Drama ", " rating ":{ " score ": 6.8 , " voters ": 12682 }, " criticisms ": [ { " journalist ": " Jordi Costa ", " media ": " El pais ", " color ": " red " }, { " journalist ": " Lou Lumenick ", " media ": " New York Post ", " color ": " green " } ]}, { " name ": " Orson Welles ", " directed_movies ": [ "1" ], " acted_movies ": [ "1" ], " type ": " director ", " _id ": "123451" }, { " type ": " director ", " directed_movies ": [ "3" ], " name ": " James Vanderbilt ", " _id ": "345679" }, { " type ": " director ", " directed_movies ": [ "2" ], " name ": " John Huston ", " _id ": "928672" ]}
} objects by the corresponding name of the embedded or target entity version, respectively. These schemas can specify both root (root version schema) and embedded objects (embedded version schema ). In Figure 3 we show (in JSON format) the root version schema for the Movie_1 entity version.
A version schema therefore involves four kinds of entity properties: (i) primitive (title, genre, and year ); (ii) reference (director_id ); (iii) aggregation, (Prize_1 ); and (iv) array type, that can have one (homogeneous) or more (heterogeneous) base types that can in turn be primitive, aggregation, reference, or another array. The criticisms field is an array of either Criticism_1 or Criticism_2. The relationship type term refers both to the reference and aggregation types. Note that these two types cause that a schema is either directly or indirectly related to others. The Movie_1 version schema involves the Criticism_1, Criticism_2, Director_1, and Prize_1 version schemas, and these schemas might refer in turn to other schemas, so that a schema graph is formed.
An entity schema can be defined as the set of version schemas of a given entity. Transformations can be performed to this set for it to be useful for different purposes. For instance, sometimes it is interesting to have a “view” of all the version schemas of an entity. This can be done joining all the properties contained in the schemas of the entity versions of the entity: An entity union schema could be constructed with the following rules: 1. For each property whose name appears only in one entity version schema, add that property to the entity union schema. 2. For each property whose name appears in more than one entity version schema: (a) If the type of the property is the same in all the entity version schemas in which it appears, add that property to the entity union schema. (b) If the type of the property differs in some entity version schemas, collect the set of different types of the property and build a union type. A union type of two types T1 and T2, denoted as U (T1; T2), can be defined as a type that describes both the elements described by T1 and those described by T2.
Figure 4 shows the entity union schema for the Movie entity of our database in JSON format.
The usual schema inference for JSON objects in NoSQL systems discovers
schemas that result of the union of the object schemas of each entity version, but
version schemas and entity schemas are not discovered. Therefore the structure
of the schema is similar to those defined for object schemas. We refer to these
schemas as union object schemas. Several strategies are used to solve the problem
of conflicting types when a field belongs to more than one version. For instance,
the type of the field can be the union of all the types encountered, can be
promoted to the Object type [
Finally we shall define two kinds of schemas for aggregate-oriented databases that involve all the entities: – Database schema. It is formed by the set of root version schemas that describe the root entity versions of the database. As a schema recursively depends on the schemas of the embedded or referenced entities, a complete schema is formed by the set of version schemas of all the entity versions that exist in the database. – Entity database schema. It is formed by the entity union schemas that describe all the root entities of the database.
Next we show how the defined types of schemas could be visually represented. We will first consider UML class diagrams. These diagrams together with ER diagrams are traditionally used to represent conceptual and logical relational schemas. Here, we will show their limitations for NoSQL schemas and the need for the definition of a specific notation. 4
PlantUML [
As explained in [
With this representation, heterogeneous arrays cannot be displayed directly. We could generate, for example, artificial “union” classes to join all the included classes in heterogeneous arrays, similarly as how we constructed the entity union schemas, but this would be more visually intrusive. Or we could generate N aggregation relationships with numbered suffixes to each of the included classes. In any case, the hierarchical structure of these schemas can be appropriately represented with class diagrams.
Entity Union Schemas. We have followed the strategy explained above for visualizing entities in root version schemas. When several version schemas have a property with identical name but different type, the union type inferred for this property can only be visualized if it has only two versions: one includes the property with a primitive type (or a tuple) and the other one is a relationship, but the rest of possible unions of types would cause an error because would have several attributes (or relationships) with the same name and different type (or association end). Figure 6 shows the union schema for the Movie entity, which includes six attributes, three composite relationships for the Criticism, Prize, and Rating union schemas, and a reference relationship to the Director entity union schema. The diagram also shows the relationships of the aggregated or referenced union schemas, for instance, two reference relationships from Director to Movie. While the previous diagram showed all the referenced or embedded entity versions involved in the definition of a version schema for a root entity, the diagram for an entity union schema only includes entities. Therefore, in the diagram in Figure 6 the reference relationships from Director to Movie are shown because no confusion is caused.
The main limitation of class diagrams for version schemas is the representation of union types, as several attributes with the same name but different type are not allowed. Note that the graph-like structure of a entity union schema can be adequately represented by class diagrams.
Entity Database Schemas. A diagram of the entity database schema is formed by superposing all the diagrams for root union schemas. It is worth to note that an entity union schema can be directly or indirectly connected to the rest of entity union schemas existing in the database, and then such a schema will be equivalent to the entity database schema. In our running example, this diagram would be the same as that shown in Figure 6. 5
Tackling the visualization of NoSQL schemas with UML class diagrams evidenced the convenience of defining a specific notation for this purpose because UML class diagrams cannot represent entity schemas or database schemas, and visualizing the rest of schemas has the limitations commented in the previous section. Moreover, the cases in which “sets” of schemas are generated, some kind of browsing capability is needed to navigate the structure of the database.
Therefore, we have developed a diagramming tool that has been specially
designed to visualize NoSQL schemas. This solution took advantage of the fact
that our inference process generates models that conform to an Ecore metamodel
to significantly reduce the development time and effort in relation to create the
editor from scratch. We have used Sirius [
In this tree, entities, entity versions, and root entity versions are represented with indicative icons. Also attributes, aggregation and reference relationships. These icons are used in all the diagrams created to provide a uniform view to the user. It is possible to navigate from the Global View Tree to the other diagrams by means of contextual menus.
Database Schema Diagrams represent the information included in a database schema as shown in Figure 8. With this diagram, the user can see at a glance (i) the database entities, (ii) the set of version schemas of each entity, and (iii) the attributes and relationships of each version schema. Version schemas are visualized as rectangles nested into a rectangle that represents the root or embedded entity schema it belongs to. Aggregation and reference relationships are visualized as a solid line and are tagged as aggregates and references respectively. Entity Schema Diagrams represent an entity schema as shown in Figure 9. They are visualized just like database schemas, but with only one entity schema shown, which can correspond to a root or embedded entity. The Figure shows that 1 https://eclipse.org/sirius/gallery.html.
Fig. 7: Global Schema Tree for the Movie example. the Movie entity has three versions, and the corresponding version schemas are shown. For instance Movie_1 aggregates a version schema of the Prize entity, as well as two version schemas of the Criticism entity, among other references.
Root version schemas and Entity union schemas are visualized with diagrams similar to those generated with PlantUML in Figures 5 and 6. 6
Some well-known modeling tools for relational databases have been recently
augmented to offer functionality for NoSQL databases. ER/Studio Data Architect [
ERwin Unified Data Modeler is a project under development with the aim
of supporting data modeling for relational and NoSQL systems [
ExSchema [
JSON Discoverer is a tool aimed to infer JSON schemas from JSON-based
Web APIs [
Modeling tools for aggregate-oriented NoSQL systems should be designed taking
into account the existence of entity versions. In this article we have explored the
schemas that could be useful to designers and developers and how they could be
visually represented. We have observed that the diagrams proposed for version
schemas and entity union schemas are appropriate and the limitations exposed
for UML class diagrams are solved in the Sirius solution. For entity schemas,
in addition to entity union schemas and browsing version schemas, a tree
representation that explicitly shows the common and specific properties among the
version schemas could be very useful, and will be studied as a future work. The
database schema diagram shown in Figure 8 is difficult to understand due to
the existence of relationships between entity versions, and may become
infeasible when entities have more than a few entity versions. Therefore, database
schemas require browsing capabilities and even query languages if the number of
schema versions is high. This is also planned for future work. For entity database
schemas, UML class diagrams are adequate, but properties with the same name
but different type should be allowed. The order of magnitude of the number of
version schemas for entities significantly affects to the mechanisms implemented
around the diagram representation. We have tested some open datasets such
as Stackoverflow, and have found that around a hundred schema versions is
common, but there are also cases with a very large number of schema versions
(tens of thousands) [