OVERVIEW

NoSQL document stores such as MongoDB allow to switch between a schema-free and a schema-fixed mode of operation, by registering a JSON Schema [4,11] declaration. Apart from solutions for isolated tasks, such as extracting a schema declaration from persisted documents, or validating documents against this schema, there are tools that combine these steps into comprehensive end-to-end schema management workflows (e.g. Hackolade [9] or Darwin [12,16]).

Towards this family of software products, we contribute a new prototype called Josch [6,7], where we enhance schema management workflows by integrating novel tools for checking JSON Schema containment. In interaction with Josch, we identify new research challenges for both practitioners and theoreticians working on search, exploration, and analysis in heterogeneous datastores.

WORKFLOWS

Our application scenario showcases a DevOps team who started application development and production operations with a MongoDB backend in schema-free mode. For data quality assurance, the team at one point decides to register a JSON Schema declaration with its MongoDB backend, so all writes are validated against this schema. * Schema extraction & validation. The DevOps team first has to extract a schema declaration from the persisted data [2,9,[13][14][15]. Often, schema extraction algorithms rely on sampling to cope with large data volumes. Consequently, the extracted schema may not faithfully describe the entire data instance. In order to avoid validation errors at runtime, the entire data instance needs to be validated against the extracted schema. This impacts database performance.

Schema refactoring & containment checking. When the schema is edited, e.g. adjusting it to account for outlier documents, or restructuring it for better readability, the team risks that the schema semantics is unintentionally changed. In JSON Schema containment checking, two JSON Schema declarations are compared based on their semantics. Thus, we can automatically decide whether the schema semantics has been changed.

For illustration, let us consider two excerpts of JSON Schema documents that describe the month of a publication, 𝑆1: {"type": ["number","string"]} and 𝑆2: {"type": ["number"]}. Schema 𝑆2 is contained in 𝑆1, and therefore more restrictive, as it requires the month to be numeric, whereas 𝑆1 also allows a string.

RESEARCH CHALLENGES

We refer to our extended version [6] of this paper for a more detailed discussion of related work. The full workflow just outlined is supported by our software prototype Josch [6,7], where Josch is geared to (but not limited to) MongoDB, and employs the thirdparty tools jsonsubschema [8] and is-json-schema-subset [10] for JSON Schema containment checking.

State-of-the-art JSON Schema containment checkers do not provide any explanation as to why two schemas differ. As a form of explainability, we may resort to generating a witness document [1], i.e., a JSON document that is valid w.r.t. one schema but not the other. At the moment, this is still a young research field.

Another limitation of current JSON Schema containment checkers are negation and recursive references [5]. While negation is rarely used in real-world schemas, it can lead to complex schemas [3].

The extracted schemas tend to be simplistic, yet highly verbose. A semi-automated refactoring that automatically extracts and introduces references for repeating structures to alleviate these shortcomings could prove helpful. Yet both schema refactorization and the extraction of complex schemas are open research challenges.

OUTLOOK

Solutions to the challenges outlined would also find application beyond NoSQL schema management, e.g., in the static validation of machine learning pipelines, as in the IBM LALE project [8].