We are dealing with ever growing amounts of data, or as some like to call it, we
are at the beginning of the era of big data. The value gained from the analysis of
big data is expected to be huge. Yet data analytics needs well-organized, clean
data to work on. The process of transforming raw data from various sources into
a form suitable for data analytics is called data wrangling [
While the term data wrangling is relatively new, transforming data from one
format into another has been a focus of the data management community for
many years now. There are systems that handle data extraction very well [
Yet, with the advent of big data, new challenges have arrived, sometimes characterized by the 4 V's of big data: volume (the scale of data), velocity (speed of change), variety (di erent forms and formats) and veracity (uncertainty). These pose challenges for each component of data wrangling itself, but also for the whole system.
So far, the challenges that big data poses for data wrangling have mostly
been met at the level of individual components (such as data extraction or
integration). Yet it is sharing knowledge between these components that has the
most potential of improving the data wrangling process, e.g. by allowing the data
management system to optimize execution based on such shared knowledge [
Overall, data wrangling today is an area highly demanded by industry, but without a clearly de ned research community or clearly de ned research objectives. The data management community is a natural t for taking up this task and shaping this new eld. The VADA (Value-Added Data Systems) programme, with the University of Oxford, the University of Manchester and the University of Edinburgh at its core, seeks to signi cantly advance data wrangling, both in practice by providing an architecture and prototype implementation for data wrangling, and in academic research by solving the challenges of wrangling big data in collaboration with the data management community.
The need to share data and knowledge between components of a data
wrangling system makes it clear that a common language is needed to express such
knowledge, and enable reasoning over it. In this paper, our goal is to describe
the vision of a lingua franca for data wrangling. Such a language should provide
a uniform way to address the di erent needs in the data wrangling process:
{ expressing knowledge in a shared knowledge-base
{ reasoning about data and transformation of data within the components
{ specifying the work ow between the components
The language should address these features by providing a uniform view of data
(independent of its source), while supporting the components by being suited to
data extraction, integration and exchange. At the same time, it must allow for
e cient processing and scalability when confronted with big data.
One of the best-established languages in the data management community for
knowledge-based reasoning is Datalog. Over the years, it has been studied in
great detail and extended in various ways [
Design Principles In this section, we present our vision for VADALOG, our proposed language for data wrangling. We will do this by following the major features, themes and properties of the language. For space reasons, we will focus on the design principles of the language, and not go into the details of its syntactic representation. Solid Foundation: Datalog. The language is based on Datalog, extended by features that are well-known in the data management community: in particular existential quanti cation (as in tgds, existential rules or Datalog ) as well as numerous other features motivated by the theoretical and practical needs of data wrangling. Having Datalog at its foundation gives VADALOG a well-understood core that has been the topic of research for many years now.
Family of Languages. One single, all-encompassing language cannot meet at the same time the goal of being highly expressive as well as having low computational complexity. VADALOG therefore consists of pro les of the language (in the same meaning as the pro les of languages such as OWL), each providing a speci c subset suited to a particular purpose. This allows simple computations to be e ciently and scalably executed over big data, while at the same time allowing complex reasoning over a smaller knowledge-base.
Combining Strengths. There exist a number of powerful knowledge-based systems, database systems, and systems that are able to deal with big data that often combine the expertise of large groups of researchers and engineers. The language is thus designed with the intent of making use of systems speci cally suited for particular tasks. For example, if a VADALOG programme is formulated in a pro le of the language that is particularly suited to an existing knowledge-based system, then this system is used as a backend-engine. This design choice does not come for free { many interesting research challenges have to be addressed to deal with multiple engines working in a uni ed system.
Handling Volume. Volume may be coped with by using an engine that is suited for handling huge amounts of data. Yet this is not always the most e cient approach. The language contains as \ rst-class citizens" the support for partitioning the data into dataspheres which may be parameterized using domain-dependent or data-dependent parameters. This language-design principle of being able to handle volume by allowing clever partitioning of the data, combined with using engines that can handle big amounts of data when necessary, allows VADALOG to e ciently deal with huge amounts of data. Modularity. Reasoning and transformation tasks are organized into self-contained modules { based on the concept of a data transducer which receives data from di erent dataspheres and produces data in di erent dataspheres. Such transducers modularly encapsulate their dependencies (which dataspheres they require to be present) and their guards (what conditions must be met to be executed). De ning all of these parts of the transducer is done using VADALOG in a single, maintainable module for each such transducer.
Dynamic Orchestration. De ning single modules { transducers { is only one part of a data wrangling system. A key part of such a system is that all its components are able to share data and knowledge between them and, importantly, react to such knowledge by dynamically selecting which next steps to take. For example, as a result of quality analysis, the system may choose to redo data extraction with di erent background knowledge, or adding a data source. A key part of VADALOG is thus a pro le for specifying such transducer networks that dynamically orchestrate components of the data wrangling system. Extensibility. A rule-based language based on Datalog is clearly suited for knowledge-based reasoning tasks. Yet, while a data wrangling system has reasoning-intensive tasks, it also has tasks which are better suited to be implemented in other languages. To harness components implemented in other languages, VADALOG allows extensibility at a number of levels: At the transducer level, which gives the components wide-ranging freedom in how to approach its task (such as an existing component analysing data quality); at the level of actions, which are middle-scale tasks (such as navigating web pages), and at the level of external functions, which can add small-scale functionality not supported in the language (such as non-supported string or number functions). 3
Data wrangling is an important challenge in practice and for research. The data management community is in a good position to take on this challenge, giving this growing eld a scienti c community and shaping its research objectives. The data management community should take an active role in this area.
Developing a language for data wrangling is an important step that is clearly needed for that purpose. In this paper, we described the design principles of VADALOG, our proposed language for data wrangling. While a number of design principles are related to solving the challenges of data wrangling itself, two important ones promote a collaborative approach to this: It allows to harness the power of existing knowledge-based reasoning systems, and it promotes extensibility by allowing integration of components.
We invite the data management community to collaborate with us both in using the power of currently-developed systems for data wrangling, as well as to shape the future of VADALOG.
Acknowledgements. This work was supported by the EPSRC programme grant EP/M025268/1. Bernd Neumayr receives funding from a habilitation grant of the state of Upper Austria. Emanuel Sallinger was partially supported by the Austrian Science Fund projects (FWF):P25207-N23 and (FWF):Y698.