A Big Data system is a tiny fraction of analytical code surrounded by a lot of “plumbing” devoted to manage the generated models and the associated data. Hence, we can consider that plumbing to be mimicking a DBMS, which is indeed a complex system that actually has to serve diferent purposes and hence provide multiple and independent functionalities. Thus, it can neither be studied nor built monolithically as an atomic unit. Oppositely, there are diferent software inter-dependent components that interact in diferent ways to achieve the global purpose. Similarly to DBMS, in a Big Data system, we have to understand among other issues how our system is going to collect data; how these are going to be used; where they are going to be stored; how they are going to be related to the corresponding metadata; if we are going to use any kind of master data, where these will come from and how they will be integrated; how are the data going to be processed; how replicas are going to be managed and their consistency guaranteed; etc. In this paper, we briefly discuss the dificulties to build such system, paying special attention to how metadata can help storage and processing.
2. Big Data Architectures
© 2022 Copyright for this paper by its authors. Use permitted under Creative
CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g CCoEmmUoRns LWiceonsrekAstthribouptionP4r.0oIncteerenadtiionnagl s(CC(CBYE4U.0)R.-WS.org)
The previously identified challenges require a complete
reconsideration of classical DBMS architecture and
components that date back to the 70s. Yet, the approach so far
adopted by the data management community has been
that of developing components addressing each of the
required functionalities (i.e., Storage, Modeling,
Ingestion, Processing, and Querying/Fetching) as eficiently
as possible. Indeed, current technological stacks for the focuses on batch processing, and the other on stream
promanagement and processing of data-intensive tasks are cessing. Nevertheless, maintaining such potentially
recomposed of independent components (commonly those dundant flows generates some management risks. For the
in the NOSQL family) that generally work in isolation and prediction to be accurate, the new arriving tuples have
are orchestrated together to map to what would be equiv- to go through the same transformation and preparation
alent to diferent functionalities of a DBMS. In complex tasks as the training data (otherwise, the validity of the
applications, where many of these tools need to interact, prediction would be compromised). The -architecture
it is definitely not wise to do it arbitrarily. Indeed, such evolved into -architecture, as a simplification with a
erroneous approach yields what is coined as a “pipeline single execution engine (hence a single implementation
jungle” [
The risk with this approach is that files can be simply by having independent components that need to
intermassively accumulated without any order, resulting in act without a solid backbone that removes the burden
what is called a “Data Swamp”, where just finding the of their connectivity from the shoulders of developers.
relevant data would be a challenge. The solution for this If we pay attention to either Velocity or Variety, we can
is creating an organization of files and semantically anno- conclude that metadata is crucial for such backbone and
tate into a metadata catalog conceptualizing the domain the governance of Big Data. However, current state of
(e.g., implemented via graph-based formalisms). Thus, to the art has not reached the required level of maturity
the already existing mappings in the catalog, we should to give the view of a single homogenous system
coordiadd links from each file to such graph containing the rel- nated through those metadata instead of adhoc scripts
evant concepts for our business. In this way, users would or specifically programmed APIs and connectors.
be able to perform guided searches over the metadata
instead of blindly navigating the files. If properly done, References
a semantic approach can even facilitate automation of
integration and queries [
Obviously, since the purpose of a prediction is to react or at least be ready to take some action, data freshness and response time is typically crucial in this case.
Consequently, since time requirements are contradictory, we have to distinguish both precessing flows, giving rise to what is known as -Architecture. This, consists of two execution branches fed from the same sources. One