Considering the increasing number of available systems, paradigms and tools related to Big Data challenges, this keynote aims at providing hints and good practices to avoid the common time-consuming pitfalls of the domain. During the last decade, the availability of large datasets has enabled the design and exploration of novel scenarios that leverage both openly accessible and private datasets for gaining competitive advantages. For example, Web users nowadays have access to general knowledge through the Wikidata endpoint [13], to public transport schedules with the GTFS format [4], to source code repositories [8], to proteins [2], to medical data1, to governments' records [1], etc. This availability has therefore opened the door to more advanced and complex analytic scenarios where multiple sources are combined together in order to build new block of knowledge, for instance touristic tours relying on geo-data, buses' schedules and reviews from previous tourists [5]. These new scenarios have practically led to the design of new paradigms where intermediate data structures are used in order to align on a same ground the useful pieces of data coming from different heterogeneous sources2. Consequently, with this profusion of data sources and more generally of avalailable data, new paradigms were designed in order to cop with the large amounts of information; this is for instance the case of the MapReduce model [3] and the associated Apache Hadoop3 or Apache Spark4 to deal, practically, with Big Data processing tasks when clusters of nodes have to be used because data is distributed. By nature, the Big Data landscape is cross-domain and the tools and systems available are numerous (with ones specifically created for particular use-cases and datasets). That is why the design of solutions for a particular problem in the Big Data context is challenging from different aspects: one needs to know which tool to select, how to structure and combine the data, where to find the missing information to complete the task, while having in mind that the solution might come a different community having an analogical problem. In this keynote, we provide several hints to avoid the common traps when having to deal with Big Data challenges.
Data Distribution Landscape. First, it is important to know where the
considered datasets are located in the data distribution landscape. Indeed, datasets
might come from several sources for a use-case linking them together, see
Figure 1’s right-hand side. And in parallel, each source could be either on a
singlenode architecture or relying on a cluster of machines in charge of distributing
the data and (maybe) the computations, see the left-hand side of Figure 1.
Figuring out where the current use-case is located will help to reach decision on the
working paradigms and more practically about the systems to be used.
Taking the use-case into consideration. To build an efficient solution, it is also
crucial to be use-case driven since the beginning. Typically, in case of a
distributed context, one needs to know, for instance, the type of Big Data the user
is dealing with i.e. is the data fitting in memory of one single node, is it fitting
over the cluster memory or is it larger than the sum of the memories of each
node? And depending on the context, the practitioner will need to select the
“best” system(s) available. Typically, it is important to choose from the
beginning the performance indicators or metrics that are going to be used to evaluate
and rank together the various potential solutions and systems which could be
used to achieve the use-case. Practically, relying on state-of-the-art benchmarks,
surveys, comparative evaluations is often helpful; however, most of the time, not
all the metrics that should be reviewed are considered at once by a single study.
For instance, to select a SPARQL evaluator, Graux et al. compared several
solution under the lights of different general use-cases and chose the relevant set
of metrics for each [
Data integration classification. Similarly to the data distribution landscape, it
is also relevant to decide on the integration paradigm. As presented in Figure 3,
there are mainly four situations depending if the datasets are structurally
homogeneous or not and depending on the distribution. For instance, if there are
several data sources having different data structures (e.g. relational tables, graphs,
documents, etc.), the data integration will have to rely on the use of wrappers
to make the intermediate results compatible. More generally, it is worth noticing
that Semantic Web technologies and the OBDA approach are good candidates to
integrate together heterogeneous sources, see e.g. Squerall [
Summary. In a nutshell, when having Big Data challenges, to save time from the very beginning, it is advised to take the following actions: 1. Check the situation of the needed datasets in the data distribution landscape; 2. Select the tool based on the final use-case, not strictly on performances and design for that a suitable set of metrics to evaluate the solution; 3. Gain awareness and decide on the data integration paradigm to be used; 4. Select the tool based on the vividness of its community.
Following these rules will significantly simplify the selection of paradigms for
data integration, and thus help the practitioner with the specific use case
implementation. To go further, we recommend to explore our open access book [