e BigDataEurope (BDE) project is developing exactly the kind of computing infrastructure that European stakeholders need when handling large volumes of data in a variety of formats; the results are open-source and their use is completely free. Coordinated by Fraunhofer IAIS, BDE is working directly with partners that represent the seven Societal Challenges identied by the European Commission (Health, Food, Energy, Transport, Climate, Social Sciences and Security). For each community, a pilot that makes use of BDEs technology stack to address the Big Data needs identied by these challenges is well under way. 1hp://sansa-stack.net/ which means that it is feasible to point a Compose app at a Swarm cluster and make its use possible in the same manner as if a single Docker host is being used. It is notable that the latest Docker components provide greater resemblance to Kubernetes in terms of orchestration features, and Swarm presents a beer choice in terms of shiing from a local/development environment to a cluster. e BDE Team provides baseline Docker images for Apache Hadoop, Spark, Flink and many others. Components were selected based on the requirements gathered from the seven Societal Challenges. us, the Platform makes it feasible to perform a variety of big data tasks, including message passing (Kaa, Flume), storage (Hive, Cassandra). e platform is able to handle RDF triples at scale using components like FOX, SemaGrow and 4Store; with particular emphasis on the triplication of geospatial data using GeoTriples, Sextant and Strabon. BDI has enriched the Docker platform, a high-level depiction of which is shown in Figure 1, with a layer of supporting services, helping in the setup, maintenance and monitoring of the pipeline and workows: e Init daemon allows to dene workows by monitoring the start-up status of inter-dependent Docker components. e Pipeline Service and Builder are developed to support the creation of workows. e Pipeline Monitor front-end demonstrates the current status of the Docker components. e Integrator UI integrates the dierent ocial Web UIs of select pipeline components under one Integrated and personalised view. Furthermore, the Swarm UI visualises the status of a swarm cluster and allows to scale and monitor the cluster services.
THE BIG DATA INTEGRATOR PLATFORM
Swarm-based networking Load Balancing Service Discovery Multi-host networking with integrated KV-Store
Fault tolerance
Docker Compose helps to create multiple containers on multiple nodes using a single command and a single compose le. Docker Compose V2 and Docker Swarm aim to implement full integration, For BDI platform progress updates please refer to the dedicated page2; or try it out or engage with our community3.
ACKNOWLEDGMENTS