The last decades witnessed a significant evolution in terms of data generation, management, and maintenance, especially in the RDF format. Moreover, in the energy domain, semantic data is finding its way and can be used for various data analytics tasks. However, since data set sizes are increasing and can now be enormous, technologies are evolving to scale with the increasing data set sizes. In this regard, tools and frameworks such as SANSA have been emerged to facilitate the analytic over semantic data. SANSA is using big data technologies such as Apache Spark (as an analytics engine for large-scale data processing) and Apache Hadoop (as a distributed file system) in its backbone to be able to perform analytics in a distributed manner over a cluster of nodes. However, to be able to use SANSA, one should set up a cluster of nodes with enabled Spark and Hadoop. This requires extensive knowledge and expertise in computer systems, networking, distributed computing and etc. Moreover, in case of having suficient technical knowledge, setting up such a cluster consumes huge manpower and is labor-intensive. To tackle the aforementioned issues, in this paper we introduce a micro-service architecture that easily brings the power of SANSA and distributed semantic data analysis in the end-user ecosystem, without having technical knowledge in the mentioned areas. The introduced architecture is based on Docker technologies and can be installed on-premise or in the cloud systems.
With the rapidly growing amount of data available on the Internet, it becomes
necessary to have a set of tools to extract meaningful and hidden information
from the online data. The Semantic Web is able to form a structural view of the
Copyright © 2021 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
existing data on the web and provides machine-readable formats [
To be able to use SANSA efectively, a cluster of Spark nodes with an HDFS ifle system is required. Even by having such a cluster, one can only interact with SANSA via a terminal. Establishing a cluster of diferent nodes containing Spark and Hadoop6 and configuring them is by nature a cumbersome task and needs a lot of knowledge and experiences. Moreover, in the case of having technical knowledge, establishing such a cluster is a time-consuming task. Moreover, end-users prefer to have a user friendlier way to interact with SANSA without having any programming and scripting knowledge. Therefore, in this paper, we introduce a micro-service structure of a SANSA-enabled Spark and Hadoop cluster with 2 user-friendly interactive communication mechanisms aka. REST API and Zeppelin Notebook7. Our introduced architecture is based on Docker technologies8. Moreover, our sample explanatory tutorial enables non-technical users to easily use SANSA without having any specific knowledge and skills. 1.1
Contributions: - Introducing a micro-service architecture of Big Data tools such as Apache
Spark, Apache Hadoop, Apache Zeppelin, HDFS File Browser, Apache Livy - Introducing for the first time REST APIs for the SANSA stack - Introducing an interactive Notebook (i.e. Apache Zeppelin) for interacting with SANSA - Making the code and the framework open-source and publicly available on
Github9 3 https://www.w3.org/RDF/ 4 http://spark.apache.org/ 5 http://sansa-stack.net/ 6 https://hadoop.apache.org/ 7 https://zeppelin.apache.org/ 8 https://www.docker.com/ 9 https://github.com/SANSA-Stack/SANSA-Stack
In recent years, it has been recognized that creating complex technical environments for big data is a major challenge. Setting up a lot of nodes and connecting them together in a way that all together perform a specific task not only is costly but requires extensive knowledge in many computer science areas such as networking, cluster computing, and big data technologies. Although nowadays, other technologies such as virtualization (e.g. VMbox3, Parallels, VMware) and its successor containerization (e.g. Docker, Swarm, Kubernetes) keep the costs lower but add up other skills to be able to set up a cluster properly.
Besides this, there are numerous centralized machine learning frameworks
and algorithms for RDF data. For example, TensorLog [
In order to use SANSA efectively, having a distributed environment is
inevitable. Although, some enterprise companies such as Databricks11 [
In this section, we present the micro-service system architecture for using SANSA. Worth mentioning that the framework is open-source and hosted on GitHub15. The main goal of the framework is to bring a simple and efortless approach to set up a Spark cluster with all the requirements without having extensive computer science knowledge. 10 https://jena.apache.org 11 https://databricks.com/ 12 https://aws.amazon.com/ 13 https://azure.microsoft.com 14 https://cloud.google.com/ 15 https://github.com/SANSA-Stack/SANSA-Stack We provided two interaction mechanisms for the end-users. a) Zeppelin Notebooks b) REST APIs. Depends on the scenario users may select one of the mentioned mechanisms. These mechanisms cover the full spectrum from simplicity to flexibility. By using the REST APIs, users will be able to call predefined functionalities from SANSA without any efort. However, in case a user is interested in the new functionalities, they can write code and stack their code via Zeppelin Notebook and submit their task to the Spark cluster. Figure 1 depicts the high-level system overview. The architecture contains 4 main components i.e. a) Java-based REST APIs b) Apache Zeppelin Notebook c) Apache Livy d) Spark-Hadoop Cluster, which are all utilized via Docker-Compose. Java-based REST API This layer provides functionality for the end-user to interact with SANSA via REST APIs. It is Java-based and is powered by Spring Boot16 technology and contains a Swagger17 UI which enables users to easily call any provided functions via a browser. Figure 2 shows the SwaggerUI and provided APIs. A sample scenario of how to use these APIs is provided in Section 4.
Apache Livy REST APIs As the Spark tasks may be long-running (up to a few days) and also there is a chance that the node which is running the task 16 https://spring.io/projects/spring-boot 17 https://swagger.io/ crashes and loses the calculations, therefore, directly connecting the REST APIs to the Spark cluster is not feasible due to asynchronous nature of such computations. To tackle this, another layer has been added by using Apache Livy18, which is able to keep tracks of Spark sessions and calculation states. Livy enables programmatic, fault-tolerant, multi-tenant submission of Spark jobs from web/mobile apps. So, multiple users can interact with the Spark cluster concurrently and reliably. Although the Livy interface is available for the user, this layer works as a background process and the user does not need to interact with it directly, because all the functionalities will be handled by the REST API layer. Spark-Hadoop Cluster To be able to run a SANSA functionality, having a Spark cluster with a Hadoop file system is inevitable. To do so we containerized Spark, Hadoop Namenode, Hadoop Datanode, and Hue HDFS file browser 19, in docker images which are publicly available. Moreover, we configured the containers to interact with each other seamlessly via Docker-Compose. Of course, all the other layers have been containerized and exposed in the same docker-compose ifle. 18 https://livy.apache.org/ 19 https://gethue.com/
To be able to run the cluster, the user needs to clone the SANSA Stack from Github and navigate to sansa-rest sub-folder. The following codes in the terminal will bring up the cluster. $ git clone https://github.com/SANSA-Stack/SANSA-Stack.git $ cd SANSA-Stack/sansa-rest $ make $ make up
To stop the cluster, the user only needs to run the following command. $ make down
As already mentioned, users will have two interaction mechanisms to
connect to SANSA, either using Zeppelin Notebook or REST APIs. Using Zeppelin
Notebook is easy and straightforward same as all the other notebook
technologies such as Jupyter Notebook20. Therefore, without losing the generality and
due to the space issue, we ignore the explanation in this paper. However, in the
following, we explain a sample scenario which shows how to use the REST APIs.
Besides its many functionalities, SANSA provides a distributed SPARQL engine
(i.e. Sparklify [
In this paper, we introduced a framework which is able to set up a Big Dataenabled cluster with all its requirement from Spark and Hadoop to Zeppelin Notebook to be able to use SANSA framework without any efort. The proposed approach is based on Docker Compose technology and can be installed on-premise or on any cloud environment. Moreover, we implemented a set of REST APIs via Swagger which enables all the non-technical users to interact with SANSA in a very straightforward manner.
This work was partly supported by the EU Horizon 2020 project PLATOON (Grant agreement ID: 872592). We would also like to thank the SANSA development team for their helpful support.