We demonstrate the open-source Semantic Analytics Stack (SANSA), which can perform scalable analysis of large-scale knowledge graphs to facilitate applications such as link prediction, knowledge base completion and reasoning. The motivation behind this work lies in the lack of scalable methods for analytics which exploit expressive structures underlying semantically structured knowledge bases. The demonstration is based on the BigDataEurope technical platform, which utilizes Docker technology. We present various examples of using SANSA in the form of interactive Spark notebooks, which are executed with Apache Zeppelin. The technical platform and the notebooks are available on SANSA Github and can be deployed on any Docker-enabled host, locally or in a Docker Swarm cluster.
SANSA5 is an open-source6 structured data processing engine for performing distributed
computation over large-scale RDF datasets [
6 https://github.com/SANSA-Stack 7 Please note that any similarities of the paper title to popular TV series are purely coincidental. large RDF datasets, SANSA leverages existing big data frameworks like Apache Spark and Apache Flink,8 which have matured over the years and offer a reliable method for general-purpose processing of large-scale data.
In this demonstration, we will present and describe our implementation of interactive Spark Notebooks for SANSA.9 These notebooks are a collaborative environment implemented as an interactive web editor. They allow access to the SANSA layers and hence provide data scientists, data engineers and students with means to easily use and execute the functionality of SANSA to explore, analyze and learn from large-scale RDF datasets. 2
The SANSA Stack
Research efforts in the areas of distributed analytics and semantic technologies have
been mostly isolated until now. We aim to proceed one step further by using the semantic
modelling standard as a basis for machine learning and data analytics. The layered
architecture of SANSA is a direct consequence of this integrated vision and is depicted
at the top of Figure 1. For a detailed description of each layer, we refer to [
Machine Learning
Inference Querying Knowledge Distribution & Representation Distributed In-Memory Processing } Analytics } bDuisttiorin- Technology
Semantic
Stack
Machine Learning Libraries Distributed Data Sets / Streams In-Memory Computing Framework
Distributed Filesystem manual data integration often simple input formats data formats often not standardised measurable benefits horizontal scalability filebrowser
Hadoop
RDF
Query SANSA provides Notebooks for an easy local deployment for development and demonstration purposes. SANSA-Notebooks is an interactive toolkit on top of Hadoop-SparkWorkbench10 with Apache Zeppelin,11 which allows the copying of files from/to HDFS and an interactive Spark code execution via a web GUI. The architecture of SANSANotebooks is depicted in Figure 2. The authors utilize SANSA-Notebooks (see Figure 3) in Big Data labs and courses as they alleviate the complicated Hadoop/Spark setup and allow the students to focus on developing distributed algorithms on top of SANSA. Cluster deployment of the examples is also possible through Docker images (see SANSA-Examples Github repository12). Additionally, SANSA is readily available from the Maven Central Repository. Thus it is straightforward to include it in other projects using Maven or SBT – the most popular build managers for Scala – for both Spark- and Flink-based setups.
During the demonstration, we will present the example notebooks.13 These examples give a quick overview of the SANSA APIs. SANSA is build on the concepts of distributed datasets (i.e RDD, DataFrame, DataSet). A dataset is inferred from the external data, then parallel operations e.g. transformations and actions are applied which trigger a job execution on a cluster. Depending on the network connection, the demonstration will be performed on a local single node cluster or a remote multi node cluster. In the following, we provide a concise description for the examples grouped by the SANSA layers. 1. RDF.
(a) Reading and writing triple files from HDFS or file system and some basic triple operations. (b) A distributed evaluation of numerous RDF Dataset Statistics dubbed RDF-Stats (see Figure 3), for example, property distribution, class distribution, distinct subjects/objects/entities as well as statistics summary. (c) Assigning weights to a given entity based on the Spark GraphX PageRank algorithm after triples have been transformed to a graph representation (i.e. PageRank for resources). 10 https://github.com/big-data-europe/docker-hadoop-spark-workbench 11 https://zeppelin.apache.org/ 12 https://github.com/SANSA-Stack/SANSA-Examples 13 The source code for all of them is provided at https://github.com/SANSA-Stack/SANSA-Examples. We will present a tale / storyline using different examples across the SANSA layers for booth visitors and adapt them interactively (with new parameters, other datasets etc.) in the web browser.
One of the powerful features of the SANSA Notebooks is that you can view the result set of the previous session within the Spark framework and, in case you have found some insight for your data and would like to share, you can easily create a report and either print or send it. 14 http://aksw.org/Projects/Sparqlify.html