The increasing adoption of the Linked Data format, RDF, over the last two decades has brought new opportunities. It has also raised new challenges though, especially when it comes to managing and processing large amounts of RDF data. In particular, assessing the internal structure of a data set is important, since it enables users to understand the data better. One prominent way of assessment is computing statistics about the instances and schema of a data set. However, computing statistics of large RDF data is computationally expensive. To overcome this challenging situation, we previously built DistLODStats, a framework for parallel calculation of 32 statistical criteria over large RDF datasets, based on Apache Spark. Running DistLODStats is, thus, done via submitting jobs to a Spark cluster. Often times, this process is done manually, either by connecting to the cluster machine or via a dedicated resource manager. This approach is inconvenient as it requires acquiring new software skills as well as the direct interaction of users with the cluster. In order to make the use of DistLODStats easier, we propose in this paper an approach for triggering RDF statistics calculation remotely simply using HTTP requests. DistLODStats is built as a plugin into the larger SANSA Framework and makes use of Apache Livy, a novel lightweight solution for interacting with Spark cluster via a REST Interface.
SANSA [
SANSA and DistLODStats use Apache Spark5 as an underlying engine, which is a popular framework for processing large datasets in-memory. Spark provides two possibilities of running and interacting with applications: 4 https://github.com/SANSA-Stack 5 http://spark.apache.org/ – Interactive - via a command line interface (CLI) called Spark Shell, or via Spark
Notebooks (e.g. SANSA-Notebooks [
Spark application is usually launched by logging first into a cluster, either in the premises or remotely in the cloud. This process presents several di culties: – It requires a sophisticated user access control management, which may become hard to maintain with multiple users. – It raises the chances of exhausting the cluster or even causing its failure. – It exposes cluster and its configurations to all the users with access.
In order to elevate those, we have investigated Apache Livy 6 – a novel open source REST interface for interacting remotely with Apache Spark. It supports executing snippets of code or programs in a Spark context that runs locally, in a Spark cluster or in Apache Hadoop YARN.
This is an accompanying poster paper for DistLODStats [
Traditionally, when running a Spark job, submitting it to a Spark cluster is done via a spark-shell or spark-submit. Usually, this process is done manually either entering the cluster gateway machines or via a dedicated resource manager (e.g. SLURM, OpenStack). 6 https://livy.incubator.apache.org/
For users with little experience in cluster management and the Hadoop infrastructure, it can be challenging to run Spark. As an alternative, we introduce STATisfy7: REST Interface for DistLODStats. Instead of computing RDF statistics directly on the cluster the interaction is done via REST APIs (as it is depicted in the Figure 1).
The client side will create a remote Spark cluster for initialization, and submit jobs through REST APIs. Livy REST Server will then discover this job and send through remote procedure call (RPC) to SparkSession, where the code will be initialized and executed. In the meantime, the client will be waiting for the result of this job coming from the same direction.
Running the STATisfy is similar to using DistLODStats via spark-submit. The difference is that this shell is not running locally, instead, it runs in a cluster and transfers the data back and forth through the network.
For demonstrating the usage of the tool, we have deployed it on the comprehensive statistics catalogue LODStats8 which crawls RDF data from metadata portals such as CKAN dataset metadata registry. By doing this, it obtains a comprehensive picture of the current state of the Web of Data. As we use DistLODStats as an underlying engine for computing RDF statistics afterwards, the limitation was that the user has to interact with the cluster manually and initiate the job for computing such statistics. By using STATisfy REST interface, LODStats will interact with the cluster from anywhere which provides the capabilities necessary to do this without compromising on ease of use or security.
As it is shown on the Figure 2, user starts a session via REST API using Livy for submitting a job to the Spark cluster. 7 https://github.com/GezimSejdiu/STATisfy 8 http://lodstats.aksw.org/ The script (see Listing 1.1) contains a spark-submit configurations which is given in the format of a JSON structure with the necessary information like spark-submit. With the POST request POST /batches user could submit a request to DistLODStats using Livy server. Using Livy, STATisfy will then help to launch this request in the cluster. As a result, the output will be curled by their end in the format of VoID description. 3
In order to deepen their understanding of the data, many users require gathering statistical information about RDF datasets. This process becomes compute-intensive when the datasets grow in size. DistLODStats is a prominent solution, however, it requires setup and managing of the the cluster configuration and job submission. To make the process easier, we have introduced STATisfy, a tool for interacting with DistLODStats via a REST Interface. This way DistLODStats can be provided as-a-service, where users only send (HTTP) requests to the remote cluster and obtain the wished results, without having any knowledge about system access or cluster management. STATisfy is used for the LODStats project and an inclusion in the new DBpedia9 community release processes is ongoing.