The wide adoption of the RDF data model demands e cient and scalable query processing strategies. For this purpose distributed programming paradigms such as Apache Spark on top of Hadoop are increasingly being used. Unfortunately, the Hadoop ecosystem lacks support for Semantic Web standards, e.g. reading an RDF serialization format, and thus, bringing in RDF data still requires a large amount of e ort. We therefore present PRoST-loader, an application which, given a set of N-Triples documents, creates logical partitions according to three widely adopted strategies: Triple Table (TT), Wide Property Table (WPT) with a single row for each subject, and Vertical Partitioning (VP). Each strategy has its own advantages and limitations depending on the data characteristics and the task to be carried out. The loader thus leaves the strategy choice to the data engineer. The tool combines the exibility of Spark, the deserialization capabilities of Hive, as well as the compression power of Apache Parquet at the storage layer. We managed to process Dbpedia (approx. 257M triples) in 3.5 min for TT, in approx 3.1 days for VP, and in 16.8 min for WPT with up to 1,114 columns in a cluster with moderate resources. In this paper we aim to present the strategies followed, but also to expose the community to this open-source tool, which facilitates the usage of Semantic Web data within the Hadoop ecosystem and which makes it possible to carry out tasks such as the evaluation of SPARQL queries in a scalable manner.
RDF is a widely used data model to build and publish knowledge graphs, many
of which are extremely large [
The most adopted logical partitioning strategies for RDF are Triple Table (TT),
Wide Property Table (WPT), and Vertical Partitioning (VP) [
it by means of a HiveQL query. This is an opportunity to lter out di erent kinds of erroneous entries. As one example, a line in the NT document which cannot be matched by the three groups de ned in the regular expression results in null values in the table. Moreover, when more than three elements are present in one line, multiple resources or literals might end up being mapped all together within the object column. We lter out these kinds of entries. Also, the predicate names are used later as column names in the Property Table. The maximum length of a column name in Hive is 128 characters. Therefore, predicates whose length exceeds this limit are discarded by ltering out rows in which they appear. The non-discarded rows are inserted into the Triple Table using the INSERT OVERWRITE TABLE clause, which at the end is stored in Parquet format. (C) To build the Wide Property Table rst we get a list of distinct predicates from the Triple Table. We need to distinguish between predicates with cardinality <1,1> and those with cardinality <1,N>. For the latter case the objects for a given subject will be stored in an array (Spark's ArrayType), which is later mapped to Parquet's logical types. Another control consists in making sure that all predicates are distinct, even in a case-insensitive scenario, because these are used as column names and Hive's table and column names are case-insensitive. Finally, the list of predicates and their cardinalities is used to compose a query and build the Property Table which is written to Parquet. Spark is used mainly for this task since we needed a complex user-de ned aggregate function. (D) Finally, using the list of predicates we build the VP tables, one at a time. Each table is created and populated using HiveQL and stored using Parquet.
We performed our tests on a small cluster of 10 machines, 1 master and 9 workers, connected via Gigabit Ethernet connection. Each machine is equipped with 32GB of memory, 4TB of disk space and with a 6 Core (12 virtual cores) Intel Xeon E5-2420 processor. The cluster runs Cloudera CDH 5.10.0 with Spark 2.2 on Ubuntu 14.04. Yarn, the resource manager, in total uses 198 GB and 108 virtual cores. The most relevant settings of the cluster can be found in our github repository along with the code3. We used our loader to create the logical partitions for Dbpedia 3.5.1. This consists of 23 les (36.5 GB) loaded into an HDFS folder. The overall number of triples is 257,869,688. In the rst stage 126 triples were removed which contained predicates with more than 128 characters. The number of distinct predicates is 39,554, of which 13,336 predicates have cardinality <1,N>. The loader was able to identify 1,328 pairs of predicates with colliding names in Hive, e.g. dbpedia:nationalchampion, and dbpedia:nationalChampion, and removed one from each pair. The Triple Table was written to HDFS in 3.55 minutes. The Vertical Partitioning nished in 3 days, 3 hours and 42 minutes. On average 8.7 tables were written per minute. For the Wide Property Table we had to remove the infobox-properties which reduced the number of properties to 1,114, since the cluster resources were not su cient to run it on the 39K predicates. The limited WPT nished in 16.8 minutes. 5
We believe there exists a de cit of tools for processing Semantic Web data in distributed frameworks such as Hadoop. Our open-source tool shows that choosing the right loading strategy is crucial to avoid wasting cluster resources and time. Hive and Parquet compression enable fast building of the logical partitions, while Spark's exibility in de ning distributed functions is essential to model data. Our tool currently uses the default physical partitioning from Hive and Spark. In the future we will explore other physical partitioning strategies.