Data2Services: enabling automated conversion of data to services Vincent Emonet, Alexander Malic, Amrapali Zaveri, Andreea Grigoriu, and Michel Dumontier Institute of Data Science, Maastricht University, The Netherlands {firstname.lastname}@maastrichtuniversity.nl Abstract. While data are becoming increasingly easy to find and ac- cess on the Web, significant effort and skill is still required to pro- cess the amount and diversity of data into convenient formats that are friendly to the user. Moreover, these efforts are often duplicated and are hard to reuse. Here, we describe Data2Services, a new framework to semi-automatically process heterogeneous data into target data formats, databases and services. Data2Services uses Docker to faithfully execute data transformation pipelines. These pipelines automatically convert tar- get data into a semantic knowledge graph that can be further refined to conform to a particular data standard. The data can be loaded in a number of databases and are made accessible through native and auto- generated APIs. We describe the architecture and a prototype imple- mentation for data in the life sciences. Keywords: ETL · data transformation · data conversion · API 1 Introduction There is a large and growing amount of valuable data available on the Web. These data contain relevant information to answer questions and make novel predictions. However, data come in a myriad of formats (e.g. CSV, XML, DB), which makes them difficult to integrate into a coherent knowledge graph for unspecified downstream use. Unsurprisingly, many tools have emerged to fa- cilitate integration and analysis of diverse data. However, data transformation and integration often require substantial technical and domain expertise to do it correctly. Moreover, such transformations are hard to find and largely incompat- ible across tool chains. Users duplicate effort and are ultimately less productive in achieving their true objectives. Thus, easy, reliable, and reproducible trans- formation and publication of different kinds of data sources in target formats are needed to maximize the potential to find and reuse data in a manner that follows the spirit of the FAIR (Findable, Accessible, Interoperable, Reusable) principles [10]. An important use case in the life sciences is the collection and analysis of clin- ical and biomedical data to elucidate molecular mechanisms that underlie the 2 V. Emonet, A. Malic et al. pathology and treatment of human disease. The National Center for Advanc- ing Translational Sciences (NCATS) Biomedical Data Translator program1 is a iterative effort to develop new software architectures and corresponding data and software ecosystems to generate and explore biomedical hypotheses. This project utilizes over 40 different datasets that are dispersed and represented in largely incompatible data formats and standards. This lack of interoperability makes it difficult to find answers to even the simplest questions (e.g. how many treatable diseases are there?), and even the more sophisticated questions that are important for the implementation of personalized medicine. In this paper, we propose a software framework called Data2Services to (semi)automatically process heterogeneous data (e.g. CSV, TSV, XML) into a set of user-facing services (e.g. SPARQL endpoint, GraphQL endpoint, API). Data2Services aims to enhance the availability of structured data to domain ex- perts by automatically providing a variety of services to the source data. This open-source tool is composed of different Docker containers for effortless usage by expert and non-expert users alike. We demonstrate the utility of our tool by applying it to a specific query relevant to the Translator program. 2 Data2Services Framework The overall framework, illustrated in Figure 1, is based on 4 key steps: 1. Automatic generation of RDF data from input data 2. Loading of RDF data into an RDF store 3. Transformation of RDF data to RDF standard 4. Auto-configuration and deployment of data access services Data2Services makes use of the Resource Description Framework (RDF) [3] as a formal, shared, accessible and broadly application knowledge representation lan- guage, in line with FAIR principle (Findable Accessible Interoperable Reusable). RDF offers a common data format for both data and their metadata, as well as data and the vocabularies used to describe them. RDF statements are largely in the form of ”subject”, ”predicate”, ”object”, ”graph” quads, and can be se- rialized in a number of standard formats including CSV, TSV, JSON, XML, JSON-LD, Turtle, RDF/XML, N-Triples. RDF, in combination with the Web Ontology Language [1], thereby making them more expressive than simple file formats such as CSV, JSON, and XML. 2.1 Automated generation of RDF data The generation of RDF data from input data is performed in a semi-automated manner. A semi-automated approach is currently needed, because the while RDF data can be automatically generated from a wide variety of data formats, these 1 https://ncats.nih.gov/translator/about, which we shall refer to as “Translator program” in the rest of the paper. Data2Services 3 Fig. 1. Overview of Data2Services Framework resulting data may generate incorrect relations and lack the intended semantics. To produce more accurate RDF data, we need a language to either add the RDF semantics prior to the transformation or after the transformation. We use of relational mapping languages such as R2RML2 , a W3C standard, while we turn to SPARQL [9] to transform an existing RDF dataset to some community standard. Input data are processed by an R2RML processor along with the R2RML mapping file to generate the output data. Tabular files and relational database transforms: Tabular files (e.g.: CSV, TSV, PSV) are exposed as a relational database tables using Apache Drill3 . Each file is represented as a table, and each column is considered as the attribute (properties in RDF) of the table. Attributes are named after the column headers. R2RML mapping files are automatically generated from relational databases and Apache Drill accessible files through SQL queries issued by our AutoR2RML4 tool. Table names are used to generate the subject type identifiers while attribute names are the basis for predicate identifiers. An R2RML processor5 generates the resulting RDF using the mapping files in combination with the source data. XML transforms: We developed an xml2rdf6 tool to stream process an XML file to RDF, largely following the XML data structure. The output RDF captures 2 https://www.w3.org/TR/r2rml/ 3 https://github.com/amalic/apache-drill 4 https://github.com/amalic/AutoR2RML 5 https://github.com/chrdebru/r2rml 6 https://github.com/MaastrichtU-IDS/xml2rdf 4 V. Emonet, A. Malic et al. name of the XML node, its XPath location, its value, any children and their attributes. We envision to broader version of this tool to process any kind of tree-like document format (JSON, YAML). 2.2 RDF Upload The generated RDF data is then loaded into an RDF database through its REST API or SPARQL interface using RdfUpload7 . RdfUpload is a project that automatically uploads an RDF file into a specified GraphDB SPARQL or HTTP Repository endpoint. 2.3 Transform RDF to target model Finally, SPARQL insert are run to transform generic RDF representation of the XML data structure into the target data model. Those SPARQL queries are manually designed by a user aware of the input file structure, the target data model and the SPARQL query language. 2.4 Access the data through services Once data is transformed into the target database, it can be accessed using a variety of services. RDF databases typically provide a SPARQL endpoint to query RDF data. Here, we envision the development or inclusion of a variety of service inter- faces on top of the RDF databases. This can include the automatic generation of REST APIs to manipulate entities and relations in the knowledge graph8 , the encapsulation of SPARQL queries as REST operations 9 , the provision of standardized dataset metadata10 and API metadata [11], the use of standardized hypermedia controls 1112 , the use of graph query languages (GraphQL, Hyper- GraphQL, openCypher, Gremlin, SPARQL) and user interfaces1314 . 3 Data2Services Evaluation To demonstrate the Data2Services framework we use following query, relevant to the Translator program: Q1. Which drugs, or compounds, target gene prod- ucts of a [gene]?. To answer the query, we use two datasets from a pool of 40 datasets used by the Translator program. These datasets are: (i) HUGO Gene 7 https://github.com/MaastrichtU-IDS/RdfUpload 8 http://www.dfki.uni-kl.de/~mschroeder/demo/sparql-rest-api/ 9 https://github.com/CLARIAH/grlc 10 https://www.w3.org/TR/hcls-dataset/ 11 https://www.w3.org/TR/ldp 12 https://spring.io/understanding/HATEOAS 13 http://yasgui.laurensrietveld.nl 14 http://www.irisa.fr/LIS/ferre/sparklis/ Data2Services 5 Nomenclature Committee (HGNC)15 , a curated repository of HGNC-approved gene names, gene families and associated resources that provides detailed in- formation about genes and gene products, available in TSV format, and (ii) DrugBank, a resource containing detailed information on drugs and the gene products they target16 , available in XML format. Listings 1.1 and 1.2 show excerpts of the datasets used. We chose these particular datasets because they contain relevant information to answer the query. In the following, we outline the steps taken to execute the Data2Services pipeline on these two datasets using two different services to answer this query. 1 hgnc_id symbol name 2 HGNC :3535 F2 coagulation factor II , thrombin 3 HGNC :3537 F2R coagulation factor II thrombin receptor Listing 1.1. Excerpt of the HGNC TSV dataset. 1 2 < drugbank xmlns = " http :// www . drugbank . ca " version = " 5.1 " > 3 < drug type = " biotech " created = " 2005 -06 -13 " updated = " 2018 -07 -02 " > 4 < drugbank - id primary = " true " > DB00001 5 < name > Lepirudin 6 < target > 7 < id > BE0000048 8 < name > Prothrombin 9 < external - identifier > 10 < identifier > HGNC :3535 Listing 1.2. Excerpt of the DrugBank XML dataset. 3.1 Automated generation of generic RDF As the first step, we downloaded the HGNC dataset from ftp://ftp.ebi. ac.uk/pub/databases/genenames/hgnc_complete_set.txt.gz and the Drug- bank dataset from https://www.drugbank.ca/releases/5-1-1/downloads/ all-full-database17 . To execute the Data2Services pipeline, the downloaded files need to be uncompressed and placed in different directories mapped into /data directory inside the Apache Drill Docker container. For convenience we have created two Shell scripts to build the Data2Service pipeline Docker contain- ers, and start both Apache Drill and Ontotext GraphDB as services, as shown in Listing 1.3. 1 $ git clone -- recursive https :// github . com / MaastrichtU - IDS / data2services - pipeline . git 2 $ cd data2services - pipeline 3 $ git checkout tags / swat4ls 4 $ ./ build . sh 5 $ ./ startup . sh Listing 1.3. Buiding the pipeline Docker images from GitHub 15 https://www.genenames.org 16 https://www.drugbank.ca/ 17 Needs an account to be created to download. 6 V. Emonet, A. Malic et al. Before continuing, a repository needs to be created for GraphDB. It can be done by accessing GraphDB at http://localhost:7200 and go to: Setup → Repositories → Create new repository. Choose ”test” as repository ID and check ”Use context index”. To automatically run the data2services-pipeline that generates RDF out of the input dataset, the only requirement is to define a YAML configuration for each dataset as shown in Listing 1.4. The YAML allows the user to configure dif- ferent parameters such as the path to the input file, Apache Drill and GraphDB parameters. 1 W O R K I N G _ D I R E C T O R Y : "/data/hgnc/hgnc_complete_set.txt" # for HGNC 2 W O R K I N G _ D I R E C T O R Y : "/data/drugbank/full_database.xml" # for D r u g B a n k 3 4 JDBC_URL : "jdbc:drill:drillbit=drill:31010" 5 JDBC _CONTAI NER : "drill" 6 GRAPHDB_URL : "http://graphdb:7200" 7 G R A P H D B _ R E P O S I T O R Y : "test" 8 G R A P H D B _ U S E R N A M E : "import_user" 9 G R A P H D B _ P A S S W O R D : "test" Listing 1.4. YAML configuration file for HGNC or DrugBank Then, Data2Services can be executed by providing the YAML configuration file to the run.sh script in the data2services-pipeline directory with the command $ ./run.sh /path/to/config.yaml. HGNC processing: As a result of executing the YAML configuration file, a R2RML mapping file is produced for HGNC, in the directory where the input file is stored, as shown in Listing 1.5. 1 @prefix rr : < http :// www . w3 . org / ns / r2rml # >. 2 <# HgncMapping > 3 rr : logicalTable [ rr : sqlQuery """ 4 select row_number () over ( partition by filename ) as a u t o r 2 r m l _ r o w n u m 5 , columns [0] as ‘ HgncId ‘ 6 , columns [1] as ‘ ApprovedSymbol ‘ 7 from dfs . root . ‘/ data / hgnc / h g n c _ c o m p l e t e _ s e t . tsv ‘;"""]; 8 rr : subjectMap [ 9 rr : termType rr : IRI ; 10 rr : template " http :// data2services / data / hgnc / h g n c _ c o m p l e t e _ s e t . tsv /{ a u t o r 2 r m l _ r o w n u m }"; 11 ]; 12 rr : p r e d i c a t e O b j e c t M a p [ 13 rr : predicate < http :// data2services / data / hgnc / h g n c _ c o m p l e t e _ s e t . tsv / HgncId >; 14 rr : objectMap [ rr : column " HgncId " ]; 15 ]; Listing 1.5. Excerpt of the R2RML mapping file for HGNC TSV file Then the R2RML implementation is executed to extract the data from the TSV input file to produce the generic RDF from the R2RML mapping files as shown in 1.6. 1 PREFIX d2s : < http :// data2services / data / hgnc / h g n c _ c o m p l e t e _ s e t . txt / > 2 d2s :3535 d2s : symbol " F2 " ; 3 d2s : name " coagulation factor II , thrombin " . Listing 1.6. Excerpt of the produced triples for HGNC TSV file Data2Services 7 DrugBank processing: Executing the Data2Services pipeline on the Drug- bank XML file produces a generic RDF model representing the DrugBank XML structure as shown in Listing 1.7. 1 d2sdata :3 a7d47b8 - c734 rdf : type d2smodel : drugbank / drug / drugbank - id . 2 d2sdata :0 c1f3d83 -5563 d2smodel : hasChild d2sdata :3 a7d47b8 - c734 . 3 d2sdata :3 a7d47b8 - c734 d2smodel : model / hasValue " DB00001 " . Listing 1.7. Excerpt of triples generated from Drugbank XML structure 3.2 RDF Upload As the next step, RdfUpload is executed to load the RDF data to the “test” repository of the GraphDB service running on http://localhost:7200. RdfU- pload has so far only been tested on GraphDB, but we envision to develop it as a generic tool that can be used on most of the popular RDF databases like Virtuoso. 3.3 Mapping to the BioLink model As part of the standardization of knowledge graphs, the Translator project has created BioLink18 , is a high level datamodel of biological entities (genes, diseases, phenotypes, pathways, individuals, substances, etc) and their associations. We crafted two SPARQL INSERT queries 19 20 to generate new RDF datasets that are compliant with the BioLink model as shown for HGNC in Listing 1.8. 1 PREFIX d2s : < http :// data2services / data / hgnc / h g n c _ c o m p l e t e _ s e t . ts / > 2 PREFIX rdfs : < http :// www . w3 . org /2000/01/ rdf - schema # > 3 PREFIX bioentity : < http :// bioentity . io / vocab / > 4 INSERT { 5 ? hgncUri a bioentity : Gene . 6 ? hgncUri rdfs : label ? geneName . 7 ? hgncUri < http :// purl . org / dc / terms / identifier > ? hgncid . 8 ? hgncUri bioentity : id ? hgncUri . 9 ? hgncUri bioentity : s y s t e m a t i c _ s y n o n y m ? symbol . 10 } WHERE { 11 SELECT ? s ? hgncid ? geneName ? symbol ? hgncUri { 12 ? s d2s : ApprovedName ? geneName . 13 ? s d2s : HgncId ? hgncid . 14 ? s d2s : Appro vedSymbo l ? symbol . 15 ?s ?p ?o . 16 BIND ( iri ( concat (" http :// identifiers . org /" , lcase (? hgncid ) ) ) AS ? hgncUri ) 17 }} Listing 1.8. SPARQL construct query to convert HGNC to BioLink The transformed RDF data contains drug data from Drugbank, linked to gene data from HGNC in a manner that is compliant with the BioLink model. The next step is to use the available interfaces to answer the research question. 18 https://biolink.github.io/biolink-model/ 19 https://github.com/vemonet/ncats-grlc-api/blob/master/insert_biolink_ drugbank.rq 20 https://github.com/vemonet/ncats-grlc-api/blob/master/insert_biolink_ hgnc.rq 8 V. Emonet, A. Malic et al. 3.4 Services Executing the Data2Services pipeline enables the BioLink data to be queried through two services: (i) SPARQL and (ii) an HTTP API. SPARQL: The original question can be answered by executing a SPARQL query which retrieves all drugs linked to a given gene, as shown for the gene “coagulation factor II, thrombin” in Listing 1.9. The results show that 23 drugs are affecting this gene. Then any other question can be translated from natural language to a SPARQL query that will extract the requested informations from the graph. 1 PREFIX rdfs : < http :// www . w3 . org /2000/01/ rdf - schema # > 2 PREFIX bioentity : < http :// bioentity . io / vocab / > 3 SELECT distinct ? gene ? geneLabel ? g en e P r o d u c t L a b e l ? drug ? drugLabel 4 { ? gene a bioentity : Gene . 5 ? gene bioentity : id ? geneId . 6 ? gene rdfs : label ? geneLabel . 7 ? gene bioentity : h a s _ g en e _ p r o d u c t ? geneProduct . 8 ? geneProduct rdfs : label ? g e n e P r o d u c t L a b e l . 9 ? drug bioentity : affects ? geneProduct . 10 ? drug a bioentity : Drug . 11 ? drug rdfs : label ? drugLabel . 12 FILTER regex ( str (? geneLabel ) , " coagulation factor II , thrombin ") . 13 } Listing 1.9. SPARQL query to answer Which drugs, or compounds, target gene products of thrombin. HTTP API: We used grlc21 to expose the SPARQL query to answer the ques- tion as an HTTP web service. grlc is a lightweight server that takes SPARQL queries curated in GitHub repositories, and translates them to Linked Data Web APIs. Users are not required to know SPARQL to query their data, but instead can access a web API. We implemented a Swagger API with one call to re- trieve URI and label of drugs that affect a gene defined by the user using its HGNC identifier e.g: HGNC:3535. This API is available at http://grlc.io/ api/vemonet/ncats-grlc-api. 4 Related Work A significant amount of research has been conducted in the domain of data conversion to a standard, semantically meaningful format. OpenRefine22 offers a web user interface a to manipulate tabular data and generate RDF23 . How- ever, this user must be knowledgeable about RDF data modeling to generate sensible RDF data. Karma [6] allows customizable RDF conversion through a 21 https://github.com/CLARIAH/grlc 22 http://openrefine.org/ 23 https://github.com/fadmaa/grefine-rdf-extension/releases Data2Services 9 web browser, but producing high quality mappings from ontologies with var- ious structured sources (formats including databases, spreadsheets, delimited text files, XML, JSON, KML) requires expert ontology knowledge. Another ex- ample of user controlled RDF conversion is Sparqlify24 , which depends on the non-standardized Sparqlification Mapping Language(SML) and can be difficult for inexperienced users [5]. Other approaches also involve the transformation of XML data by using XSLT stylesheets [2] or templates [7]. SETLr [8] is another tool which can convert a variety of data types to RDF using JSLDT, together with Jinja Templates and Python Expressions, and is a great option for profes- sionals familiar with those languages. 5 Conclusions, Limitations and Future Work In this paper, we describe Data2Services, a software framework to automati- cally process heterogeneous data with multiple interfaces to access those data. This open-source framework makes use of Docker containers to properly con- figure software components and the execution of the workflow. We demonstrate the utility of Data2Services by transforming life science data and answering a question that has arisen in the Translator program. This work represents a preliminary effort in which there are several limita- tions. While the automatic conversion of data does produce an RDF graph, it lacks a strong semantics that is obtained by mapping data to domain ontologies. Our strategy in this work was to show how a second transformation, expressed as a SPARQL INSERT query over the automatically converted data, could be mapped to a community data model (BioLink) for use by that community. We also acknowledge that it may be possible to edit the autogenerated R2ML file to produce BioLink data, but this would not be available for the XML conversion. Indeed, it would be desirable to have one declarative language for the trans- formation of a greater set of initial data format into RDF. RML [4] promises such a language, with existing processors for relational, XML, JSON data. How- ever, preliminary work with the RML processor revealed that it does not scale currently with large data files because it loads the entire data into memory into so called logical structures, which enable joining data from so called logi- cal sources25 . Nonetheless, efforts to craft friendly user interfaces that hide the complexity of mapping languages could be useful to generate mappings from non-traditional users for all kinds of data files. Our future work will explore the automated capture of metadata such as provenance of the data collection and processing in a manner that is compliant to community standards, the incorporation of data quality assessments on RDF data [12], and the evaluation of the framework in terms of performance with other use cases and data sources, both large and small. 24 https://github.com/SmartDataAnalytics/Sparqlify 25 http://rml.io/spec.html#logical-join 10 V. Emonet, A. Malic et al. 6 Acknowledgements Support for the preparation of this project was provided by NCATS, through the Biomedical Data Translator program (NIH awards OT3TR002019 [Orange] and OT3TR002027 [Red]). Any opinions expressed in this document are those of the Translator community writ large and do not necessarily reflect the views of NCATS, individual Translator team members, or affiliated organizations and institutions. References 1. Bechhofer, S., van Harmelen, F., Hendler, J., Horrocks, I., McGuinness, D.L., Patel- Schneider, P.F., Stein, L.A.: OWL Web Ontology Language Reference. Tech. rep., W3C, http://www.w3.org/TR/owl-ref/ (February 2004) 2. Breitling, F.: A standard transformation from xml to rdf via xslt. Astronomische Nachrichten: Astronomical Notes 330(7), 755–760 (2009) 3. Brickley, D., Guha, R.: Rdf vocabulary description language 1.0: Rdf schema. Tech. rep., W3C Recommendation (2004) 4. Dimou, A., Vander Sande, M., Colpaert, P., Verborgh, R., Mannens, E., Van de Walle, R.: RML: A Generic Language for Integrated RDF Mappings of Het- erogeneous Data. In: Proceedings of the 7th Workshop on Linked Data on the Web (Apr 2014), http://events.linkeddata.org/ldow2014/papers/ldow2014_ paper_01.pdf 5. Ermilov, I., Auer, S., Stadler, C.: Csv2rdf: User-driven csv to rdf mass conversion framework. In: Proceedings of the ISEM. vol. 13, pp. 04–06 (2013) 6. Knoblock, C.A., Szekely, P., Ambite, J.L., Goel, A., Gupta, S., Lerman, K., Muslea, M., Taheriyan, M., Mallick, P.: Semi-automatically mapping structured sources into the semantic web. In: Extended Semantic Web Conference. pp. 375–390. Springer (2012) 7. Lange, C.: Krextor–an extensible xml rdf extraction framework 8. McCusker, J.P., Chastain, K., Rashid, S., Norris, S., McGuinness, D.L.: Setlr: the semantic extract, transform, and load-r. PeerJ Preprints 6, e26476v1 (2018) 9. Prud’hommeaux, E., Seaborne, A.: SPARQL Query Language for RDF. W3C Recommendation (January 2008), http://www.w3.org/TR/rdf-sparql-query/, http://www.w3.org/TR/rdf-sparql-query/ 10. Wilkinson, M., et al: The FAIR Guiding Principles for scien- tific data management and stewardship. Scientific Data 3 (2016). https://doi.org/http://doi.org/10.1038/sdata.2016.18 11. Zaveri, A., Dastgheib, S., Wu, C., Whetzel, T., Verborgh, R., Avillach, P., Ko- rodi, G., Terryn, R., Jagodnik, K., Assis, P., Dumontier, M.: smartapi: Towards a more intelligent network of web apis. In: Blomqvist, E., Maynard, D., Gangemi, A., Hoekstra, R., Hitzler, P., Hartig, O. (eds.) The Semantic Web. pp. 154–169. Springer International Publishing, Cham (2017) 12. Zaveri, A., Rula, A., Maurino, A., Pietrobon, R., Lehmann, J., Auer, S.: Quality assessment for Linked Data: A survey. Semantic Web Journal (2016)