The transportation domain is characterised by a multitude of di erent formats to represent data, thus creating a problem of (lack of) interoperability between systems and a need for data conversion. In order to cope with the speci c requirements of production systems, special attention should be given to performance and scalability of conversion solutions. In this paper, we present a thorough evaluation of the Chimera framework for semantic data conversion through declarative mappings, in both a dataset and message conversion scenarios. We illustrate the experimental results and we o er our considerations and recommendations for the successful implementation of conversion pipelines exploiting Semantic Web technologies.
Interoperability in the transportation domain is the main challenge to
provide travellers with multi-modal door-to-door travel solutions involving di
erent transport service providers. The Shift2Rail initiative, nanced by the
European Commission, has been addressing this challenge within the Innovation
Programme 41 by de ning an Interoperability Framework to enable a seamless
data exchange between di erent transport stakeholders [
a standard A to a standard B, exploiting Semantic Web technologies and the reference ontology.
The main challenges in the adoption of this approach are the de nition of
the mapping rules and the performance and scalability requirements related to
the technical implementation of converters. In the SPRINT project
(Semantics for PerfoRmant and scalable INteroperability of multimodal Transport) we
addressed these two aspects [
The performance and scalability evaluation of a converter should consider the two main conversion scenarios in the transportation domain: the harmonisation of static data, like timetables and scheduled transport, and the transformation of dynamic data, like journey planning messages. The Dataset Conversion (batch conversion) scenario considers the case where medium-sized to big archives of transportation data, usually static data, should be converted. Conversions are required with low frequency, but the conversion procedure should minimise the resource usage to obtain scalability with respect to the size of the dataset. The Message Conversion (service mediation) scenario considers the case where a message, usually dynamic data, should be converted to guarantee communication between two di erent systems at runtime. A small amount of data is converted for each request, but the conversion procedure should minimise the processing time to avoid introducing overhead and to obtain scalability with respect to a high frequency request rate.
To address these two scenarios, we designed and developed a generic, modular
and exible solution, the Chimera framework2 [
The reminder of the paper is organised as follows: Section 2 deals with preliminaries and related works; Section 3 describes the testing activities designed to evaluate converters; Section 4 discusses the results obtained and elicits a set of recommendations for the performance and scalability of converters; Section 5 draws the conclusions and de nes potential future works. 2
A semantic data conversion procedure, following the any-to-one centralised
mapping approach [
A possible implementation of the described conversion procedure is based on a Object-Relational Mapping (ORM) approach using unmarshalling/mar
shalling libraries to obtain an in-memory representation of data as objects, and
then exploiting annotations in the code to map each class and attribute to class
and properties of the reference ontology. This approach is implemented in
RDFBeans3, Empire4, and was studied for the de nition of the ST4RT converter.
This implementation improved the pre-existing approaches providing more
exibility in the de nition of the annotations to address complex mappings in both
the lifting and the lowering phase [
Chimera is an open-source framework based on Apache Camel5 and adopts
a modular approach to build exible pipelines for conversions based on
Semantic Web technologies. The main goal of Chimera is to minimise the amount of
code to be written, allowing to create a converter just by con guring the various
blocks o ered by the framework. In particular, this paper evaluates conversion
procedures adopting the following Chimera blocks: (i) a lifting block for
materialisation through a custom version of the RML-Mapper library6 and employing
RML [
proach is available in the repository.
more detailed overview of the Chimera framework is available in [
RML is a mapping language that extends the R2RML recommendation [
A real-world use case and evaluation of the discussed approach for batch data
conversion in the transportation domain is presented in [
A comparison of Chimera with others state-of-the-art tools for knowledge
graph materialisation is available in [
In this section we present the performance and scalability evaluation, for both dataset and message conversion, and the testing infrastructure.
Dataset Conversion For the evaluation in the dataset conversion scenario,
we chose the GTFS-Madrid-Bench9 [
The planned testing activities considered a roundtrip conversion GTFS !
Linked GTFS [
(CSV, JSON or XML), while the lowering phase always produces CSV les. In the conversion pipeline we used an in-memory RDF repository to store and query the materialised RDF graph. However, we also evaluated the impact on performances of using an external triplestore.
Message Conversion To evaluate the message conversion, we selected a realistic journey planning test case involving a deployed web service, converting a response message from the HaCon VBB journey planning endpoint12 to the TRIAS format13. During the lifting, a VBB TripList message (representing travel solutions for a requested itinerary, dimension 43KB) is mapped onto the IP4 IT2Rail ontology14 through RML mappings. The materialised graph is stored through an in-memory RDF repository. During the lowering, the data modelled through the IT2Rail ontology are mapped onto a TRIAS TripResponse message using a Apache Velocity template with speci c SPARQL queries. We employed the JMeter tool15 to test the performances of the converter with a increasing size of concurrent requests (number of threads: 10, 50, 100, 150, 500, 1000, 2500, 5000; ramp-up period: 1 second; loop count: 1).
Testing infrastructure All tests were run using a Docker16 container to guarantee reproducibility on a machine running CentOS Linux 7, with Intel Xeon 8-core CPU and 64 GB Memory. We set a memory limit to 24GB, a timeout of 24 hours and no limits on CPU usage. We run each test 5 times averaging the obtained results. We also monitored resource usage of containers in execution. 4
In this section, we discuss the performance and scalability test results and
illustrate the identi ed bottlenecks and their possible solutions. Additional details
and data are available in [
In Table 1, we provide the complete results for the dataset conversion scenario
considering di erent sizes and di erent formats. Execution times are measured
12 http://fahrinfo.vbb.de
13 https://github.com/VDVde/TRIAS
14 http://www.it2rail.eu/
15 https://jmeter.apache.org/
16 https://www.docker.com/
17 A complete report discussing the core and nal releases is available in [
We also tested di erent con gurations of the pipeline (not fully reported here). The results in Table 1 refer to the best-performing con guration for each format and size. As a preliminary comment, we notice that the overall conversion time is mainly in uenced by the lifting phase (as also shown later in Table 2).
Scale 1 5 10 50 100 Input Size 4.9 MB 10.42 MB 23.64 MB 106.1 MB 247.5 MB Num. Triples 565,489 1,800,911 3,663,380 18,009,100 36,633,800 Release core nal core nal core nal core nal core nal CSV 22.77 10.83 164.95 55.37 544.41 154.13 11,624.45 3,441.67 TO OoM JSON 50.41 30.89 659.11 394.21 2,471.29 1,467.70 66,003.36 34,901.65 TO TO XML TO 16.26 TO 123.29 TO 434.05 TO 12,648.65 TO OoM
The results show a consistent performance improvement in the conversion time of the nal release with respect to the core one, respectively for CSV (2x), JSON (1:6x) and XML (> 1; 000x) data sources. The nal version was able to convert CSV, JSON and XML datasets up to 100 MB and generating 18 M triples with the available resources, thus demonstrating its capability to process even large dataset of static transportation data.
The nal version mainly improved with respect to the lifting phase, due to the adoption of di erent libraries to access the input data sources, a simpli ed mechanism to handle the generated triples (triples are directly written to the RDF repository of the Chimera pipeline during the lifting procedure) and the introduction of di erent concurrency strategies in the RML block (at the record level and/or triple map level). However, concurrency naturally increases CPU usage and memory consumption, thus, in some cases, it may be preferable to use single-threading, with a longer conversion time but lower resource usage.
Additional tests, available in [
CSV conversion time outperforms the JSON/XML one because of the impact of the libraries to access the input datasources in the lifting procedure. Also intuitively, accessing rows in a CSV le and iterating over them is less expensive than querying a JSON/XML le resolving a JSONPath/XPath query and iterating over nodes retrieved. This aspect a ects the execution and worsens with the size of the le to query, hence the di erences in timings. Considering the XML data format, the core version did not complete the conversion within the 24 hours timeout even for the 1-xml dataset, while the nal converter completed it up to the 50-scale size, performing even better than the JSON conversion thanks to the new XML parser. The conversion of the JSON dataset obtained more limited improvements between the core and nal version, mainly because the performances of the JSON parser limit the advantages of introducing concurrency. The results obtained for JSON and XML show not only the importance of lifting mapping optimisation via concurrency, but also the impact on performances of the parsing procedure.
Finally, we checked the impact of join conditions in the RML mappings on the conversion time. Mappings de ned in the GTFS-Madrid-Bench maximise the number of joins among the di erent les composing a GTFS feed. As in SQL queries, a growing number of items (in this case, the scale of the input dataset) increases non linearly the number of needed comparisons for a non-optimised join operation. In the RML mappings it is often possible to avoid the usage of join conditions by adopting the same IRI generation pattern in di erent triple maps. With this approach, we managed to optimise the RML mappings for the GTFS-Madrid-Bench dataset. Two RML mappings producing exactly the same knowledge graph were used to compare conversion times in Chimera with and without joins conditions. The results showed that the optimised mappings reduced the conversion time up to 2=3 in the case of 50-scale CSV (6205.25s with joins, 2269.62s without joins).
Table 2 compares, for the 50-scale CSV dataset, the results obtained considering an in-memory and an external RDF repository (triplestore). It is important to point out that the performance strictly depends also on the employed triplestore, in this case GraphDB Free v9.0.0 18. On one hand, the usage of an external repository with incremental writes ( nal-csv-ext ) drastically reduces the memory consumption (2x reduction) with respect to the same con guration run using an in-memory repository ( nal-csv ). The conversion time using incremental writes is higher, but it can be acceptable because it comes with a substantial decrease in resource usage; the adoption of a triplestore without concurrency limitations19 would likely bring an even better time-resource trade o . In any case, it is important to take into account that an external repository implies also its own resource usage.
Table 2 also details the lifting and lowering time considering the di erent formats for the 50-scale dataset in the nal release. As previously commented, lifting times are in uenced by the input data format whereas the lowering times are similar since the same lowering mappings are executed on the same knowledge graph. In general, our results show that lifting through materialisation is a time-consuming approach in case of large datasets. However, it is important to highlight that the really good lowering performance is mainly due to the possibility of querying a materialised knowledge graph. The lowering time is higher when using an external repository ( nal-csv-ext ) due to the concurrency limitations for the SPARQL queries in the template. However, complex queries in 18 https://graphdb.ontotext.com/ 19 Free version of GraphDB is limited to two concurrent queries
Conversion Lifting Lowering Max Mem Max CPU
time (s) time (s) time (s) (GB) Usage (%)
core-csv 11,624.45 11,583.49 40.97 18.84 185.56
nal-csv 3,441.67 3,407.39 34.28 18.58 516.51
nal-csv-ext 3,784.34 3,659.39 88.95 9.63 314.61
nal-json 34,901.65 34,861.97 39.69 18.45 153.97
nal-xml 12,648.65 12,614.62 34.03 18.44 540.01
the templates applied to large knowledge graphs can e ectively bene t of an
external triplestore [
In Table 3 we report the results of the tests performed for the message conversion scenario. For the nal release, we compare two con gurations: nal-m-1 introducing concurrency in lifting only at the record level, and nal-m-2 introducing it also at the triple map level. The best performing con guration ( nal-m-1 ), resulted in a conversion times in the order of 100ms, thus perfectly acceptable in a runtime scenario. Moreover, a 5x improvement in the conversion time was obtained from the core to the nal release, thanks to the already mentioned optimisations, but also thanks to a speci c con guration for the message conversion scenario that executes the lowering transformation avoiding input/output operations on the lesystem. The results obtained for nal-m-2, demonstrate that for small messages it is preferable not to introduce excessive concurrency, since the structure initialization time (e.g., threads) is not compensated by an overall speedup. Finally, it is worth noting the very limited resource usage, especially memory, in the di erent tested con gurations.
Table 4 reports the scalability test results with an increasing number of concurrent requests. A single instance of the converter managed to handle up to 100 concurrent requests per second (at 150 concurrent requests the processing time overcomes one second), handling successfully also a peak of 2,500 concurrent requests. After 3,000 pending requests the queue mechanism provided by
Number of concurrent requests 10 50 100 150 500 1,000 2,500
Apache Camel starts dropping requests. The maximum length of the queue can be increased, however, a high number of pending requests is associated with noticeable performance degradation. The low resource usage and the optimisations resulted in very good scalability for increasing workloads, even considering a single instance of the converter. 4.3
The reported results show the improvements obtained in the development of the Chimera framework for semantic data conversion pipelines and its applicability to both the dataset and message conversion scenarios in the transportation domain. Moreover, from the performed testing activities, we derive a set of additional recommendations for performance and scalability using the proposed approach.
Performance Considering the RML-based lifting, the speci c RML mappings
de ned for a conversion pipeline (join conditions, number of triple maps, number
of logical sources, path to access the records, . . . ) can in uence the performances
of the lifting portion. As a result, the choice of the pipeline con guration should
take into account the trade-o between the conversion time and the resource
usage, for example with di erent concurrency strategies. In particular, in some
cases the gain in conversion time obtained does not justify the higher resource
usage. Additional recommendations for the RML-based lifting are:
{ to e ciently exploit concurrency, it is important to tune the di erent
parameters, e.g., the number of concurrent threads adopted;
{ concurrency may cause issues in case of RML mappings generating blank
nodes without specifying a deterministic identi er (each thread may assign
di erent random identi ers to the same blank node generating it multiple
times);
{ the presence of many functions in the RML mappings (RML and FnO
integration [
Scalability In the dataset conversion scenario, the scalability of the solution is
limited by memory consumption due to the materialisation of large knowledge
graphs. A potential alternative could exploit virtualisation techniques for the
lifting phase, but the state-of-the-art tools are still not mature enough to be
employed in a conversion scenario [
In the message conversion scenario, to cope with a higher number of concurrent requests, it is possible to deploy more than one instance of the converter exploiting a load balancing mechanism. However, standards in the transportation domain usually require dealing with a large set of di erent message types which implies de ning a high number of converters. In this situation, an e cient scalability strategy is to deploy (multiple instances of) a universal converter which is able to dynamically select and execute the relevant mappings with respect to the input/output message.
Semantic interoperability in the transportation domain can be addressed e ectively exploiting Semantic Web technologies to enable the communication between actors and the de nition of new services for travellers. To guarantee the adoption by relevant stakeholders, it is however extremely important to address their performance and scalability requirements. Considering both the dataset and message conversion scenario, this paper identi ed two test cases from the transportation domain and evaluated performances and scalability of the semantic data conversion approach using the Chimera framework, with a declarative approach based on RML mappings for lifting and on Apache Velocity templates and SPARQL queries for lowering.
Our analysis showed the potential and exibility of the Chimera solution: in the dataset conversion scenario, we managed to generate and handle knowledge graphs with millions of triples; in the message conversion scenario, we obtained very low conversion times and a proved robustness also with hundred of concurrent requests per second. Moreover, on the basis of our tests, we de ned a set of recommendations to improve performance and scalability with the discussed approach.
As future work, we would like to investigate and implement in Chimera
further optimisations for the lifting procedure, for example adopting the data
structures de ned in the SDM-RDFizer tool [
The presented research was partially supported by the SPRINT project (Grant Agreement 826172) and the RIDE2RAIL project (Grant Agreement 881825), co-funded by the European Commission under the Horizon 2020 Framework Programme. 20 https://developers.google.com/transit/gtfs-realtime