We address the problem of benchmarking ontology matching and link discovery frameworks at large scale. In particular, we aim to ensure that the benchmarks generate comparable results for the various systems and approaches. Our solution lies in implementing our benchmarks into the HOBBIT benchmarking platform, which provide means for the unified benchmarking of Big Linked Data solutions.
to a certain experiment, i.e., the benchmark components as well as the benchmarked system. Figure 1 shows the layout of the HOBBIT platform components and how the different parts interact for the Linking and Spatial Benchmarks that are introduced in HOBBIT Link Discovery Track.2 Below we provide a brief overview of the operation of the platform in the case of Spatial and Linking Benchmarks.3
Evaluation
Module
User Management
Front End
Analysis data flow creates component The Platform Controller makes sure that the benchmark chosen by the user can be started and ensures that all nodes of the cluster are available. It communicates with the system to be benchmarked, ensures that it is working properly and generates the benchmark controller that is responsible for producing the data and task generators as well as the evaluation storage. The Data Generator produces the source dataset that is sent to the Benchmarked System, and the target dataset as well as the Gold Standard which are sent to the Task Generator. The Task Generator sends the target dataset to the Benchmarked System and forwards the Gold Standard to the Evaluation Storage. When the system finishes its task, it sends the answers to the Evaluation Storage. The Evaluation Module receives the system and the Gold Standard answers and returns the Key Performance Indicators for the experiment. 2 http://oaei.ontologymatching.org/2017/ 3 A detailed presentation of the orchestration of the different components can be found in [?]