Evaluation of semantic web technologies at large scale, including ontology matching, is an important topic of semantic web research. This paper presents a web-based evaluation service for automatically executing the evaluation of ontology matching systems. This service is based on the use of a web service interface wrapping the functionality of a matching tool to be evaluated and allows developers to launch evaluations of their tool at any time on their own. Furthermore, the service can be used to visualise and manipulate the evaluation results. The approach allows the execution of the tool on the machine of the tool developer without the need for a runtime environment.
Evaluation of matching tools aims at helping designers and developers of such
tools to improve them and to help users to evaluate the suitability of the
proposed methods to their needs. The Ontology Alignment Evaluation Initiative
(OAEI)3 has been the basis for evaluation over the last years [
4 Semantic Evaluation at Large Scale http://about.seals-project.eu/ entry point to the application. This interface is deployed as a web application in a Tomcat application-server behind an Apache web server. It invokes the BPEL work ow, which is executed on the ODE5 engine. This engine runs as a web application inside the application server.
The BPEL process accesses several services that provide di erent functionalities. The validation service ensures that the matcher web service is available and full lls the minimal requirements to generate an alignment in the correct format. A redirect service is used to redirect the request for running a matching task to the matcher service endpoint. The test iterator service is responsible for iterating over test cases and providing a reference to the required les. The evaluation service provides measures such as precision and recall for evaluating the alignments generated by the matching system. A result service is used for storing evaluation results in a relational database.
Once the web service matcher implementation has been deployed and published at a stable endpoint by the tool developer, its matching method can be invoked within the BPEL work ow. For that reason, an evaluation starts by specifying the web service endpoint via the web interface. This data is then forwarded to BPEL as input parameters. The complete evaluation work ow is executed as a series of calls to the services listed above. The speci cation of the web service endpoint becomes relevant for the invocation of the validation and redirect services. They implement internally web service clients that connect to the URL speci ed in the web user interface.
Test and result services used in the BPEL process require to access additional data resources. For accessing the test data, the test web service can access the metadata describing the test suite, extracts the relevant information and forwards the URLs of the required documents via the redirect service to the matcher, which is currently evaluated. These documents can then be accessed directly via a standard HTTP GET-request by the matching system. The result web service uses a connection to the database to store the results for each execution of an evaluation work ow. For visualizing and manipulating the stored results we use an OLAP (Online Analytical Processing) application, which accesses the database for retrieving the evaluation results. Results can be re-accessed at any time e.g., for comparing di erent tool versions against each other. 3
For demonstration purposes, we have extended the Anchor-Flood [
6 Available at http://mindblast.informatik.uni-mannheim.de:8080/sealstools/ aflood/matcherWS?wsdl tutorial/tutorial5/, there is complete information about how to create a valid matcher.
In order to start an evaluation, one must specify the URL of the matcher service, the class implementing the required interface and the name of the matching system to be evaluated (Figure 1). Three of the OAEI datasets have been selected, namely Anatomy, Benchmark and Conference. In this speci c example, we have used the conference test case.
Submitting the form data, the BPEL work ow is invoked. It rst validates the speci ed web service as well as its output format. In case of a problem, the concrete validation error is displayed to the user as direct feedback. In case of a successfully completed validation, the system returns a con rmation message and continues with the evaluation process. Every time an evaluation is conducted, results are stored under the enpoint address of the deployed matcher (Figure 2).
The results are displayed as a table (Figure 3), when clicking on one of the three evaluation IDs in Figure 2. The results table is (partially) available while the evaluation itself is still running. By reloading the page from time to time, the user can see the progress of an evaluation that is still running. In the results table, for each test case, precision and recall are listed. Moreover, a detailed view on the alignment results is available (Figure 4), when clicking on the alignment icon in Figure 3. 4
Automatic evaluation of matching tools is a key issue for promoting the development of ontology matching. In this demo we have presented a web-based tool for automatic evaluation of matching systems that is available for the research community at any time. The major bene t of this service is to allow developers to debug their systems, run their own evaluations, and manipulate the results immediately in a direct feedback cycle.
Acknowledgements The authors are partially supported by the SEALS project (IST-2009-238975).