The matching of ontologies is a problem solved by many di erent matching systems using various algorithms. To test di erent methods or a complete system or to compare the systems among each other a common data set is needed. There are already some benchmarks containing many test scenarios available, but they mainly focus on concept-based matching algorithms or on instance matching (the process of nding similar instances). Instance-based methods cannot be tested su ciently, because the ontologies do not contain instances at all or the number of instances is very small. In this poster we introduce a new benchmark, ONTOBI, which makes use of Wikipedia to create a benchmark test series with ontologies that contain many instances.
Ontologies represent knowledge in a structure way. In many application areas there is a need to match ontologies, e.g. in the eld of query answering on heterogeneous sources. In the past many matching systems have been developed to cope with this problem, an overview can be found in [ES07]. Generally, the used methods can be divided into concept-, structure- and instance-based approaches and in most cases a matching system uses a combination of those approaches. To test the e ciency of single algorithms or complete systems, or to compare systems among each other based on a common data set, appropriate ontologies and test cases have to be developed.
Currently, there are already some benchmarks, e.g. the one published by the OAEI [OAE09], the STBenchmark [ATV08] or the IIMB [FLMV08]. The OAEI benchmark ontologies only contain a very small number of instances for a few concepts. In 2009 some instance-matching tasks have been added, but they cannot easily be adapted for instance-based ontology matching methods, because the concept information remains the same for all tests and the reference alignment only contains instance-to-instance correspondences. The instances of the ontologies created with the STBenchmark are created arti cially and do not contain instance variations/modi cations and the IIMB benchmark is again designed for instance matching tasks and only provides a small ontology with no changes on the concept level. The lack of a reasonable amount of instances in existing benchmarks motivates the development of an additional benchmark, which we present in this paper. 2. DEVELOPING THE BENCHMARK Under consideration of the advantages and disadvantages of existing benchmarks and regarding the personal experiences with matching systems, we de ned a set of requirements that our new benchmark should ful ll. First of all, the general requirements for evaluation frameworks as described in [ES07] should be considered, i.e. systematic procedure, continuity, quality and equity, dissemination and intelligibility. Additionally, we formulated some more criteria: bigger amount of instances, varying structure, di erent data formats, spelling mistakes and 1:n mappings. As described before, our focus is set to a huge amount of instances to enable the evaluation of instance-based matching methods, but we also want to provide a complete benchmark with which all kinds of methods and systems can be tested.
Similar to the OAEI benchmark, our ONTOBI benchmark consists of di erent test scenarios, whereas a reference ontology provides the basis for each test case. The reference ontologies gets modi ed by applying one or more of the modi cations described in Table 1. The modi cations are applied on di erent parts of the ontology (instance set, concept names, etc.), and in most cases only a subset of the according data set is changed. This modi ed ontology has to be matched against the original reference ontology, an overview of the process is given in Figure 1. The reference alignment is given as well (the format for this alignment is borrowed from the Ontology Alignment API [Euz06]), such that the results can be evaluated by e.g. calculating Precision and Recall.
We decided to use Wikipedia as the basis for our reference ontology, because Wikipedia provides a lot of information within its structured infoboxes, and developed a tool, that extracts concepts, attributes, relations and instances out of these infoboxes. The reference ontology consists of 17 classes, 13 object properties and 128 data type properties. It is constructed around di erent concepts describing the geographical structure of our world, i.e. countries, states,
modi cation spelling mistakes changed format di erent naming conventions suppressed comments
no data types overlapping data sets
subset data sets expanded structure attened structure another language random names
synonyms disjunct data sets
cities and languages. Additionally there are concepts describing di erent kinds of entertainment instruments, such as books, movies or songs with their corresponding authors, actors and singers. Another part of the ontology deals with companies and their products, e.g. cars, mobile phones or magazines. The most important issue for ONTOBI is the instance set. Currently, the reference ontology contains more than 3500 instances, but the number grows constantly. The di erent combinations of modi cation that we applied on the reference ontology and hence the di erent test cases can be found in Table 2. All modi cations are executed manually by using an ontology editor like Protege [Pro09]. The complete benchmark, including the ontologies and the reference alignmenta, are available for download on demand.
The work on this benchmark is still in progress. Currently, we enhanced the quality of our benchmark by directly deriving the ontologies from DBPedia [LBK+09] (see www.dbs.cs.uni-duesseldorf.de/projekte/ONTOBI). We also increased the number of concepts, attributes and instances, the modi cations have been slightly changed and the test cases have been reorganized. Additionally, an ontology modi cator has been implemented which automatically applies selected transformation on the reference ontology. In future modi cation(s)
M S1 I2 L4 L2 L3 H1 H2 work we want to focus on developing more complex modi cations on the instance level.