WeSeE-Match is a simple, element-based ontology matching tool. Its basic technique is invoking a web search engine request for each concept and determining element similarity based on the similarity of the search results obtained. Multi-lingual ontologies are translated using a standard web based translation service. Furthermore, it implements a simple strategy for selecting candidate mappings interactively.
1.1
The idea of WeSeE-Match is to use information on the web for matching ontologies. When developing the algorithm, we were guided by the way a human would possibly solve a matching task. Consider the following example from the OAEI anatomy track1: one element in the reference alignment are the two classes with labels eyelid tarsus and tarsal plate, respectively. As a person not trained in anatomy, one might assume that they have something in common, but one could not tell without doubt.
For a human, the most straight forward strategy in the internet age would be to search for both terms with a search engine, look at the results, and try to figure out whether the websites returned by both searches talk about the same thing. Implicitly, what a human does is identifying relevant sources of information on the web, and analyzing their contents for similarity with respect to the search term given. This naive algorithm is implemented in WeSeE-Match.
Furthermore, WeSeE-Match uses a basic method for interactive matching, which tries to adaptively set the threshold for selecting the final alignment from the set of candidates. 1.2
The core idea of our approach is to use a web search engine for retrieving web documents that a relevant for concepts in the ontologies to match. For getting search terms 1 http://oaei.ontologymatching.org/2013/anatomy/ from ontology concepts (i.e., classes and properties), we use the labels, comments, and URI fragments of those concepts as search terms. The search results of all concepts are then compared to each other. The more similar the search results are, the higher the concepts’ similarity score.
To search for websites, we use URI fragments, labels, and comments of each concept as search strings, and perform some pre-processing, i.e., splitting camel case and underscore separated words into single words, and omitting stop words. For every search result, all the titles and summaries of web pages provided by the search engine are put together into one describing document. This approach allows us to parse only the search engine’s answer, while avoiding the computational burden of retrieving and parsing all websites in the result sets. The answer provided by the search engine contains titles and excerpts from the website (i.e., some sentences surrounding the occurance of the search term in the website). Therefore, we do not use whole websites, but ideally only relevant parts of those web sites, i.e., we exploit the search engine both for information retrieval and for information extraction.
For each concept c, we perform a single search each for the fragment, the label, and the comment (if present), thus, we generate up to three documents docfragment(c), doclabel(c), and doccomment(c). The similarity score for each pair of concepts is then computed as the maximum similarity over all of the documents generated for those concepts: sim(c1; c2) := maxi;j2ffragment;label;commentgsim (doci(c1); docj (c2)) (1) For computing the similarity sim of two documents, we compute a TF-IDF score, based on the complete set of documents retrieved for all concepts in both ontologies.
Using the TF-IDF measure for computing the similarity of the documents has several advantages. First, stop words like and, or, and so on are inherently filtered, because they occur in the majority of documents. Second, terms that are common in the domain and thus have little value for disambiguating mappings are also weighted lower. For example, the word anatomy will occur quite frequently in the anatomy track, thus, it has only little value for determining mappings there. On the other hand, in the library track, it will be a useful topic identifier and thus be helpful to identify mappings. The TF-IDF measure guarantees that the word anatomy gets weighted accordingly in each track.
The result is a score matrix with elements between 0 and 1 for each pair of concepts
from both ontologies. For each row and each column where there is a score exceeding
, we return that pair of concepts with the highest score as a mapping. Furthermore, the
filter chain explained in [
For multi-lingual ontologies, we first translate the fragments, labels, and comments
to English as a pivot language [
While the threshold is set to fixed values for the standard tracks in the evaluation,
we use an interactive method of selecting the threshold in the interactive track. We
use a binary search for selecting the threshold, as discussed in [
Given that the ordering of candidates is optimal, i.e., all wrong candidates have a lower confidence score than all correct candidates, that algorithm will yield a threshold that separates correct from incorrect candidates. 1.3
The 2012 version of WeSeE-Match [
The other new feature for this year’s evaluation was the inclusion of the interactive
post-processing method. The same code for handling interactive matching was also used
in the Hertuda matching system [
The parameter was set to 0.42 for multi-lingual and to 0.51 for mono-lingual matching problems in the non-interactive tracks. 1.4
The system is available from http://www.ke.tu-darmstadt.de/resources/ ontology-matching/wesee-match/. 2 2.1
benchmark The results from the benchmark track are not surprising. For those problems where labels, URI fragments and/or comments are present and contain actual terms, i.e., they 2 http://www.ke.tu-darmstadt.de/resources/jfreewebsearch 3 http://www.faroo.com/ 4 http://sourceforge.net/projects/webtranslator/ are not replaced by random strings, WeSeE-Match provides reasonable results. As soon as those strings are removed, the F-measure of WeSeE-Match drops, since no other evaluation (e.g., ontology structure) is used by WeSeE-Match.
The evaluation of runtime also reveals that WeSeE-Match is one of the slowest matching systems participating in the campaign. This is due to the fact that the search engine used restricts the usage to one request per second. Thus, WeSeE-Match spends a lot of idle time to fulfill that requirement. 2.2
anatomy and conference On anatomy, WeSeE-Match is one of the worst performing matchers, suffering both in precision in recall. It is in particular interesting to see the drop from last year’s performance (F-measure 0.829) to this year’s (F-measure 0.47), where the only significant change was the use of a different search engine. This shows that the choice of the web search engine in search-engine based matching can make a huge difference in the results.
In the conference track, the differences to last year are not that significant, ending at a comparable F-measure of 0.55, which makes WeSeE-Match an average matcher in the conference track. 2.3
multifarm For multifarm, the F-measure has dropped from 0.41 in OAEI 2012 to 0.15 in OAEI 2013. As discussed above, the performance on the English-only conference track, which underlies multifarm, has remained stable, so this is mainly an effect of the use of a different translation engine. Again, it becomes obvious that the choice of a different translation engine clearly influences the results.
In detail, the 2012 version of WeSeE-Match was capable of matching Chinese and Russian ontologies, because these languages were supported by the Microsoft Translator API, but not the Web Translator API. Furthermore, the results for Czech are significantly below the 2012 results, which shows a suboptimal support for that language in the Web Translator API. For the other languages, the results have remained on a similar level.
Since WeSeE-Match is not optimized for scalability (in particular, necessary waiting
times to avoid blocking from search engine providers slow down the matcher), it could
not be run on those larger tracks due to time limits. However, the approach does in
principle scale up to larger ontologies as well. In the OAEI 2012 campaign, for example,
WeSeE-Match was capable of completing the library track, when the time limit was set
to one week instead of one day [
It can be observed that the thresholds chosen by our algorithm are most often far from the optimum, which is also reflected in the F-measure achieved. In most cases, the selected threshold is higher than the optimal, i.e., the selection is biased towards precision. In fact, 1:0 is often chosen as a threshold. The reason is that that mapping elements such as conference#Conference = eads#Conference, which naturally re5 There are cases where the F-measure achieved by the actual selection is higher. This is due to the fact that in a post-processing step, all information obtained from the oracle is exploited by removing the wrong mappings and including the correct ones, even if they do not fall into the interval defined by the final threshold selection. Furthermore, in some cases, the precision is lower despite a higher threshold. This is due to the fact that the interactive track uses a slightly different reference alignment than the original conference track, on which the optimal thresholds have been determined. ceive a confidence score of 1:0, are rated wrong by the oracle, so that our binary search approach sets the threshold to the maximum possible, i.e., 1:0.
In general, while our interactive approach would work well for ideally sorted
candidates, its performance with real confidence scores provided by WeSeE-Match (and also
Hertuda) are not satisfying. Thus, using a greedy algorithm here is maybe not the best
choice, and other means to determine an optimal threshold, such as estimating the
Fmeasure based on user interaction, as discussed in [
In this paper, we have discussed the results of the 2013 version of WeSeE-Match. It can be observed that the choice for free services instead of commercial ones has changed the performance of WeSeE-Match for the worse. The trade-off between the use of commercial services and high-quality results is not easy to address.
Furthermore, it can be seen that a naive greedy approach for selecting a threshold parameter interactively does not provide satisfying results, which calls for more sophisticated methods.