We present our publicly available semantic annotator bbw (boosted by wiki) tested at the second Semantic Web Challenge on Tabular Data to Knowledge Graph Matching (SemTab2020). It annotates a raw CSV-table using the entities, types and properties in Wikidata. Our key ideas are meta-lookup over the SearX metasearch API and contextual matching with at least two features. Avoiding the use of dump les, we kept the storage requirements low, used only up-to-date values in Wikidata and ranked third in the challenge.
the locally-deployed SearX1 metasearch API without using the Wikidata dump les.
Relying on Web-APIs releases us from storing large data sets and processing them locally. Moreover, the queries run every time on the most recent data without the need of developing any updating mechanism.
Related work. According to the results of SemTab 2019 [
Structure. The rest of this work is organized as follows: We describe how bbw works, show and discuss our results at SemTab2020, and make conclusions. 2
The annotator bbw is implemented in Python3 and is distributed as an open
source library with MIT License7. We added the command-line interface (CLI)
using argparse8 library. A simple graphical user interface (GUI) is developed
1 https://github.com/searx/searx
2 https://github.com/phucty/MTab
3 https://github.com/IBCNServices/CSV2KG
4 https://github.com/danielapoliveira/iswc-annotation-challenge
5 https://bitbucket.org/disco unimib/mantistable-tool-3
6 http://lod4all.net/frontend/index/applicationtop
7 https://pypi.org/project/bbw
8 https://docs.python.org/3/library/argparse.html
with Streamlit9. All scripts are available at the project page10. The bbw-CLI
is parallelized using GNU parallel [
Wikidata as a knowledge graph can be easiest thought of simple statements, i.e. triples containing a subject, a property, and an object/literal. For the subject and object the Wikidata items are identi ed by a QID (Wikidata Q identi er) and the property is identi ed by a PID (Wikidata property) which can be pre xed accordingly in order to receive an URI. The data types for literals in Wikidata are strings, quantities (i.e. numbers), times, coordinates and more. The actual data model of Wikidata11 is more complex but we do not focus on it.
We assume that a row in a CSV- le contains data from an item in Wikidata. We refer to that item as the subject or subject item. The rst cell in a row usually contains the label (rdfs:label) or an alias (skos:altLabel) of the subject. The other (tail) cells in a row correspond to the values of properties of that subject. Such a value can either be a literal (e.g. string, number, date) or the label (or an alias) of the object item. Three kinds of annotation tasks for each row can be de ned:
Entity annotation. If a cell contains a label or alias of an item in Wikidata, the cell can be annotated via the QID of this item.
Property annotation. If the values of two cells in a row are related via a property in Wikidata, the relation can be identi ed by its PID.
Type annotation. If a cell contains a label or alias of an item in Wikidata and the item has a statement with the property P31 (instance of), the cell's type can be annotated via the object's QID.
These annotations may not exist, are not necessarily unique and may be false positive due to mistakes in the values. The CPA and CTA tasks from the challenge are then an aggregation over all rows, such that hopefully all possible ambiguity or uncertainty vanishes.
Example: In our annotated example from Figure 1 we see in the rst column two subjects Q2119 (Mannheim) and Q23436 (Edinburgh). The second and forth column contain the objects annotated with their QIDs and the third column contains literals of type number. Between the rst row with the subjects and the other rows the property relations are annotated with P206, P2044, P131. The type annotation for Baden-Wurttemberg is Q1221156 (state of Germany) and the type annotation for the City of Edinburgh is Q15060255 (Scottish council area), but in the end the CTA for the whole column will result in Q12076836 (administrative territorial entity of a single country). Both items Q1221156 and Q15060255 are subclasses of Q12076836. 9 https://github.com/streamlit/streamlit 10 http://github.com/UB-Mannheim/bbw 11 https://www.mediawiki.org/wiki/Wikibase/DataModel 2.2
The data in some input tables have been changed by the challenge organizers. For example, the labels can contain typos or other artefacts: \Monnhem" instead of \Mannheim" or \dinbur" instead of \Edinburgh". The values can di er from the currently known ones in Wikidata, e.g. 9.78 instead of 9, or 1492-09-11 instead of 1492-10-12. Simple lookup and matching mechanisms fail when the data contain such errors. We do a meta-lookup rst and, if needed, perform several contextual matching steps. However, before going into details, we will explain these two key concepts here.
Meta-lookup. Meta-lookup is a lookup over many search and metasearch engines. SearX can search over more than 80 engines. The use of some of them is constrained technically or legally, see further information in Subsection 2.4. Thus, we use only a subset of the engines: Wikidata, Wikipedia, Wikibooks, Wikiquote, Wikisource, Wiktionary, Wikiversity, Wikivoyage, Startpage, Bing, DuckDuckGo, DuckDuckGo De nitions, eTools, Erowid, Etymonline, MyMemory, Qwant, Mojeek, Naver, DictZone.
We run a docker container locally with the SearX docker image12. The bbw sends a GET-request to the SearX API. SearX sends the requests to the search and metasearch engines, collects the responses, ranks them and returns the search results, suggestions, corrections and infoboxes as JSON data. The bbw collects those results and takes the best matches using the edit distance from di ib13 library. To add more candidates, bbw takes the best match in suggestions and corrections, sends them to the SearX API and collects the infoboxes. We use a few out of these candidates as an input for matching.
For example, SearX resolves \Monnhem" and \dinbur" to the correct spellings \Mannheim" and \Edinburgh".
Contextual matching. In contrast to a simple matching, a contextual matching is based on at least two features. For example, it can be 1) a label of a subject item and a label of an object item (or literal), 2) a label and type of an object item, or 3) a label of an object item (or literal) and a property.
The actual matching is performed as follows: We try an exact matching. If it does not give any results, we try a case-insensitive matching for strings. If there are still no results, we match a string with the edit distance. The names are matched ignoring abbreviations. The date, which is nearest to the value, is matched if it is within six months. All the numbers are matched within a 2 % range. 2.3
We implemented bbw as the 7-steps work ow illustrated in Fig. 2. Step 2 is performed for all rows, but some rows might not be annotated. These rows are then considered in Step 3 including all annotations we already have after Step 12 https://hub.docker.com/r/searx/searx 13 https://docs.python.org/3/library/di ib.html 2. The same pattern continues up to Step 6. The postprocessing Step 7 takes the annotations from all steps as input.
Step 1: Preprocessing. Encoding problems are xed by the ftfy library [
didates. We use the rst cell in the row for a meta-lookup, assuming that it is the subject's label. The resulting candidates are used to query the Wikidata SPARQL endpoint and we get all triples for the subjects with a label from the candidates' list. We match the value in a tail cell with object's labels in the resulting triples. If the matching is successful, we take the subject's QID, the subject's type and the property. If, additionally, the object is a literal, we take also the object's QID and the object's type.
Step 2 is performed for each row. We use the most frequent property per column as the input property for Step 3.
Step 3: Querying the objects' labels with their properties. An annotated row is a row with all property annotations found. Otherwise we say that the row is still unannotated. Step 3 is executed only for unannotated rows. We query the endpoint with the values of the tail cells and their properties inferred from Step 2. We match the value in the rst cell to the subject's label using the edit distance. Then, we match the resulting triples with the values in the tail cells as in Step 2.
Step 4: Querying the objects' labels of entity columns. We call a column an entity column if it has at least one type annotation inferred from previous steps. In contrast to Step 3, we do not use properties in Step 4. We query the endpoint with the values of the tail cells from all entity columns. It results in the subject's labels and the types of a subject and the objects. The cell values are matched as in Step 2. 14 https://docs.python.org/3/library/re.html
for this step we use the most frequent type per column inferred from previous steps. For every entity column the endpoint is queried using the value of a tail cell and its type. We take the object's QID from the received results.
Step 6: Querying the subject's types. For unannotated rows the endpoint is asked for the labels of all items with a certain type. We do this for the two most frequent type annotations of the rst column. This list of labels is matched with the subject's label via the edit distance. We send another query to the endpoint and receive all triples, which have the subject's label equal to one of the labels from the candidate items. The values in the tail columns are matched as in Step 2.
Step 7: Postprocessing. After Steps 1-6 we might have a few candidates for each task. We choose the most frequent property and entity for CPA and CEA tasks correspondingly. Postprocessing for CTA task is di erent. If there exists the unique most frequent type, we choose it. If there are multiple most frequent types, we nd the rst common class w.r.t. to the subclass relation using a special query15 to the endpoint with the SSSP class16 in com.bigdata.rdf.graph.analytics package17. However, if the query returns the generic class (Q35120), we choose the most frequent type. As the nal postprocessing step we choose only those annotations which are mentioned in the target les. 2.4
The annotator bbw has both technical and legal constrains related to the use of multiple API services.
Technical constraints. The current limits of the Wikidata SPARQL endpoint are ve parallel queries per IP, thirty error queries per minute and one minute runtime per single query.18 For example, in round 3 some of our requests were blocked by the endpoint with the error code 429 (Too Many Requests) due to the use of six parallel processes. We reduced the number of CPUs in round 3 after the deadline and added an extra request after waiting the number of seconds returned by the \Retry-After" header.
SearX can throw an error message due to its own technical reasons and due to constraints of a search engine. For example, it happens when an engine asks to resolve a captcha. We experienced that during our initial tests with Google Search. Thus, we did not use Google Search for our submissions.
We found a bug in SearX in the 2nd round and reported it19. It was xed shortly and we used the improved code in the next rounds. 15 https://w.wiki/h8X. The user Pasleim proposed it. 16 https://blazegraph.com/database/apidocs/com/bigdata/rdf/graph/analytics/SSSP.html 17 https://blazegraph.com/database/apidocs/com/bigdata/rdf/graph/analytics/packagesummary.html 18 https://www.mediawiki.org/wiki/Wikidata Query Service/User Manual#Query limits 19 https://github.com/searx/searx/issues/2188
Legal constraints. Possible legal restrictions for our usage of the search engines may occur from copyrights and sui generis database rights20. In accordance with the jurisdiction of the EUGH21, these rules do not restrict our usage of search engines, because \if the database maker makes the contents of that database accessible to third parties, [. . . ] his sui generis right does not enable him to prevent such third parties from consulting that database for information purposes"22. As described above, we \consult [. . . ] the database"23 for information purpose in contrast to build \a parasitical competing product"24 and therefore we do not harm the investment of the database makers. Though, the terms of use of the di erent search engines may restrict our usage.
The terms of use of the search engines were examined for restrictions concerning metasearch. While some search engines do not have written terms of use at all, and therefore no contractual restrictions a ecting their use via metasearch25 exists, others have terms of use without restrictions on metasearching26. Google Search is a special case: Whereas their former terms explicitly prohibited \metasearch[ing]", their current terms merely demand to not use their services \abusively". This change can be understood as a waiver of restricting the use of their service for metasearch. Unfortunately, the terms of use of some search engines include formulations, that can be understood as a refusal of metasearching without permission27. However, within the territorial scope of German contract law, even the most extensive of these restrictions could be seen as invalid according to § 307 I, II Nr.1 BGB as they are in violation of § 307 II in relation with basic principle of privileging scienti c research purposes stated in § 60d UrhG or at least for privileged scienti c use cases unenforceable according to § 60g I UrhG. 3
Round
1 2 (<1st deadline) 2 (>1st deadline) 2 (>2nd deadline) 3 (<deadline) 3 (>deadline) 4 (AG) 4 (2T) 4 (avg)
CPA annotations were submitted, and 2) a bug in the evaluator at SemTab2020 which counted the out-of-the target annotations. We interpret our low CEA score as a consequence of using only Step 2 in contextual matching.
Our CPA-score dropped in round 3 in comparison to round 2 due to a bug in bbw leading to the incorrect counts of the most frequent property. After xing it, our CPA scores were comparable to our CPA scores in round 2. Steps 3-6 of the work ow increased our CTA and CEA scores in round 3. We continued our tests after the deadline and found out that disabling Steps 3 and 4 increased our scores further.
Only steps 2, 5 and 6 for contextual matching were used in round 4. The
dataset contained two parts: AG { automatically generated and 2T { tough
tables [
We presented the semantic annotator bbw for matching tabular data to the Wikidata knowledge graph. The annotator is based on 1) meta-lookup over many search and metasearch engines, and 2) contextual matching with at least two features.
We were continuously developing bbw between rounds 2 and 4. Starting with the low CEA and CTA scores in round 2, we improved the work ow and ranked in second place in round 4. Thus, a meta-lookup is a competitive approach compared to processing the Wikidata dump les. In contrast to the latter, the meta-lookup keeps the storage requirements low and uses only up-to-date data in the knowledge graph.
Further improvements to bbw can be made by using more features [
The annotator bbw is distributed as an open source Python library with MIT license: https://pypi.org/project/bbw. It can be easily installed via \pip install bbw". The command line interface, a simple graphical user interface and a script for parallel computations are also provided. Everyone is welcome to adapt bbw at http://github.com/UB-Mannheim/bbw.
This work was funded by the Ministry of Science, Research and Arts of BadenWurttemberg through the project \Business and Economics Research Data Center Baden-Wurttemberg". 28 https://www.wikidata.org/wiki/Wikidata:SPARQL query service/query optimization