Semantic Web technology has successfully facilitated many RDF models with rich data representation methods. It also has the potential ability to represent and store multimodal knowledge bases such as multimodal scene graphs. However, most existing query languages, especially SPARQL, barely explore the implicit multimodal relationships like semantic similarity, spatial relations, etc. We first explored this issue by organizing a large-scale scene graph dataset, namely Visual Genome, in the RDF graph database. Based on the proposed RDF-stored multimodal scene graph, we extended SPARQL queries to answer questions containing relational reasoning about color, spatial, etc. Further demo (i.e., VGStore) shows the efectiveness of customized queries and displaying multimodal data.
hasRegion
hasObject
(a)
inSynset relation hasAttr rdfs:Literal x y
Region h h x y rdfs:int rdfs:int rdfs:int rdfs:int rdfs:int rdfs:int rdfs:int rdfs:int (b) whasName
rdfs:Literal description w rdfs:Literal rdfs:Literal primaryColor <mm.color> semanticSimilarity ?mm.simrelativePosition ?mm.pos Count
?mm.count
Visual
Block
Visual
Block
(c)
methods in multimodal datasets. Although RDF has suficient representation capability to
describe multimodal data, the lack of multimodal semantic relationships in standard SPARQL
has become a major challenge in applying SP-based KBQA methods to multimodal domains.
Researchers have attempted to extend SPARQL for this purpose. For example, SPARQL-MM [
In this demo, we designed an ontology to organize the multimodal scene graph and store it with RDF. Furthermore, we implemented semantic multimodal SPARQL queries by extending the SPARQL engine, enabling the ability to answer questions related to multimodal information such as visual and spatial reasoning.
Visual Genome (VG) [
For querying convenience, we formalize the elements of VG in RDF. Fig. 1(a) shows the
designed ontology of RDF-stored VG (RDF-VG)2. Fig. 1(b) demonstrates properties of the defined
classes in RDF-VG. The properties (, , ℎ, ) determine the visual block of region or object
1http://visualgenome.org/VGViz/explore demonstrates the dataset.
2Due to space limitations, the detail of data processing and ontology organization are attached to the code repository.
by tailoring image. We store RDF-VG in gStore [
Traditional SPARQL engines (such as our backbone - gStore) cannot perform queries involving
multimodality and thus cannot directly answer such questions (e.g. what color is this cat?).
Therefore, we developed a VGStore extension based on standard SPARQL grammar and
pyparsing [
VGStore analyzes, matches, and replaces the clauses in the original SPARQL query that contain custom predicates. It recursively replaces all non-standard query patterns with the standard SPARQL syntax, and stores the replacement process in an operation stack temporarily. The standard SPARQL query can be handed over to the backbone graph database for execution.
Sort the results via customized variables, e.g., ORDER BY DESC(?mm.count).
Query Response
Result Gallery
After getting the result, the inverse operation is successively performed according to the staged replacement operations in the stack. Finally, the intent of the original SPARQL query would be restored. Fig. 2 (Right) demonstrated how a multimodal query containing the non-standard custom predicate variable is to be executed. Table 2 illustrates part of the supported query patterns in VGStore, covering questions involving color, counting, and relative position in the VG question-answer dataset, which account for 15.0%, 11.4%, and 7.0% of all questions, respectively. Other simple questions (e.g., “What is this?”) can be expressed and queried by native SPARQL directly, and the remaining non-factual questions (e.g., “What is this man’s motivation?”) or inference questions (e.g., “When was the picture taken?”) are out of scope in this demonstration.
Although VGStore successfully supports multimodal queries by extending virtual predicates, it still has some limitations. VGStore is written in Python, which brings additional latency in runtime, and it is possible to reduce the performance loss by native support in the SPARQL engine. In addition, when the VGStore queries large data, and there are multiple extended query statements, it would cause severe performance problems. This drives us to schedule and parallelize the third-party tool calls.
VGStore currently only supports several basic query patterns (as listed in Table 1) specialized for the RDF-VG dataset, and does not adapt to other VQA datasets, nor does it support richer query patterns. Therefore, our next steps for improvements include supporting more query patterns and extending the applicability of VGStore to more graph databases with large-scale multimodal graphs.
This paper presented VGStore, an extension to SPARQL for querying multimodal information on RDF. The demo showcased the web user interface of VGStore for querying multimodal SPARQL on RDF-VG, as shown in Fig. 3. With this demo, we provided a potential solution to the main challenge of SP-based multimodal KBQA and laid the foundation of multimodal knowledge graph. Our code of RDF-VG builder, preliminary parser and frontend of demonstration are available at https://github.com/pkumod/VGStore.
This work was supported by NSFC under grant 61932001, U20A20174. The corresponding author of this paper is Lei Zou (zoulei@pku.edu.cn). We sincerely thank reviewers for their valuable comments and advises.