Computer Vision (CV) has recently achieved signi cant improvements, thanks to the evolution of deep learning. Along with advanced architectures and optimisations of deep neural networks, CV data for (cross-datasets) training, validating, and testing contributes greatly to the performance of CV models. Many CV datasets have been created for di erent tasks, but they are available in heterogeneous data formats and semantic representations. Therefore, it is challenging when one needs to combine di erent datasets either for training or testing purposes. This paper proposes a uni ed framework using the Semantic Web technology that provides a novel way to interlink and integrate labelled data across di erent data sources. We demonstrate its advantages via various scenarios with the system framework accessible both online and via APIs.4
Image datasets (e.g., ImageNet [
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
COCO dataset. Therefore, it is challenging when one needs to combine di erent datasets for training or testing purposes.
Recently, the Semantic Web technologies have o ered a exible and powerful
mechanism to integrate data from di erent sources [
Such shortcomings motivate us to build a uni ed knowledge graph (KG) to
realise the FAIR principles [
As a step towards the above vision, we propose a uni ed framework, called VisionKG, that facilitates a novel way to organise CV datasets. Our on-going implementation of VisionKG employs the Semantic Web technology to interlink labelled data across di erent datasouces. This demo paper will show its advantages via three scenarios: (i) exploring labelled images across datasets, (ii) building training pipelines with mixed datasets , and (iii) validating and testing a trained DNN.
Additionally, VisionKG system includes a front-end web interface ○7 that allows users to explore the KG as shown in the rst scenario of our demonstration. Furthermore, our framework contains a DNN Training Engine ○5 and an Evaluator ○6 . The image data as the tensor inputs for the Evaluator and the DNN Training Engine can be stored in a tensor storage (i.e. TensorDB). And the labels can be retrieved with SPARQL as demonstrated in the second scenario. The trained models are stored in our Model Zoo Directory and are evaluated by the Evaluator.
The tutorials for the training pipelines based on [
guration steps are automated so that Semantic Web developers familiar with SPARQL can easily try out the pipelines.
In the current version (by August, 2021), VisionKG has 67 million triples which cover Visual Genome, COCO, and KITTI datasets with the total number of 239k images, a million of labels (including ones for bounding-box), and hundreds of object categories. These categories are reused but aligned with Wikidata concepts/classes. VisionKG also contains millions of detection results that are evaluated with popular pretrained models such as Yolov3, Yolov4, E cientDet, FRCNN, etc. 3
The demonstration session will consist of three scenarios with a demonstration video at https://vision.semkg.org/iswc2021-demo.html. For all mentioned scenarios, we provide both Python APIs for developers and a Web interface for end-users in the training/testing phase and data exploration phase respectively. ○a Graph-based Exploration Across Visual Label Spaces: In this scenario, we demonstrate our web-based image explorer that is used for retrieving images using SPARQL (Figure 3 ○a ). The demonstration shows that with VisionKG, users will be able to search for images containing di erent labels, e.g., images that contain a cat and a person; or images that have 10 cars. ○b Building Training Pipelines with Mixed datasets: In the second part, we demonstrate the scenario to obtain a mixed dataset for training purposes (Figure 3 ○b ). A user starts a training pipeline by writing a SPARQL to retrieve the images and labels as desired. This includes an advanced setting like merging training data with the same label, e.g. Person, from di erent datasets (as shown in the right query of Figure 1). ○c Cross-dataset Validation and Testing: The third part demonstrates the scenario of getting the mixed dataset for validating purposes (Figure 3 ○c ). Similar to the scenario for training data, this includes the case where test data with the same labels are combined from di erent datasets. In advanced settings, one can test di erent models on speci c labels in one speci c dataset or over di erent datasets. This scenario is particularly useful in the case that developers want to target speci c applications. For example, one can test the trained models to detect Car on images of Car in crowded tra c scenes or in mountain areas.
Our proposed framework opens various new research venues. First, we plan to
build DNNs with a uni ed label space powered by VisionKG. Next, we will
extend VisionKG to analyse the robustness of DNNs models using testing samples
with semantic similarities via KG embeddings. Such KG embeddings can be
combined with visual features of DNN-based visual models to investigate the
interpretability and explainability of these models, e.g. [
Acknowledgments This work was funded by the German Research Foundation (DFG) under the COSMO project (ref. 453130567), the German Ministry for Education and Research via The Berlin Institute for the Foundations of Learning and Data (BIFOLD, ref. 01IS18025A and ref. 01IS18037A), and the German Academic Exchange Service (DAAD, ref. 57440921).