Linked Data rarely takes into account multimedia content, which forms a central part of the Web. To explore the combination of Linked Data and multimedia, we are developing IMGpedia: we compute content-based descriptors for images used in Wikipedia articles and subsequently propose to link these descriptions with legacy encyclopaedic knowledge-bases such as DBpedia and Wikidata. On top of this extended knowledge-base, our goal is to consider a uni ed query system that accesses both the encyclopaedic data and the image data. We could also consider enhancing the encyclopaedic knowledge based on rules applied to co-occurring entities in images, or content-based analysis, for example. Abstracting away from IMGpedia, we explore generic methods by which the content of images on the Web can be described in a standard way and can be considered as rst-class citizens on the Web of Data, allowing, for example, for combining structured queries with image similarity search. This short paper thus describes ongoing work on IMGpedia, with focus on image descriptors.
Wikipedia centres around the curation of human-readable encyclopaedia articles,
where at the time of writing it contains about 38 million articles in almost 200
languages. In terms of structured content, the DBpedia project [
Because of this we are developing IMGpedia [
In previous work [
Aside from simply providing the URL of an image, we wish to compute visual descriptors for each image. These descriptors are vectors produced after performing di erent operations over the matrix of pixels in order to obtain certain features, such as color distribution, shapes and textures. Later, those vectors are stored as a part of a metric space in which similarity search can be performed. We are considering four di erent image descriptors which we have already implemented in Java, Python and C++, and we have proven to give equivalent results among di erent languages over a dataset of 2800 Flickr images. The rst three descriptors require a preprocessing step, which is to convert the image to greyscale using intensity Y = 0:299 R + 0:587 G + 0:114 B, where R, G and B are the values of each color channel: red, green and blue, respectively. The four descriptors we compute are: Gray Histogram Descriptor: images are partitioned in a xed amount of blocks.
Per each block a histogram of gray intensity is computed; typically intensity
takes 8 bit values. Finally, the descriptor is the concatenation of all histograms.
Oriented Gradients Descriptor: image gradient is calculated via convolution
with Sobel kernels [
DeCAF7: a Ca e neural network pre-trained with the Imagenet dataset [
The implementations and their dependency/compilation documentation can be found at https://github.com/scferrada/imgpedia under GNU GPL license. 3
In order to build the knowledge-base, preliminary steps must be done. First of all, it is necessary to obtain a local copy of the Wikimedia Commons image dataset so we can keep IMGpedia updated in the future. To do so, Wikimedia provides an rsync-protocol server for downloading the 21TB of images of the dataset. Currently we have downloaded about 19TB at an average speed of 550GB per day, corresponding to 14 million images downloaded from a total of 16 million. Each image is stored locally, along with its four descriptors.
We benchmarked the computation of descriptors in order to know the time such extraction will take. Experiments were performed on a machine with Debian 4.1.1, a 2.2 GHz (24 core) Intel(r) Xeon processor, and 120 GB of RAM. We computed the descriptors for a sub-folder of the Wikimedia Commons dataset, containing 57,377 images. The process is comprised of three steps: read the image and load it into memory, extract the descriptor itself and save the vector on disk. Two experiments were performed, one using a single execution thread and the other using 28 threads, where each thread handles an image at a time. The neural network for DeCAF7 was used in CPU mode (rather than GPU mode).
Results are shown in Table 1, where we see that DeCAF7 is the most expensive descriptor to run. While we can see an improvement of one order of magnitude in calculation time for the rst three descriptors, DeCAF7 does not bene t: the Ca e implementation for neural networks already uses multithreading for its computation on a single image, so assigning di erent images to di erent threads gives a slight overhead since multiple cores are already being exploited. On the present hardware, we estimate it would take 1.8 years to run DeCAF7 over all 16 million images; hence, we will have to select a subset of images on which to run this descriptor. 4
Linking IMGpedia with DBpedia, Wikidata, etc. To complete our knowledgebase and be able to infer new relations, we must combine these visual descriptors with the semantic entities of DBpedia and Wikidata. For this, we must extract the links between the articles and the images that they use since they are not present in DBpedia Commons and only present in DBpedia for some images. This is an ongoing e ort in which we are researching the possible options to perform this task: we hope to exploit the Wikimedia dumps, either to derive this information directly, or to parse it from the articles themselves. Subsequent to this, we wish to investigate using descriptor-based user-de ned-functions in SPARQL to enable visuo-semantic queries that combine, e.g., image similarity with knowledge-base queries. E cient algorithms for nding relations between similar images To facilitate querying IMGpedia, we propose to compute static relationships between images; e.g., if two images have a descriptor distance that tends to 0, then we can state that those images are nearCopy of each other. Other distance-based relations can be computed based on other xed thresholds, such as similar. The brute-force approach for nding all pairs of images within a given distance takes quadratic comparisons. Thus, the challenge is to do this e ciently, where we propose to explore: building an index structure for similarity search over the dataset; using approximate similarity search algorithms; using self-similarity join techniques; and so forth.
Labelling images for multimedia applications By linking IMGpedia to existing
knowledge-bases, we hope to label images with categories, types, entities, etc. Thus
IMGpedia could serve as a useful resource for the multimedia community; e.g.,
since we are using Ca e framework for neural networks to compute DeCAF7 [
In this short paper, we have given updates of our ongoing work on IMGpedia. We benchmarked four image descriptors for which reference implementations are made public. We discussed a number of open engineering challenges, as well as some open research questions we would like to study once the knowledge-base is completed. Acknowledgments This work was supported by the Millennium Nucleus Center for Semantic Web Research, Grant NC120004, and Fondecyt, Grant 11140900. We thank Camila Faundez for her assistance.