Aiming at multi-modal knowledge representation we construct a dataset with pairs of digital photos of objects. We manually score image pairs for semantic object similarity. A pre-trained ImageNetbased deep neural network predicts the objects and we use the output to estimate the similarity between two images. With a linkage between the neural network and Wikidata, we augment the model and incorporate knowledge graph information into the similarity measure. We compare the machine-based predicted similarity with the human-based semantic similarity.
At the interface between machine learning and knowledge graphs lies
interesting avenues of research and the combination of the two techniques may yield
increased task performance. For instance, state-of-the-art results were obtained
with a combination of word embedding models and the ConceptNet knowledge
graph on a classical word similarity task [
In purely text based semantics several word similarity datasets exist, see,
e.g., [
WordSimilarity-353_Test_Collection_(State_of_the_art). 2 https://sites.google.com/site/imagesimilaritydata/
In the following, we describe the construction of a semantic similarity dataset (imagesim-353 ) and the combination of a pre-trained deep learning neural network used together with the Wikidata knowledge graph as a means for machinebased visual semantic similarity estimation. 2 2.1
Our inspiration for constructing a visual semantic similarity data is the
wordsim353 dataset [
To collect the images, we used the \Random le" MediaWiki facility in the
Wikimedia Commons wiki at https://commons.wikimedia.org/wiki/Special:
Random/File, and iterated until a suitable image was identi ed. Using the
Wikidata Query Service, we also queried for images that was associated with a
Wikidata item linked to a ImageNet WordNet synset with a SPARQL query [
We sought to select images and pair them so the range of similarities was roughly equally represented. The images were not selected equally frequent. In the pairing, 163 images were used once, 108 twice, while one image was used 11 times. 2.2
We constructed a small Flask web application running on the local computer
for scoring the similarity of 353 image object pairs. For scoring the similarity,
we stressed the di erence between similarity and relatedness [
We use the Keras4 Python deep learning framework to analyze the images. There
are various models implemented in Keras which are pre-trained on the ImageNet
image recognition dataset. They yield a 1000-dimensional output representing
probabilities over 1000 prede ned and xed classes. We use the ResNet50 model
[
From the output of the neural network for the i'th image, yi, we compute the correlation coe cient between image pairs, yi and yj, i.e., using the entire distributed representation of an image, rather than just the most probable class label. To explore the information content in the class probabilities for the nondominant classes we also compute the correlation coe cient from the elementwise logarithm of the output vectors, ln(yi) = y~i and ln(yj) = y~j.
As a form of (low) baseline measure we compute the similarity between two images as the correlation coe cient between pixel values. We expect this method to perform poorly, but better than chance.
To incorporate a knowledge graph into the similarity computation, we query the Wikidata Query Service with a SPARQL query to obtain a list of properties for each Wikidata item that is linked to ImageNet via a WordNet synset URI. SELECT ? item ( SAMPLE (? synset_ ) AS ? synset ) ( GROUP_CONCAT (? pid ) AS ? properties ) WITH {
SELECT ? item ? uri WHERE { ? item wdt : P2888 ? uri .
FILTER STRSTARTS ( STR (? uri ),
" http :// wordnet - rdf . princeton . edu / wn30 /") } } AS % items WHERE { 3 https://www.cl.cam.ac.uk/~fh295/simlex.html 4 https://keras.io/
INCLUDE % items ? item ? p [] . ? property wikibase : d i r e c t C l a i m ? p .
BIND ( SUBSTR ( STR (? p ) , 37) AS ? pid )
BIND ( CONCAT ( 'n ' , SUBSTR ( STR (? uri ) , 39 , 8)) AS ? synset_ )
}
GROUP BY ? item
With this at hand, we build a bag-of-properties matrix, W, where each row
corresponds to one of the 1000 ImageNet classes (i.e., the ResNet50 outputs
and ?item variable in the SPARQL query) and each column corresponds to a
speci c Wikidata property. The element values of the matrix are set to the count
of the number of times a Wikidata property is used for a speci c class. The
bagof-properties matrix is scaled via the t df transformer in scikit-learn [
To compare the machine-based similarity score with the human ground truth,
we use Spearman's correlation, | the standard measure for word similarity [
We nd a Spearman correlation between human similarity scores and machinebased similarity score of 0.62 for correlation coe cients on untransformed neural network output and on 0.67 for correlation coe cients on logarithm-transformed output. The simple baseline on correlation of image (color) pixel values yields 0.12. We did not generally see an improvement when we projected the neural network output through the transformed bag-of-properties matrix, obtaining a resulting matrix on 192 columns. For instance, with Y~ W~ = Z, a Spearman correlation of 0.60 showed a deterioration in performance. However, combining the logarithm-transformed output matrix with a projection, [ln(Y); ln(YW~ )], to a total of 1192 columns provided a slight improvement in performance to a Spearman correlation of 0.70. Figure 1 shows the swarmplot between the human scoring and similarities computed with this 1192-dimensional space. 4
Objects in the natural world are usually \surrounded by context" which make it di cult to dissociate the object from the context. For instance, liquids such as co ee are usually found in a container, such as a co ee mug. For a viewer (human or machine), it may be ambiguous whether a photo of co ee is a photo of co ee or a co ee mug. Another example is a bridge over a river: Is it the bridge or the river that is the object? The distinction between similarity and relatedness is harder to maintain for these cases. Another problem for the determination of a ground truth similarity may be the recognizability of the objects and the required detailed knowledge of the similarity between the objects depicted. Furthermore, what similarity should be assigned to an object photographed from di erent angles or at di erent times? There are probably no de nite answer to this question, cf. the philosophic discussion of temporal parts.
Many extensions of this work are possible: Keras contains several other pretrained models apart from ResNet50. The evaluation of these models should be straightforward. The use of the last internal layers of the neural network|instead of just the output|could also be interesting to explore.
Our dataset is small, restricting the use of machine learning to optimize a model for similarity. Crowd-sourcing could create a larger dataset, Multiple human annotators of the similarity might also improve the quality of the human scoring and give an indication of the variability of the human visual semantic similarity scoring.
Acknowledgment: This research is funded by the Innovation Foundation Denmark through the DABAI project.