2
Our approach to incorporate semantics of structured knowledge bases into the image classification task can be subdivided into two main steps. First is a technique to obtain accurate concept representations, which will define the vector space for our architecture. Next is mapping visual features of images obtained via a visual-semantic embedding model to the vector space of the concept representations. The goal of this is to have both image representations and concept representations in a single vector space, so that they can be compared during the image classification task. 2.1
Concept representations define the vector space for our architecture. Figure 2 a) shows this process in a flow chart where, initially we have a graph G with nodes V and edges E. We assume either that all edges are is-a (directed specialisation) edges or some edges are labelled as such. The reason for this is that, we identify is-a as the basic form of a specialisation hierarchy that would be used to inform the semantics of the concepts in concern. Of course, moving forward the edges can have more complex relational representations, which is beyond the scope of our current study. Also we do not restrict the structure of the graph. An important thing to consider when selecting a graph is that, the nodes should contain the training image label classes of the classification task (Li in Figure 2 b)). And the graph should represent the relations of concepts to each other in the considered domain.
Chakaveh Saedi et al. [SBRS18] recently proposed a technique to compute embeddings, specifically on WordNet [Mil98], that could represent lexical semantics of the nodes. Their study showed how the resulting embedding vectors are superior to the performance of word embeddings [MCCD13] in semantic similarity tasks. In our approach we make use of these findings to obtain the embeddings as our concept representations.
G is converted to an adjacency matrix M such that, if two words in G, wi and wj , are directly related by an edge, the entry Mij is set to 1 (otherwise 0). Also to account for words that are not directly connected to each other, M is further enriched taking distantly connected nodes and aggregated as in Equation 1.
MG = 1 X( M )n = (I n=0
M ) 1
Where, n is the length of the path between two nodes, (< 1) is a decay factor that determines the effect of path length on M . Longer the path between two words (larger the n), lesser it affects MG. MG is normalised using L2-norm and reduced to a set of vectors with a lower dimensionality using Principal Component Analysis (PCA). These vectors will be our concept representations. 2.2
After defining the concept vector space as shown in Section 2.1, we want to train our visual-semantic embedding model to map input images to the relevant concept representations. Figure 2 b) summarises the procedure where, in the visual feature representation learning step, the learnt concept representations RC are used to inform the semantics of the concepts.
We employ a deep neural network here, with an additional project layer at the top of the network that outputs a representation matching the dimensionality of the concept representations. The projection layer is trained using the cosine similarity between the predicted embeddings and the pre-computed concept representations as a measure of loss according to Equation 2.
loss(I; RC ) = 1 cos(E(I); RC ) (2)
In Equation 2, I and RC are the image and the corresponding concept representation of the image label class respectively. E is the function that represents both feature extraction and mapping of the projection layer. We minimise the value of 1 minus the cosine similarity during training since the more similar the two vectors are, they should result in a cosine similarity closer to 1. After training, we use the function E to calculate the visual embeddings of all the training images and store them. RI in Figure 2 b) represents the set of these embeddings. 2.3 2.3.1
We introduce a novel evaluation framework which we name as Taxonomy-Aware Measure for Errors (TAME), that analyses the classification errors with respect to the property of subsumption extracted from the hierarchical structure of the knowledge base used. Subsumption is formed by the is-a relations in the knowledge base, for example, if C is related to D via an is-a link, we can say that D subsumes C and that all properties of C is also present in D. We make use of this property to check if our architecture is effective in generalising to the concepts that subsumes the ground-truth classes of the images. We can check this at different levels of subsumption going up the is-a hierarchy.
In the presence of background knowledge, it makes sense to distinguish bad from not so bad errors; for example, if the system predicts a subsumer class of the ground-truth class instead of the ground-truth class itself, this is a less bad error than if it predicted a less closely related class. For example, referring to Figure 3, if the system classifies an image of a barrel as a vessel, TAME will consider it to be almost correct. TAME captures this aspect of our system and quantifies image classification results with semantically accurate errors. Results are shown in Section 3.2. Next a novel approach to evaluate zero shot learning is presented. We divide the set of chosen zero-shot classes into two sets, namely, Sibling and N onSibling classes. The notion of being sibling classes is derived from the property of sharing common subsumer classes. For example, referring to Figure 3, vessel and bag share the same subsumer, container. Hence we categorise such classes as Sibling classes in the zero-shot evaluation, while the other being N onsibling classes. During evaluation, we distinguish the difference in performance of these two sets in the image classification task. 3
In this study, we make use of ILSVRC 2012 [RDS+15] as the image dataset and WordNet [Mil98] as the knowledge base for obtaining concept representations. Since ILSVRC 2012 uses WordNet entities as the ground truth class labels for the objects in the images, it qualifies as a reliable structured knowledge base for all the concepts found in ILSVRC 2012. WordNet is a lexical ontology for English language that consists over 120k concepts, 25 types of relations between these concepts and over 155k words (lemmas) that are categorised to nouns, verbs, adjectives and adverbs. In this implementation, we adopt the same approach of [SBRS18] and extract a sub graph of 60k words from all parts-of-speech in WordNet. All relations were considered and weighted equally during the embedding calculation according to Section 2.1. The dimensionality is chosen to be 850 for the resulting concept representations [SBRS18]. We employ a deep residual network [HZRS16] (Resnet 50), pre-trained for an image classification task with the ILSVRC 2012 dataset. The softmax prediction layer at the top of the network is replaced with a projection layer, as explained Section 2.2. The projection layer acts as a linear transformation that maps a 512 dimensional feature vector of the image to a 850 dimensional concept embedding produced in our previous step. To evaluate our goal of semantically informed image classification, we use the process shown in Figure 4.
Throughout our evaluation we compare the results with the DeViSE [FCS+13] architecture trained on the same classes and tested against the same settings. DeViSE takes a similar approach to embed semantics in image classification with the only difference of concept representations being word embeddings [MCCD13]. This gives us the opportunity to compare the difference in performance when concept representations are calculated using structured knowledge versus unstructured text. During the training phase of visual-semantic embedding model (denoted by E in Figure 2 b)), we choose 300 randomly selected classes out of the ILSVRC 2012 1K dataset [RDS+15]. The reason behind selecting the 300 classes was that, the nodes of WordNet graph considered during the concept extraction phase included these 300 classes. We compare the Hit@k image classification accuracies of our approach with a standard Softmax image classification network baseline (Resnet50 [HZRS16]) and DeViSE [FCS+13], both trained on the same training classes. The results are shown in Table 1.
From the results in Table 1, we see that although the softmax model attains higher accuracies in image classification for the training set, both DeViSE and our proposed model show higher accuracies for the test set, with our model having the highest accuracy in both cases of Hit@1 and Hit@5. The test set consists of unseen images of the same training classes and higher accuracies with this set implies the superior generalisation ability of the systems. Nevertheless, for the focus on this study we understand that our visual-semantic embedding model, in fact has promising performance in image classification.
In addition to the above results, our system inherits the capability of incorporating semantics of the classes with their relationships according to the structured knowledge base used during training (i.e. WordNet). We move on to our novel evaluation framework introduced in Section 2.3.1 to evaluate this capability. ILSVRC 2012 provides the hierarchy of the selected classes according to the hypernym tree in WordNet, similar to the example is shown in Figure 3. We obtain three sets of subsumers for all the 300 classes selected, at three levels, each above the 300 training classes, named as 1-step, 2-step and 3-step subsumers. As the levels increase, the classes become more general in meaning. Note that these subsumer classes are not used during the training of the systems. The system will have to deduce the semantic relationship informed by the concept representations provided during the training process in order to accurately identify the more general subsumer classes. The prediction is taken as correct if the system outputs either the ground truth class or any subsumer class in each of the tests. The results are compared the DeViSE [FCS+13] retrained using the same 300 training classes and tested with the same subsumer sets. The results are shown in Table 2. To demonstrate zero shot learning, we pick a random set of 30 totally unseen classes from ILSVRC 2012 dataset. We identify the sibling properties of these classes, as explained in Section 2.3.2, with respect to the 300 classes used during training. It turned out that 14 out of the 30 zero-shot classes were sibling classes and the rest were Non-sibling classes.
We use the same DeViSE architecture from Section 3.2 for comparison of results. The tests were carried out under two settings of RC (from Figure 4), where results were taken with and without RC containing the training class embeddings. The classification is considered as correct if the correct label is present among the k outputs in each Hit@k task. The results are shown in Table 3.
DeViSE (Only zero-shot class labels) DeViSE (Zero shot + training class labels) Proposed (Only zero-shot class labels) Proposed (Zero shot + training class labels)
Both models demonstrate capabilities of zero-shot learning, with higher accuracies when RC does not contain the training class embeddings. The reason for this is that, the models map the zero-shot image embeddings more towards the known class labels when they are present in the system. One takeaway from the results in Table 3 is that, the sibling classes are more accurately classified in both models when compared with non-sibling classes. The intuition behind this being the model’s awareness of similar classes during training to the zero-shot classes (effect of having common subsumers). Our proposed model outperforms DeViSE in all instances in the sibling category. Another interesting insight is how our proposed model performs worse with non-sibling classes in several instances compared to DeViSE. This can be seen as a result of the structure of knowledge enforced on our embeddings. Next we extended the zero-shot evaluation to include TAME introduced in Section 2.3.1.
We take the same subsets of the Sibling and the Non-Sibling classes of the zero-shot set in Table 3 and include 1-step, 2-step and 3-step subsumers of those classes to obtain the Hit@k classification accuracy. This evaluates the generalisation abilities of our architecture on the totally unseen zero-shot classes. With the results presented in Table 4, again we first see the better overall performance of the sibling classes of both models, going in-line with the results of Table 3. An observation to note is the decrease in accuracies seen with both approaches in several Hit@k instances with increasing step. The reason for this is that, as the subsumer steps increase, more concept representations are introduced to RC for the systems to choose from. Hence it increases the possibility of the models choosing wrong concepts for the zero-shot classes. 4
Image understanding is often challenged by the accurate semantic understanding of concepts [ZLT17, MMBG04] according to human-level knowledge. As Wengang Zhou et al. [ZLT17] points out, translation of human-level semantic knowledge to low-level visual feature representations is a crucial hurdle to overcome which is referred as the semantic gap [ZLT17]. Even though techniques of machine learning and deep learning have been applied to solving this problem [LZLM07], accurate definition of human-level knowledge is lacking in the existing systems.
For the goal of bridging the semantic gap we turn to the many approaches presented in the area of Zero-shot learning [PPHM09]. These are known for incorporating attributes to represent concepts in vector spaces and mapping visual features to them, giving more meaning to features extracted from an image [FEHF09, LNH09]. Many recent deep learning inspired techniques of zero-shot learning make use of semantic embeddings learnt from a language base in an unsupervised setting, which are then used to guide visual feature representations [SGMN13, RPT15, FS16, ZS15, ZS16]. Our study was largely inspired by the findings of Andrea Frome et al. [FCS+13], where word embeddings learnt from unstructured text [MCCD13] were employed for zero-shot classification of ILSVRC 2012 1K dataset [RDS+15]. Our approach differs when we replace the word embeddings with concept representations captured via a structure knowledge base (such as a knowledge graph) to inform the visual-semantic embedding model.
Xiaolong Wang et al. [WYG18] recently explored the idea of incorporating relations from knowledge graphs into zeroshot learning with the use of Graph Convolutions Networks (GCN) [KW16]. They used the relation information from the edges in the graph to enrich the word embeddings in the process of inferring visual features of unseen images. Our study differs from this approach, where we completely replace the word embeddings with embeddings produced from a structured knowledge base that encompasses the relations between the entities concerned. 5
We propose a technique to incorporate concept representations obtained from a structured knowledge base to inform a visual-semantic embedding model. We show how these embeddings are calculated using the graph structure of the knowledge base and also how they are used to train a projection layer at the top of a traditional image classification model. Informed by the semantics, the image classification model is able to attain higher generalisation with enhance classification accuracies in the test sets compared with traditional methods. We introduce two novel evaluation frameworks that demonstrates how to assess the classification errors with respect to the structure of the knowledge base used. Our approach shows promising results in TAME, where the ability of our approach to identify more general classes of the ground-truth classes was evaluated according to the class hierarchy.
We demonstrate zero-shot classification and evaluate its performance with the novel addition of notions of sibling and non-sibling classes. The results show how sibling classes perform better in a zero-shot classification setting and how the structure of the external knowledge play a part in the classification.
Overall with the results, we identify how concept representations extracted from structured knowledge are more effective than ones from unstructured text when informing a visual-semantic embedding model.
References [DDS+09] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. [FCS+13]
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Tomas Mikolov, et al. Devise: A deep visual-semantic embedding model. In Advances in neural information processing systems, pages 2121–2129, 2013.
Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. Describing objects by their attributes. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 1778–1785. IEEE, 2009.
Yanwei Fu and Leonid Sigal. Semi-supervised vocabulary-informed learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5337–5346, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, 2012.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016. [PPHM09] [RDS+15]
Mark Palatucci, Dean Pomerleau, Geoffrey E Hinton, and Tom M Mitchell. Zero-shot learning with semantic output codes. In Advances in neural information processing systems, pages 1410–1418, 2009.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211–252, 2015.
Bernardino Romera-Paredes and Philip Torr. An embarrassingly simple approach to zero-shot learning. In International Conference on Machine Learning, pages 2152–2161, 2015.
Chakaveh Saedi, Anto´nio Branco, Joa˜o Anto´nio Rodrigues, and Joa˜o Silva. Wordnet embeddings. In Proceedings of The Third Workshop on Representation Learning for NLP, pages 122–131, 2018. [SWM17]
Wojciech Samek, Thomas Wiegand, and Klaus-Robert Mu¨ller. Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models. arXiv preprint arXiv:1708.08296, 2017. [WYG18] [ZLT17]
Ziming Zhang and Venkatesh Saligrama. Zero-shot learning via semantic similarity embedding. In Proceedings of the IEEE international conference on computer vision, pages 4166–4174, 2015.
Ziming Zhang and Venkatesh Saligrama. Zero-shot learning via joint latent similarity embedding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6034–6042, 2016.