We present the Visual Topic Model (VTM)-a model able to generate a topic distribution for an image, without using any text during inference. The model is applied to an image-text matching task at MediaEval 2021. Though results for this specific task are negative (the model works worse than a baseline), we demonstrate that VTM produces meaningful results and can be used in other applications.
We present a novel approach for Visual Topic Modelling (VTM), i.e. assigning to an image a topic distribution, where 2-3 topics are the most probable ones. A topic is represented as a list of words, so an image is labeled with a set of predefined keywords.
VTM is an extension of Contextualized Topic Models (CTM) [
In this paper, we apply VTM for MediaEval 2021 NewsImage
Task 1, i.e. matching news articles with corresponding images [
To train aligned visual and text topic models we use knowledge
distillation [
Our experiments with text to image matching produced negative
results: a solution based on VTM works worse than a baseline,
based on cosine similarity between out-of-the-box text and image
embeddings [
VTM is an extension of CTM [
Thus, to train a model, each input instance has two parts: text embeddings and bag-of-words representation (BoW). Our main contribution is that we replace text embeddings with visual embeddings and demonstrate that they can be used to train a topic model. The ZeroShot CTM model uses the BoW representation only to compute loss, i.e. this information is not needed during inference time. Since we have a training set that consists of aligned text and image pairs we can use the texts to produce the BoW representation and use it to train a model.
To obtain image embeddings we use CLIP—a pretrained model
that produces text and image embeddings in the same space [
This basic procedure, i.e. training image and text models independently, produces similar but not aligned topic models. Topics could be slightly diferent and even similar topics are organized in diferent (random) order. To increase similarity between text and image models we use knowledge distillation. In this approach a student model uses a diferent input than a teacher—e.g. image instead of text—but should produce the same result.
CTM uses a sum of two losses: reconstruction loss and divergence loss. The reconstruction loss ensures that the reconstructed BoW representation is not far from the true one. The divergence loss, measured as KL-divergence between priors and posteriors, ensures a diversity property, that is desired for any topic model: only few words have large probabilities for a given topic and only few topics have high probabilities for a given document.
In the knowledge distillation approach we leave the reconstruction loss intact but replace divergence loss with KL-divergence with regards to the teacher output. The assumption here is that since a teacher model is already trained to be diverse and a student model is trained to mimic the teacher, the student does not need priors. Experiments supported this assumption.
We use knowledge distillation in two versions: joint model and text-teacher. In the joint approach we first train a joint model that takes as an input a concatenation of text and image embeddings, then train two student models for image and text separately. In the second approach, we first train a text model and then train an image model as a student.
We try 60 and 120 topics with both joint and text-teacher approaches. Preliminary experiments showed that the more topics are used the higher is the model performance in text-image matching.
As a baseline, we use raw cosine similarities between CLIP embeddings, without any domain adaptation for the text. We use an implementation provided as a part of Sentence Bert package (https:// www.sbert.net/examples/applications/image-search/README.html). 3
The results are presented in Table 1. As can be seen from the table, the best results are obtained with CLIP embeddings, that are used without any fine-tuning to the training set. They are able to find the correct image in 1225 cases out of 1915 and has a Mean Reciprocal Rank (MRR) of 0.17. The best VTM model finds correct image in 816 cases out of 1915 and yields an MRR of 0.03.
These results to some extent correspond to our previous
experiments, where we showed that topic modelling does not work well
for document linking [
Another interesting observation is that models that use the text model as a teacher for a visual model work better than joint models. This is an unexpected result, since one would expect that a model that has access to full information could serve as a better teacher. It is possible that text bears less noise: a text model uses the same text for contextual and BoW representation, while an image could be completely irrelevant to a corresponding article.
The fact that embeddings and topic modelling work on diferent principles is illustrated in Figure 1, where we reproduce images found by the model for the text about the Anna Semenova trial. CLIP model finds photos of Anna Semenova, probably due to the huge text and image base used to train the embeddings. VTM returns images with a statue of Themis, a personification of Justice, which represent the text topic rather than specific facts. Though according to our results, CLIP embeddings outperform VTM, the ability to illustrate text topic might be a desirable property for some applications, as well as topic interpretability.
Our code is available at https://github.com/lmphcs/media_eval_ vctm.
This work has been partly supported by the European Union’s Horizon 2020 research and innovation programme under grants 770299 (NewsEye) and 825153 (EMBEDDIA).