This paper summarizes the participation of Imcube at the Retrieving Diverse Social Images Task of MediaEval 2015. This task addresses the problem of result diversification in the context of social photo retrieval where the results of a query should contain relevant but diverse items. Therefore, we propose a multi-modal approach for filtering and re-ranking in order to improve the relevancy and diversity of the returned list of ranked images.
The Retrieving Diverse Social Images Task of MediaEval 2015 [
A photo is considered relevant if it is a common representation of the overall query concept in good visual quality (sharpness, contrast, colours) and without people as main subjects except for queries dealing with people as part of the topic. The results are considered diverse if they depict different visual aspects (time, location, view, style, etc) of the target concept with a certain degree of complementarity.
The refinement and diversification process can be based on the social metadata associated with the collected photos in the data set and/ or on the visual characteristics of the images. Furthermore, the task provides information about user annotation credibility as an automation estimation of the quality of a particular user’s tag.
In this section, we present our approach that combines textual, visual and credibility information to filter and re-rank the initial results. Our approach consists of two steps – relevancy improvement and diversification – as depicted in Figure 1.
The goal of the first step is to improve the relevancy of the ranked image list by re-ranking the images based on more reliable textual and visual criteria and filtering them in order to remove images which are irrelevant for the given application scenario. The goal of the second step is to improve the diversity of the ranked image list through textual filtering and visual clustering and re-ranking. The individual modules will be described in the following sections.
This step exploits additional information extracted from the corresponding Wikipedia article that is provided together with the query. Diversification
Text
Visual Fusion Fusion
Credibility
Visual
To improve the ranking of the images the query is expanded with the most frequent words from the Wikipedia article and the images are re-ranked using a bag-of-words representation. The relevancy is further improved by removing images that do not match the original query and the location information from the Wikipedia article. The location information is extracted by analysing the original query or Wikipedia title (e.g. , “Great Sphinx of Giza”) taking into account typical prepositions for locations (e.g. “in”, “at”, “on”, “of”, “de”). In the case that no location information can be extracted (e.g. “Niagara Falls”) toponyms are not considered for relevancy filtering. 2.2
Visual information is also used to improve the relevancy by reranking them according to different criteria. For each visual feature a ranked image list is derived based on the computed relevancy scores.
Since images with persons as main subjects are considered
irrelevant, we employ a face detector [
Additionally, photos taken from the target location but not
displaying it are considered irrelevant. We model that relevancy by
computing the visual similarity between the retrieved images and
the images which are available from the associated Wikipedia
article. We use histogram of oriented gradients (HOG) features and a
clusterless BoW approach [
We further incorporate aesthetic aspects to emphasize more
visually appealing images, since less blurry and salient images are
usually considered more relevant. The sharpness is calculated as
the ratio of magnitude of image gradients between different blurred
versions of the original image. The larger that ratio, the more
relevant the image. Saliency is measured using a spectral residual
approach [
This step is intended to be the baseline approach for improving
the relevance. It re-ranks the image list according to the
credibility of the owner of an image. The re-ranking is based on 3 scores
which describe the user credibility [
The final image list should not only contain relevant images but also diverse ones, i.e. , depicting different aspects of the topic. With the assumption that images which have an identical textual description often depict very similar content, the images are clustered based on their textual similarity. The ranked image list is then obtained by ranking the clusters in descending order according to their relevancy and iteratively selecting the most relevant image from each cluster. 2.5
The visual diversification considers multiple visual
characteristics including colour (ColorMoment), structure (HOG, clusterless
BoW approach [
The number of clusters is estimated by Hartigan’s Leader
clustering algorithm [
The following experiments were performed based on the system and the individual modules described above following the guidelines of the task.
Run1 is an approach using visual information only (as described in Sec. 2.2 and 2.5). Run2 is an approach based on purely textual information (as described in Sec. 2.1 and 2.4). Run3 is an approach based on textual and visual information (as described in Sec. 2.1, 2.2 and Sec. 2.5). Run5 is an approach using credibility based relevancy and visual diversity (as described in Sec. 2.3 and 2.5).
These experiments are performed on the test set provided which contains 69 one-concept location queries and 70 multi-concept queries related to events.
Table 1 shows the results on the test set for all the runs defined above. Since we want to evaluate our filters for different conditions, scores for the one-concept and multi-concept queries are also provided.
In general, the textual run (run2) achieves the best results. It achieves the higher precision (P @20 = 0:748) and also a slightly better recall (CR@20 = 0:418) compared to the visual run (run1). The advantage is more significant for one-concept queries than for multi-concept queries. The textual run fails for queries which main topic is not correlated to a location (e.g. “chinese new year in Beijing” (its main topic is firework), “paragliding in the mountains” or “tropical rain”). For these cases, the visual run reaches considerably higher F1 scores. Generally, the purely visual run achieves a better recall (CR@20 = 0:4209) and thus a slightly better F1 metric (F 1@20 = 0:5027) for multi-concept queries.
Since, the combination of visual and textual features (run3) constantly achieves lower scores than the individual modalities, we analyse the cases where improvements were made. For example the previously mentioned query (“chinese new year in Beijing”) benefits from visual information with a considerable increase of the F1 measure ( F 1@20 = 0:18). In comparison to run2, run3 achieves a lower precision ( P @20 = 0:015) and a similar recall ( CR@20 = 0:001) leading to slightly lower F1 score ( F 1@20 = 0:006). However, it is interesting to note that the results differ for one-concept and multi-concept queries. The recall of run3 is actually higher than that of run2 for multi-concept queries ( CR@20 = 0:008) while it is lower for one-concept queries ( CR@20 = 0:009). 4.
The results of the different runs show that overall the best results can be achieved with textual information only and that the fusion of visual and textual information leads to slightly worse results. Analysing the results in more detail shows that visual information provides better results for multi-concept queries and queries where the main topic is not correlated to a location while textual information achieves better performance for one-concept queries. This shows that a more advanced fusion approach for combining textual and visual information may improve the results further.