In this paper we present the results achieved during the 2017 MediaEval Retrieving Diverse Social Images Task, using an approach based on pseudo-relevance feedback (RF), in which human feedback is replaced by an automatic selection of images. The proposed approach is designed to have in priority the diversification of the results, in contrast to most of the existing techniques that address only the relevance. Diversification is achieved by exploiting a hierarchical clustering (HC) scheme followed by a diversification strategy. Methods are tested on the benchmarking data and results are analyzed. Insights for future work conclude the paper.
An eficient information retrieval system should be able to provide
search results which are in the same time relevant for the query
and cover diferent aspects of it, i.e., diverse. The 2017 Retrieving
Diverse Social Images Task [
Despite the current advances of machine intelligence techniques
used in the area of information retrieval and multimedia, in search
for achieving high performance and adapting to user needs, more
and more research is turning now towards the concept of “human
in the loop” [
Due to good performance achieved in [
In traditional RF techniques, recording actual user feedback is
ineficient in terms of time and human resources. The proposed approach,
denoted in the following HC-RF, attempts to replace user input with
machine generated ground truth. It exploits the concept of
pseudorelevance feedback. The concept is based on the assumption that
top k ranked documents are relevant and the feedback is learned
as in traditional RF under this assumption [
Similarly to [
First, we employ a pseudo-relevance feedback scheme based on
an automatic selection of the images. We consider that the first
returned results are relevant (i.e., positive examples). For instance,
on devset [
HC-RF visual 0.575 step is the actual diversification scheme, which is a round robin approach. We select from each of the relevant classes one image which has the highest rank according to the initial ranking of the system. Then, we remove the selected images from the clusters and proceed by selecting the remaining ones in the same manner. The process is repeated until a maximum number of images is reached. The resulting images represent the output of the proposed system. 3
This section presents the experimental results achieved on devset which consists of 110 multi-topic queries and 32,487 images and testset, respectively, which consists of 84 multi-topic queries and 24,986 images. We optimized the parameters of the proposed approach on devset to obtain best precision and diversity. The final benchmarking is conducted however on testset.
In our approaches, images are represented with the content
descriptors that were provided with the task data, i.e., visual (e.g.,
convolutional neural network based descriptors), text (e.g., term
frequency - inverse document frequency representations of metadata)
and user annotation credibility (e.g. upload frequency)
information. Detailed information about provided content descriptors is
available in [
Several tests were performed with diferent descriptor combinations and various cutof points. Descriptors are combined with an early fusion approach (normalization and concatenation). We varied the number of initial images considered as positive examples (N p) from 100 to 280 with a step of 20 images, the number of last images considered as negative examples (N n) from 0 to 20 with a step of 10, and the inconsistency coeficient threshold for which HC divides the data into well-separated clusters (N c) from 0.5 to 1.3 with a step of 0.2. We select the combinations yielding the highest F 1@20, which is the oficial metric.
While experimenting, we observed that, by increasing the number of analyzed images, precision tends to decrease as the probability of obtaining non-relevant images increases; in the same time, diversity increases as having more images is more likely to get more diverse representations. For brevity reasons, in the following we focus on presenting only the results at a cutof of 20 images which is the oficial cutof point. These results are presented in Table 1. We present also the Flickr initial retrieval results to serve as baseline for the evaluation. From the modality point of view, visual-text information (visual-all textual-all) with the parameter setup (N pN n-N c)=(180-0-1.1) lead to the highest results (F 1@20=0.4773). This metric/run
Run1 0.6333 performance was followed by visual (visual-all without CNN), textual (textual-all), CNN and credibility descriptors (F 1@20=0.4473) all with (180-20-1.1) parameter setup. 3.2
Following the previous experiments, the final runs were determined for the best modality/parameter combinations obtained on devset (see Table 1). We submitted five runs, computed as following: Run1 - automated using visual information only: HC-RF all visual; Run2 - automated using text information only: HC-RF all text; Run3 automated using visual-text information: HC-RF all visual-all text; Run4 - everything allowed: HC-RF CNN.; and Run5 - everything allowed: HC-RF cred. Results are presented in Table 2.
What is interesting to observe is the fact that the highest precision is achieved using visual information, (Run1 - P @20 = 0.6333), whereas maximum diversification is achieved using credibility information (Run5 - CR@20 = 0.6045), with more than 2% over other types of descriptors. Relevance was also preserved, which leads to the conclusion that credibility information was useful in the context of overall diversification. Credibility information estimates the quality of tag-image content relationships, telling which users are most likely to share relevant images in Flickr. Best diversification is achieved in this case due to a high probability that diferent and relevant images belong to diferent users with a good credibility score. In terms of F 1 metric score, the use of credibility information, Run5 - F 1@20 = 0.5777, allows for best performance, followed closely by visual descriptors, Run1 - F 1@20 = 0.5753. Visual-textual information achieved also good performance, Run3 - F 1@20 = 0.5741, followed by textual information, Run2 - F 1@20 = 0.5733. The CNN descriptors had the lowest performance, by more than 5% under the credibility information, Run4 - F 1@20 = 0.5253. 4
We approached the image search result diversification issue from the perspective of relevance feedback techniques, when user feedback is substituted with an automatic pseudo-relevance feedback approach. Results show that in general, the automatic techniques improve the precision and diversification, which proves the real potential of relevance feedback to the diversification. Future developments will mainly address diferent eficient exploitations of re-ranking approaches, e.g., relevance-score estimation techniques, to improve the relevance and consequently the overall diversification. Another perspective is to also exploit the advantages of deep neural networks and use them in the context of automatic relevance-feedback-based diversification scenarios, by classifying the selected positive and negative examples using unsupervised deep-learning-based classifiers.