In this paper, we describe our method and results for the MediaEval 2013 Retrieving Diverse Social Images Task. To accomplish the task objective, we adopt VisualRank [5] and Ranking with Sink Points [2], which are common methods to select representative and diverse photos. To obtain an affinity matrix for both ranking methods, we used only the officially-provided features including visual features and tag features. We submitted three required runs including only visual feature run, only textual feature run and textualvisual fused feature run.
In this paper, we describe our method and results for the
MediaEval 2013 Retrieving Diverse Social Images Task [
To obtain a similarity matrix for both ranking methods, we used only the officially-provided features including visual features and tag features. We submitted three required runs including only visual feature run, only textual feature run and textual-visual fused feature run, which are the minimum requirements to participate this task.
To obtain representative and diverse photos in the
upper rank, we adopt VisualRank [
VisualRank is an image ranking method based on
PageRank [
VisualRank uses a similarity matrix of images instead of hyper-link structure. Eq.(1) represents an equation to compute VisualRank.
ri+1 = αSri + (1 − α)p, (0 ≤ α ≤ 1)
(1)
S is the column-normalized similarity matrix of images, p
is a damping vector, r is the ranking vector each element of
which represents a ranking score of each image, and α plays
a role to control the extent of effect of p. The final value of r
is estimated by updating r iteratively with Eq.(1). Because
S is column-normalized and the sum of elements of p is 1,
the sum of ranking vector r does not change. Although p
is set as a uniform vector in VisualRank as well as normal
PageRank, it is known that p can plays a bias vector which
affects the final value of r [
Because VisualRank is a ranking method considering only
representativeness of items, higher ranks are sometimes
occupied with items which are similar to each other. This is,
VisualRank cannot accomplish ranking considering diversity
of items. Therefore, we adopt Ranking with Sink Points [
To address the diversity in ranking, the concept of sink points is useful. The sink points are data objects whose ranking scores are fixed at zero during the ranking process. Hence, the sink points will never spread any ranking score to their neighbors. Intuitively, we can imagine the sink points as the “black holes” on the ranking manifold, where ranking scores spreading to them will be absorbed and no ranking scores would escape from them.
First we apply VisualRank to select the most representative photo with the obtained affinity matrix. Then we re-rank the remaining photos by Ranking with Sink Points as shown in Eq.(2), after removing the first-ranked photo as “a sink point”. We repeat re-ranking by Ranking with Sink Points and removing the first-ranked photos until 50 photos are selected as following: ri+1 = αSIiri + (1 − α)p (2) Ii is an indicator matrix which is a diagonal matrix with its (i, i) − element equal to 0 if xi ∈ Xs and 1 otherwise. Xs is a set of “sink points”.
Note that α is set as 0.85 in the experiments. 2.3
We used the ten kinds of visual features officially provided by the task organizers such as Global Histogram of Oriented
Gradient and Color Moments on HSV Color Space. The
detail on official visual features is explained in [
With histogram intersection, we calculate similarities for each of visual features. Finally we construct an affinity matrix by averaging similarity on ten kinds of visual features. 2.4
We use social TF-IDF weights provided by the task organizers. We extract bag-of-words vectors from Flickr metadata with social TF-IDF weights for all the given images. We calculate an affinity matrix with cosine similarity between bag-of-words vectors within each place.
To obtain an affinity matrix for the visual-textural-fused runs, we simply averaged both visual-feature-based affinity matrix and textual-feature-based affinity matrix.
Tables 1 and 2 show the evaluated results of our three submission runs by experts and crowds, respectively. Note that the results by experts is based on evaluation for the entire dataset of 346 locations, while the results by the crowds is based on evaluation for only 50 locations in the dataset and are obtained by averaging evaluations by three crowd persons.
Basically, the results by only visual were better than the results by only textual and the results by visual-textual, although the difference were not so large.
We show the top six photos of an successful example by the proposed method with three kinds of features: textual, visual and visual-textual features in Figure 1. These photos represents “The Gate of Forbidden City in Beijing, China.” In this example, the photos selected by the visual-textual feature is more representative and diverse than the photos selected by the only textual or only visual features. This indicates that our proposed methods works successfully.
In the case of the above example, most of the photos included in the given photo set are relevant and only a few noise photos are included. However, given photo sets of some landmark include many noise photos. In such case, the proposed methods sometimes failed to select relevant photos and selected noise photos in the upper ranking. Therefore, removal of noise photos is one of our important future works.
ing dataset, GPS data coordinates and Wikipedia photos. In fact, if you had enough time, we should have used the training data for estimating optimal parameters such as α in the VisualRank formulation and a mixing weight of visual similarity and textual similarity. 5.