In this paper we present our approach to the Crowd Sourcing Task of the MediaEval 2013 Benchmark [2] using transfer learning and visual features. For the visual features we adopt an existing approach for search based classification using content based image retrieval on global features with feature selection and feature combination to boost the performance. Our approach gives a baseline evaluation indicating the usefulness of global visual features, hashing and search-based classification.
The benchmarking task at hand [
We define the framing of an image as the sum of the
visible reflexes of the specific decisions that the photographer
makes when the image is captured. The photographer has
many different choices when taking a photo of a certain
object, event, person or scene. During the capture process
the photographer does not click the shutter randomly, but
rather makes use, either consciously or unconsciously, of a
set of conventions that can be thought of as a recipe for a
certain kind of image. The recipe leads to a distinguishable
framing that is used by the viewer in interpreting the
image. For example, a picture of a person framed in one way
is most easily interpreted as a fashion image and framed
in another way most easily interpreted as a holiday
memory. Choices photographers make to achieve certain types
of framing include color distribution, lighting, positions of
objects and people etc. They also include the choice of the
exact moment during ongoing action at which the image is
shot. In this way, the photographer also influences exactly
what is depicted in the image, e.g., facial expressions of the
people appearing in the image. Especially for fashion use
cases the framing theory is applicable. Due to the nature of
framing we employ global visual features using and
modifying the LIRE framework [
Using LIRE we extracted the global features CEDD, FCTH,
JCD, PHOG, EH, CL, Gabor, Tamura, LL, OH, JPEGCoeff
and SC (which are described and referenced in [
The task includes one required condition, which only
allows the use of the workers’ annotations. However, it is
noted that those annotations are error prone. Therefore,
we integrated a reliability measure for workers based on the
work of Ipeirotis et al. [
We compare a rating rij ∈ R from worker wi ∈ W for image xj ∈ I with R being the set of ratings, W being the set of workers and I being the set of images, to the majority votes V (xj) of all workers for an image. This gives a measure of reliability Q(wi) of a specific worker wi. The computed weight Q(wi) is then multiplied with the vote rij of the worker wi for the image xj.
V (xj) = arg max | {rij : wi ∈ W ∧ rij = v} |
v∈{0,1} Q(wi) = | {rij : xj ∈ I ∧ rij = V (xj)} |
| {rij : xj ∈ I} | Additionally, the familiarity of the worker with the fashion topic is also added as a feature. So the feature vector for an image ik with ratings of three workers w1, w2, w3 is (r1k · Q(w1), r2k · Q(w2), r3k · Q(w3), fw1, fw2, fw3). 3.
We submitted five different evaluation runs. For the
crowdsourcing task, a two part data set was available. The first
part (MMSys dataset) is described in [
The first evaluation run – the required one, run #1 – made use of the feature vector of worker annotations, mentioned above, and the Weka Random Forest Classifier, which yielded good results in cross validation on DM . Using a model built from the DM data set we labeled the images from DF and retrained our model using the newly labeled images.
For the visual content classification (run #2) we used DM
to build a model for classification. The classifier is search
based, which means that the image being classified is
considered as query and the label is derived from the result list. A
similar approach was used in [
Classification performance in terms of time and scaling is promising. In the worst case with 12 features combined, classification per image takes about 240 ms. In the best case – if only one feature is used – classification time is down to 16 ms per image.
Run #3 uses the same techniques as described for run #2, but uses the worker annotations of DF for training the model. Run #4 uses the images labeled in run #1 for training, and run #5 combines run #1 and run #4 in a way, that classification based on visual features is used when the random forest classifier returns an uncertain result.
To estimate the performance of each run we used the test data set and DM experts votes for ground truth. We split the dataset 80% for training and 20% for test. The results of these tests can be seen in Table 1 for both labels (L1, L2). L1 stands for whether an image is fashion related or not and L2 stands for whether the content of the image matches with the category for the fashion item depicted in the image. For the evaluation we used weighted F1 score (WF1), because the positive and negative classes are not comparable on size. The results of the Benchmark can be seen in Table 2. The tests results show that the crowd sourcing classifier has the best performance. Also the official results support this fact. The outcome for the workers information based runs in the final results compared to our test results indicates that transfer learning worked well.
Visual features based on classification performs much better in our tests than in the final results (cp. runs #2#4). It’s common that metadata, even when generated by crowdsourcing, leads to better results, but still the performance drop between preliminary and official results is obvious. However, WF1 scores are more suitable for a steady judgment as shown in Table 1 (e.g. run #3, F1 vs. WF1 scores in the preliminary runs).
Nevertheless, taking all constraints into account, the visual features perform quite well. Their benefit is that, unlike crowdsourcing, which costs money, the effort to extract them and get a small amount of training data is minimal. Moreover, metadata quality depends on the actual workers and the quality control mechanism of the crowdsourcing platform. This is also indicated by the lower WF1 measure of run #3 compared to run #2, as in run #2 expert votings were used to train the model, while run #3 also takes crowdsourcing workers into account for training.
Another interesting effect is that a combination of crowdsourcing metadata and visual content can improve the performance, if it is used to build the model of the visual classifier. In the other direction, it seems to lower performance. Visual information based models have already worked well with small number of training data and a small amount of crowdsourcing can help to boost visual information retrieval systems performance.
We further assume that our theory of framing is supported by the results. Especially our test results, because Label 1 is very good detectable by our global features classifier. On the other side, the Label 1 detection was not good. This is logical, because for the task of object detection local features are better suitable.
For future work it will be interesting to take a closer look on the relationship between crowdsourcing and how it could be used to improve the performance of visual features and vice versa. Another interesting direction would be to use crowdsourcing to create a specific dataset for framing. This would help to draw a clearer definition and show the usefulness of the framing theory.