In this paper, we describe the LIA system proposed for the MediaEval 2013 Crowdsourcing for Social Multimedia task. The aim is to associate an accurate label to an image among multiple noisy labels collected from a crowdsourcing platform. In particular, the task participants have to predict two types of binary labels for each considered image. The rst one mentions that an image is truly fashion-related or not, while the second label indicates that the fashion tag assigned to the image is related to the content of the image or not. The proposed system combines noisy crowdsourcing labels, image metadata and external resources.
Since the advent of Web 2.0 [6], Internet users actively participate in information construction and propagation. Thus, many companies rely on users to give their opinion on movies or on musics, to annotate speci c data... In June 2006, Je Howe [1] de ned for the rst time this new behavior with the term of crowdsourcing.
Thus, crowdsourcing makes it possible to rapidly and cheaply collect supervised labeled data. Nonetheless, annotation quality is uneven since it depends of the annotator, its implication, or its expertise level. As part of the collected data can be very noisy and inaccurate, solutions should be proposed to evaluate the relevance of the labeled data with a minimum time cost.
For these reasons, the Crowdsourcing task becomes a more and more popular and helpful task [3, 2]. In this paper, we describe the LIA system presented at the MediaEval 2013 Crowdsourcing task. The proposed system uses di erent parts of the image metadata: the annotator contribution (con dence1, label), the image context, its geographic coordinates, and text descriptors (title, tags). All this described image metadata will be used in our system to decide if an image is truly fashion-related or not (Label1) while the relevance of the fashion category (Label2) will be chosen using the annotator content only.
The proposed approach focuses on the textual part of the images such as the image metadata, its geo-localization and 1Self-reported con dence of the image category. the metadata of the images occurring in the same context. Five systems (i.e., runs) are submitted. Each run is divided into two subtasks. The rst one is to evaluate a score x~i (with fi = 1; : : : ; 5g) for each label j (j = 1 for the label Yes and j = 0 for No) and then to select the label li with the highest score x~i as: li =
Y es if x~i0 x~i1
N o otherwise
The crowdsourcing annotation contributed by the
general crowd is used to decide if an image is
fashionrelated or not, and if it has been correctly tagged. The
best label x~1 knowing a set of workers W for both
questions is estimated in a similar way with a Naive Bayes
method. This Yes/No classi cation is expressed by:
(
x / P (Ywj jXj)P (Xj) = Y P (Ywj jXj )P (Xj )
w2W where we assume that labels of each worker are conditionally independent. Ywj corresponds to one label of the worker w; Xj also corresponds to one label and x is the true label. Thus, each image is labeled with the label l1 as explained in the previous section.
Each image is potentially related to other images: they have been grouped into sets or pools c according to annotators whether they were annotated by the same person. The annotator label of these images is used to estimate a score of the image context x~j2 for fashionrelatedness: x~j2 = x~1 j c d X X P (Xj jdik) i=1 k=0 where dik is the kth image of the context (set or pool) i, Xj is a label (Yes or No), and x~j1 the crowdsourcing annotation score.
An image could be localized close to one or several other ones. Here, a geo-localization score x~j3 is de ned. This score is calculated for the images that have a distance of zero to the current image. Furthermore, only images that have a label probability Xj greater than 23 (estimated on the development set) are considered close and relevant for the fashion-relatedness question: x~j3 = x~1 j where di is an image from the set s of images with a distance equals to 0, and x~j1 the crowdsourcing annotation score.
Each image comes with metadata (title and tags2) and
a search rank on the hosting service. The rst step is
to collect a set of relevant pictures that are
fashionrelated. The Flickr API is used to compose a new set
of 4; 328 images that responds to the query fashion.
The metadata are extracted from this set of images.
The second step computes the probability score of each
word based on the frequency of the word in the
metadata. Thus, a metadata score x~j4 is calculated for an
image d by using the probabilistic model and the rank
r of the image for the question 1 (fashion or not):
x~j4 = x~j1 +
(
X P (wijd) r i=1 where P (wijd) is the probability of a word wi from the image metadata of d knowing the model m of term frequencies. An image is labeled with the label l4 knowing the annotator score x~j1 and the term frequency model.
A nal run using a combination of the 4 scores x~ij
described above is submitted. A label is assigned to a
picture if this label responds to the di erent aspects of
the image metadata:
(
Run Id 1 2 3 4 5
Crowdsourcing Annotation Context Geo-Localization External Resources Combination F1 Label1 0.7239 0.7171 0.7236 0.7176 0.7183 4.
In this paper, an automatic image classi cation system
has been proposed. This system combines di erent aspects
of the metadata content. The main observation is that best
results are obtained with the crowdsourcing annotation only
(Label1). No gains have been observed using metadata on
the test set, but e orts on new approaches and metadata
selection should be continued to improve classi cation
performance. Finally, the use of the image metadata will be
explored for the fashion tag task (Label2).
x~j5 = x~1
j
xj2
xj3
xj4
(
This score allows to associate an image with the general label l5. We can notice that the score xij is: xij = with i 6= 1 3.
The proposed system is evaluated in the MediaEval 2013 benchmark [4]. In this task, we use the fashion social dataset [5]. Figure 3 presents 3 images extracted from the train dataset with di erent combinations of label mode-classi cation: (a) is not fashion-related (Culottes); (b) is not fashionrelated but well categorized (Androgyny) and (c) is fashionrelated and well categorized (Cowboy hat).
Table 1 presents results obtained in fashion-relatedness classi cation of images (Label1) in terms of F-measure. We can see that the best results are obtained using the crowdsourcing annotation only. Although slight gains were observed during the development phase of our systems, we have to note that the use of metadata information sources in addition to the crowdsourcing annotation do not improve the classi cation performance on the test set in comparison to 2Note that experiments on the development set showed that description or personal notes do not improve the results.
This work was funded by the SUMACC project supported by the French National Research Agency (ANR) under contract ANR-10-CORD-007.