=Paper=
{{Paper
|id=Vol-1263/paper47
|storemode=property
|title=CERTH @ MediaEval 2014 Social Event Detection Task
|pdfUrl=https://ceur-ws.org/Vol-1263/mediaeval2014_submission_47.pdf
|volume=Vol-1263
|dblpUrl=https://dblp.org/rec/conf/mediaeval/RigaPPSK14
}}
==CERTH @ MediaEval 2014 Social Event Detection Task==
https://ceur-ws.org/Vol-1263/mediaeval2014_submission_47.pdf
CERTH @ MediaEval 2014 Social Event Detection Task
Marina Riga, Georgios Petkos, Symeon Papadopoulos, Emmanouil Schinas, Yiannis
Kompatsiaris
Information Technologies Institute / CERTH
6th Km. Charilaou-Thermis
Thessaloniki, Greece
{mriga,gpetkos,papadop,manosetro,ikom}@iti.gr
ABSTRACT are not equally important. More specifically, the average
This paper describes the participation of CERTH in the So- size of an event in the training set is roughly 20 images.
cial Event Detection Task of MediaEval 2014. For Challenge In practice though, the set of candidate neighbours needs
1, we use a “same event model” to construct a graph on which to be quite larger than the average. For instance, in our
we perform community detection to obtain the final cluster- experiments we experimented with at most 500 candidate
ing. Importantly, we tune the model to have a higher true neighbours. The primary reasons for this is that a) the dis-
positive rate than true negative rate, leading to significantly tribution of the sizes of the events is much wider and b) in
improved performance. The F1 score and NMI for our best large datasets one needs to consider a larger number of can-
run are 0.9161 and 0.9818, respectively. For Challenge 2, we didate neighbours in order to have higher confidence that
developed probabilistic language models to classify events the actual neighbours of some image appear in the set of
according to the criteria of the different queries. Our best candidate neighbours. Therefore, since the number of can-
run on Challenge 2 achieved an average F-score of 0.4604. didate neighbours will be much larger than the number of
actual neighbours, and assuming that the classifier has been
1. INTRODUCTION trained to achieve similar true positive and true negative
The paper presents the approaches developed by CERTH rates, we can expect that the SEM will give a significantly
for the two Challenges of the MediaEval 2014 Social Event larger number of false positive predictions than false neg-
Detection (SED) task. Challenge 1 asks for a full clustering ative predictions. Too many false positive predictions are
of a collection of Flickr images, so that each cluster corre- likely to result in a lot of merged clusters as they will cre-
sponds to a social event. Challenge 2 examines a retrieval ate too many incorrect edges in the graph. If on the other
scenario in which, given a set of social events, the goal is to hand we opt for a higher true positive rate at the cost of
determine those events that match particular criteria. More a lower true negative rate (by increasing the classification
details about the task can be found in [3]. threshold), we will have far fewer incorrectly merged clus-
ters, but will also have some fragmented clusters. The way
2. PROPOSED APPROACH to deal with this problem is to increase the set of candidate
2.1 Overview of method in Challenge 1 neighbours. In our experiments, we observed that when in-
Our approach for Challenge 1 utilizes what is termed the creasing the threshold so that the true positive rate is 0.9999,
Same Event Model (SEM)[2]. The SEM takes as input the the true negative rate does not drop below 0.95, which in
set of per modality similarities between two items and pre- practice appears sufficient for our purpose.
dicts how likely it is that these two items belong to the same
event or not. Subsequently, a graph is constructed, in which
2.2 Overview of method in Challenge 2
the nodes represent the images to be clustered and the exis- In Challenge 2, we utilize regularized unigram language
tence of an edge between a pair of nodes denotes the positive models [1] to classify clusters (or images in Run 5, as will
prediction of the SEM for the two respective images. Finally, be explained later) according to the given retrieval criteria
a community detection algorithm is performed on the graph (location, type of event, entities involved). For learning the
to obtain a full clustering. Moreover, in order to limit the language models for the event types and entities of inter-
number of evaluations of the SEM and make the approach est we collected sets of images from Flickr using the rel-
scalable, we deploy a candidate neighbour selection step: evant keywords that appear in the queries. Moreover, we
for each image we utilize appropriate indices in order to ob- retrieved an additional random collection of images, in or-
tain the most similar images according to each modality and der to learn a general language model that does not focus
evaluate the SEM only for them. This is a technique that on any particular event type or entity, against which the
is commonly referred to as blocking. This overall approach type- or entity-specific language models are compared. For
is similar to that of [5] and that which we deployed in last some cluster (or image) i the comparison is performed by
year’s task [6]. Importantly though, we introduce a tweak computing the ratio of the probability given by the specific
which improves performance significantly. The key idea is language model pspecif ic (i) over the probability given by the
that false positive and false negative predictions of the SEM general language model pgeneral (i); if the ratio is above some
threshold θ, then we assign the event (or image) as match-
ing the examined criterion. In a second variation we utilize
Copyright is held by the author/owner(s). a language model that has trained both with the type and
MediaEval 2014 Workshop, October 16-17, 2014, Barcelona, Spain entity specific datasets and the general dataset and com-
Challenge 1 Challenge 2 Average scores Challenge 2, F1 per query
Run F1 NMI Div. Recall Precision F1 1 2 3 4 5 6 7 8 9 10
1 0.4514 0.7594 0.4498 0.6101 0.3458 0.3431 0.6207 0.6588 0.2137 0.2694 0.8193 0.1524 0.4578 0.0868 0.1375 0.0145
2 0.4515 0.7592 0.4498 0.7505 0.2669 0.2723 0.6505 0.6744 0.0338 0.2671 0.5965 0.1214 0.2774 0.0141 0.0748 0.0126
3 0.8312 0.9627 0.8304 0.5556 0.4120 0.4043 0.6505 0.6744 0.0338 0.4568 0.9444 0.2143 0.4211 0.4902 0.1311 0.0266
4 0.9133 0.9808 0.9124 0.3915 0.7080 0.4604 0.6207 0.6588 0.4828 0.2500 0.8947 0.3529 0.6383 0.4324 0.2189 0.0543
5 0.9161 0.9818 0.9152 0.3798 0.3569 0.2806 0.5828 0.5195 0.0406 0.3136 0.9444 0.1405 0.1538 0.0000 0.0874 0.0229
Table 1: Scores achieved in the two Challenges
pute the ratio pspecif ic,general (i)/pgeneral (i). For inferring ified classification threshold show a very clear improvement
location we adopted the per grid-cell language model based over Runs 1 and 2 that do not. Moreover, it appears that ap-
approach of [4]. It should be noted though that for clusters propriately expanding the candidate neighbours (Run 4 over
that contain geotagged images, we do not use the language Run 3) can also provide a significant improvement. Addi-
models, but rather use the explicit coordinates to estimate tionally, there is some further improvement in Run 5, that
the location. does not use blocking, over Run 4, but the improvement is
very small. All in all, it can be said that strong blocking is
3. EXPERIMENTS useful in order to make the application of the method more
3.1 Runs description in Challenge 1 scalable, but can lead to somewhat decreased performance.
In all runs of Challenge 1 we utilized a SVM classifier to 4.2 Challenge 2
learn the SEM. The following features were used to compute Table 1 shows the average scores that we achieved over
the input to the SEM for a pair of images: user (1 if both all 10 queries of Challenge 2. We note that Run 3 and Run
images have been uploaded by the same user, 0 otherwise), 4 give the best average scores meaning that the selected
textual (title, tags and description, similarity computed us- threshold has a significant influence in the accuracy of the
ing BM25 and cosine), taken and upload time, spatial (if classification results. Test queries perform better when a
available) and visual information (SURF descriptors aggre- calibration of the threshold value comes first. The classifi-
gated using a VLAD scheme [8] as well as features extracted cation of an event by handling photos in cluster uniformly
using Overfeat [7], a popular convolutional net, similarity for performs better than having an individual classification re-
both is computed using Euclidean distance). In Run 1 we sult per photo. It should also be mentioned that considering
apply our basic approach, without using any visual features only queries that include location criteria, the performance
and we take the predictions of the SEM as they are, i.e. we is significantly higher. In particular, for those queries, in
do not change the classification threshold. In Run 2 we only Run 4 we achieve an F-score of 0.6331.
add the visual features. In Run 3 we use the probabilities
that are provided by the SVM classifier and set the thresh- 5. ACKNOWLEDGMENTS
old to 0.995, achieving the true positive and true negative The work was supported by the European Commission
rates that were mentioned earlier. In Run 4 we attempt to under contract FP7-287975 SocialSensor.
improve the results by increasing the set of candidate neigh-
bours: after the graph has been constructed by predicting
6. REFERENCES
the SEM output for each image’s candidate neighbours, we [1] D. Jurafsky and J. H. Martin. Speech and Language
add to the candidate neighbours of each image the neigh- Processing. Prentice Hall PTR, Upper Saddle River,
bours of its actual neighbours and predict the output of the NJ, USA, 1st edition, 2000.
SEM for them as well. In Run 5 we do not use blocking and [2] G. Petkos, S. Papadopoulos, and Y. Kompatsiaris.
compute the output of the SEM for all pairs of images. Social event detection using multimodal clustering and
integrating supervisory signals. In Proc. of ICMR 2012.
3.2 Runs description in Challenge 2
[3] G. Petkos, S. Papadopoulos, V. Mezaris, and
In Run 1 of Challenge 2 we perform the classification by Y. Kompatsiaris. Social event detection at MediaEval
computing the ratio pspecif ic (i)/pgeneral (i) and setting the 2014: Challenges, datasets, and evaluation. In
threshold θ to 1. In Run 2, we perform the classification by Proceedings of the MediaEval 2014 Multimedia
computing the ratio pspecif ic,general (i)/pgeneral (i) and again Benchmark Workshop, 2014.
setting the threshold to 1. In Run 3 and Run 4 we use the
[4] A. Popescu. CEA list’s participation at MediaEval 2013
models of Run 2 and Run 1 respectively, but with different
Placing Task. In Proceedings of the MediaEval 2013
threshold values per query. Each threshold is selected ac-
Multimedia Benchmark Workshop, 2013.
cording to the evaluation results of the methodology in the
[5] T. Reuter and P. Cimiano. Event-based classification of
corresponding development queries. For queries Test-9 and
social media streams. In Proceedings of ICMR 2012.
Test-10 where there are no analogous development queries,
we used the maximum threshold from the other queries. In [6] M. Schinas, E. Mantziou, S. Papadopoulos, G. Petkos,
Runs 1 to 4 we perform classification per event, that is, we and Y. Kompatsiaris. CERTH @ MediaEval 2013 Social
aggregate all images of an event and then perform the clas- Event Detection Task. In Proceedings of the MediaEval
sification. In Run 5 on the other hand we perform classifica- 2013 Multimedia Benchmark Workshop, 2013.
tion per item and then perform the aggregation by majority [7] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu,
vote. Also, in Run 5, the same approach in language models R. Fergus, and Y. LeCun. Overfeat: Integrated
and threshold values as in Run 3 has been followed. recognition, localization and detection using
convolutional networks. CoRR, abs/1312.6229, 2013.
4. RESULTS AND DISCUSSION [8] E. Spyromitros-Xioufis, S. Papadopoulos,
4.1 Challenge 1 I. Kompatsiaris, G. Tsoumakas, and I. Vlahavas. An
Table 1 shows the scores we achieved in Challenge 1. The empirical study on the combination of SURF features
main thing to note is that Runs 3, 4 and 5 that use the mod- with VLAD vectors for image search. WIAMIS, 2012.