This paper provides an overview of the Verifying Multimedia Use task that takes places as part of the 2016 MediaEval Benchmark. The task motivates the development of automated techniques for detecting manipulated and misleading use of web multimedia content. Splicing, tampering and reposting videos and images are examples of manipulation that are part of the task de nition. For the 2016 edition of the task, a corpus of images/videos and their associated posts is made available, together with labels indicating the appearance of misuse (fake) or not (real) in each case as well as some useful post metadata.
Social media, such as Twitter and Facebook, as means of
news sharing is very popular and also very often used by
government or politicians to reach the public. The speed of news
spreading on such platforms often leads to the appearance
of large amounts of misleading multimedia content. Given
the need for automated real-time veri cation of this content,
several techniques have been presented by researchers. For
instance, previous work focused on the classi cation between
fake and real tweets spread during Hurricane Sandy [
Main task. The de nition of the main task is the following: \Given a social media post, comprising a text component, an associated piece of visual content (image/video) and a set of metadata originating from the social media platform, the task requires participants to return a decision (fake, real or unknown) on whether the information presented by this post (a) (b) (c) Figure 1: Examples of misleading (fake) image use: (a) reposting of real photo claiming to show two Vietnamese siblings at Nepal 2015 earthquake; (b) reposting of artwork as a photo of solar eclipse (March 2015); (c) spliced sharks on a photo during Hurricane Sandy in 2012. su ciently re ects the reality." In practice, participants receive a list of posts that are associated with images and are required to automatically predict, for each post, whether it is trustworthy or deceptive (real or fake respectively). In addition to fully automated approaches, the task also considers human-assisted approaches provided that they are practical (i.e., fast enough) in real-world settings. The following definitions should be also taken into account:
A post is considered fake when it shares multimedia content that does not represent the event that it refers to. Figure 1 presents examples of such content.
A post is considered real when it shares multimedia that legitimately represents the event it refers to.
A post that shares multimedia content that does not represent the event it refers to but reports the false information or refers to it with a sense of humour is neither considered fake nor real (and hence not included in the task dataset).
Sub-task. This version of the task addresses the problem of detecting digitally manipulated (tampered) images. The de nition of the task is the following: \Given an image, the task requires participants to return a decision (tampered, non-tampered or unknown) on whether the image has been digitally modi ed or not". In practice, participants receive a list of images and are required to predict if this image is tampered or not, using multimedia forensic analysis. It should also be noted that an image is considered tampered when it is digitally altered.
In both cases, the task also asks participants to optionally return an explanation (which can be a text string, or URLs pointing to evidence) that supports the veri cation decision. The explanation is not used for quantitative evaluation, but rather for gaining qualitative insights into the results.
Development dataset (devset): This is provided together
with ground truth and is used by participants to develop
their approach. For the main task, it contains posts related
to the 17 events of Table 1, comprising in total 193 cases
of real and 220 cases of misused images/videos, associated
with 6,225 real and 9,596 fake posts posted by 5,895 and
9,216 unique users respectively. This data is the union of
last year's devset and testset [
The data for both datasets are publicly available1.
Similar to the 2015 edition of the task, the posts were collected
around a number of known events or news stories and
contain fake and real multimedia content manually veri ed by
cross-checking online sources (articles and blogs). Having
de ned a set of keywords K for each testset event, we
collected a set of posts P (using Twitter API and speci c
keywords) and a set of unique fake and real pictures around
these events, resulting in the fake and real image sets IF , IR
respectively. We then used the image sets as seeds to
create our reference veri cation corpus PC P , which includes
only those posts that contain at least one image of the
prede ned sets IF , IR. However, in order not to restrict the
posts to the ones pointing to the exact image, we employed
a scalable visual near-duplicate search strategy [
To further extend the testset, we carried out a crowdsourcing campaign using the microWorkers platform2. We asked each worker to provide three cases of manipulated multimedia content that they found on the web. Furthermore, they had to provide a link with information and description on each case, along with online resources containing evidence of its misleading nature. We also asked them to provide the original content if available. To avoid cheating, they had to provide a manual description of the manipulation. We also tested the task in two pilot studies to be sure that the 1https://github.com/MKLab-ITI/image-verification-corpus/ tree/master/mediaeval2016 2https://microworkers.com/ information we got would also be useful. Overall, the data collected was very useful. We performed 75 tasks and each worker earned 2; 75$ per task.
For every item of the datasets, we extracted and made
available three types of features, similar to the ones we made
available for the 2015 edition of the task: (i) features
extracted from the post itself, i.e., the number of words,
hashtags, mentions, etc. in the post's text [
Overall, the main task is interested in the accuracy with which an automatic method can distinguish between use of multimedia in posts in ways that faithfully re ect reality versus ways that spread false impressions. Hence, given a set of labelled instances (post + image + label) and a set of predicted labels (included in the submitted runs) for these instances, the classic IR measures (i.e., Precision P , Recall R, and F -score) are used to quantify the classi cation performance, where the target class is the class of fake tweets. Since the two classes (fake/real) are represented in a relatively balanced way in the testset, the classic IR measures are good proxies of the classi er accuracy. Note that task participants are allowed to classify a tweet as unknown. Obviously, in case a system produces many unknown outputs, it is likely that its precision will bene t, assuming that the selection of unknown is done wisely, i.e. successfully avoiding erroneous classi cations. However, the recall of such a system will su er in case the tweets that are labelled as unknown turn out to be fake (the target class). Similarly, in the sub-task case, given the instances of (image + label), we use the same IR measures to quantify the performance of the approach, where the target class is tampered.
This work is supported by the REVEAL and InVID projects, partially funded by the European Commission (FP7-610928 and H2020-687786 respectively).