The Verifying Multimedia Use task aims to detect misuse of online multimedia content and verify them as real or fake. This is a highly challenging problem because of strong variations among tweets from di erent events. Traditional approaches train the classi er at message level, which ignores inter-message relations. We propose a two-level classi cation model to exploit the information that tweets of a same topic are probably have same credibility values. In this model a topic level is introduced to eliminate message variations. Messages are aggregated into topics as a higher level representation. Pre-results gained from classi cation at the topic level are then fused with original message level features to train a better classi er. Results indicate that topic level is very helpful and our two-level approach o ers signi cantly better results than a traditional one-level method. Our best result on this task achieves an F-score of 0.94 using features extracted only from tweet content.
The paper presents the approach developed by MCG-ICT
for the MediaEval 2015 Veri cation Multimedia Use task.
The task deals with the automatic detection of
manipulation and misuse of Web multimedia content. Online content
veri cation is a fairly new problem, participants are
encouraged to propose e ective features and methods. The goal
of the task is to evaluate a set of tweets from several events
and identify them as real or fake. More details about the
task can be found in [
Traditional approaches formulate the veri cation problem
as a two-class classi cation task [
Rather than classifying each tweet individually, some
recent studies propose to verify tweets as a whole with
intertweets information. Gupta et al. [
Our network model is designed to evaluate the credibility
of a speci c event. However, in the presented data set of
the target task [
As illustrated in Figure 1, the proposed model has two levels of classi cations: One is the message level which is just the same as previous message level methods. Features extracted from tweets text content, user information and other aspects are used for training; the other is the topic level which is the main contribution of this paper. By assuming tweets under a same topic probably have similar credibility values, we cluster tweets into di erent topics. A topic is a speci c subject in an event, it consists of all tweets concerning the same subject. Compared with raw tweets, topics eliminate variations of tweets by taking the average of them. Thus, it also reduces the impact of noisy data. Compared with event, topics maintain most of tweet details.
Topics Clustering: In [
Topics Labeling: We label each topic as the average labels of its tweets: if more than a half of tweets in a topic is real then we label this topic as real. The labels are used for training the topic level classi er. (In fact, with the proposed topic formation, almost all tweets in a topic have the same label.)
Topic Level Feature Aggregation: We take the average of message level features of all tweets in a topic as the topic level feature. Some nominal features, such as "contains question/exclaimation mark", are also aggregated into corresponding numeric features.
Fusing Topic Level Result: After topic level classication, we can get a probability value for each topic on predicting how likely it is fake. Then for each tweet in the topic, we add this pre-result value as a feature to its original feature vector. Finally, we train a message level classi er with extended message features and give the nal result. 1.2
In [
We extract the commonly used term frequency (tf) features and tf-idf features to represent each tweet. With experiments on the training set (development set), this kind of feature was found to be very over- tting. It reached very high performance on cross validation and very low performance on event-separation validation. Because few words co-occur in di erent events, we assume other pure termbased features (e.g. LDA features) would contribute little on this task.
Almost each tweet contains an image in the dataset, so we extract several features concerning images (e.g. image popularity, resolution). These image features can replace the topic level features to train classi er at topic level, because a topic is generated for each image as mentioned earlier. Experiments on the development set show that these image features result in slightly worse performance for the topic level classi cation than content features but much worse performance after fusing with message level features to generate the nal result. Moreover, these image features cannot be applied directly on videos included in the test set. As they are not the main concern of this paper, we leave these features to future research.
In the task requirements de nition, runs 3-5 are experiments with external resources. As our approach focuses on the classi cation method rather than using external materials, we only submitted results for the rst two runs (Table 1). Run 1 uses only content features while run 2 uses both content and user features. Both runs follow our two level classi cation model illustrated in Figure 1. We use J48 decision tree classi er for topic level classi cation and Random Forest classi er for message level classi cation. The topic level classi cation for training set is built by a 10-fold cross validation on it. The reported three evaluation measures in Table 1 are computed with respect to fake tweets.
From the results we can observe that our two level classi cation method achieves very promising results on both runs. Speci cally, it reaches a veri cation F-Score of 0.9423 for run 1 and a slightly worse result 0.9296 for run 2. Moreover, our method achieves high recall performance as well as high precision. This demonstrates the strong distinctive ability of our method for both fake and real tweets. We also notice that the result of run 2 is slightly worse than that of run 1, which indicates the user features may be redundant. In fact, we get a similar result in our experiments on the development set.
In the future, we want to explore other features, such as image forensics features, with our model. This model also need to be tested on a much larger data set or in real-time situations to validate its e ectiveness.
This work was supported by National Nature Science Foundation of China (61172153, 61571424) and the National High Technology Research and Development Program of China (2014AA015202).