Journalists are increasingly turning to technology for pre-filtering and automation of the simpler parts of the verification process. We present results from our semi-automated approach to trust and credibility analysis of tweets referencing suspicious images and videos. We use natural language processing to extract evidence from tweets in the form of fake & genuine claims attributed to trusted and untrusted sources. Results for team UoS-ITI in the MediaEval 2015 Verifying Multimedia Use task are reported. Our 'fake' tweet classifier precision scores range from 0.94 to 1.0 (recall 0.43 to 0.72), and our 'real' tweet classifier precision scores range from 0.74 to 0.78 (recall 0.51 to 0.74). Image classification precision scores range from 0.62 to 1.0 (recall 0.04 to 0.23). Our approach can automatically alert journalists in real-time to trustworthy claims verifying or debunking viral images or videos.
Content from social media sites such as Twitter, YouTube,
Facebook and Instagram are becoming an important part of modern
journalism. Of particular importance to real-time breaking news is
amateur on the spot incident reports and eyewitness images and
videos. With breaking news having tight reporting deadlines,
measured in minutes not days, the need to quickly verify suspicious
content is paramount [
Current tools available to journalists can be broadly
categorized as dashboard and in-depth analytic tools. Dashboard
tools display filtered traffic volumes, trending hashtags and maps
of content by topic, author and/or location. In-depth analysis tools
use techniques such as sentiment analysis, social network graph
visualization and topic tracking. These tools help journalists
manage social media content but unverified rumours and fake news
stories on social media are becoming both increasingly common [
In the REVEAL project we are developing ways to automate the simpler verification steps, empowering journalists and helping them to focus on cross-checking tasks that most need human expertise. We are creating a trust and credibility model able to process real-time evidence extracted using a combination of natural language processing, image analysis, social network analysis and semantic analysis. This paper describes our work on text analysis, extracting and processing fake and genuine claims from tweets referencing suspicious images and videos. Our central hypothesis is that the 'wisdom of the crowd' is not really wisdom at all when it comes to verifying suspicious images and videos. Instead it is better to rank evidence from Twitter according to the most trusted and credible sources in a way similar to human journalists. We describe a semi-automated approach, automatically extracting claims about real or fake content and their source attributions and comparing them to a manually created list of trusted sources. A cross-checking step ranks conflicting claims and selects the most trustworthy evidence on which to base a final fake/real decision.
Named Entity Patterns @ (NNP|NN) # (NNP|NN) (NNP|NN) (NNP|NN) (NNP|NN) Attribution Patterns <NE> *{0,3} <IMAGE> ... <NE> *{0,2} <RELEASE> *{0,4} <IMAGE> ... ... <IMAGE> *{0,6} <FROM> *{0,1} <NE> ... <FROM> *{0,1} <NE> ... <IMAGE> *{0,1} <NE> ... <RT> <SEP>{0,1} <NE> Faked Patterns ... *{0,2} <FAKED> ... ... <REAL> ? ... ... <NEGATIVE> *{0,1} <REAL> ...
Genuine Patterns ... <IMAGE> *{0,2} <REAL> ... ... <REAL> *{0,2} <IMAGE> ... ... <IS> *{0,1} <REAL> ... ... <NEGATIVE> *{0,1} <FAKE> ...
Key <NE> = named entity (e.g. trusted source) <RT> = RT variants (e.g. RT, MT) <IMAGE> = image variants(e.g. pic, image, video) <SEP> = separator variants (e.g. : - = ) <FROM> = from variants(e.g. via, from, attributed) <IS> = is | its | thats <REAL> = real variants (e.g. real, genuine) <NEGATIVE> = negative variants (e.g. not, isn't)
Our trust and credibility model is based on a classic natural
language processing pipeline involving tokenization, Parts of
Speech (POS) tagging, named entity recognition and relational
extraction. The innovation in our approach lies with our choice of
regex patterns, which are modelled on how journalists verify fake
and genuine claims by looking at the source attribution for each
claim. This allows us to provide a novel conflict resolution
approach based on ranking claims in order of trustworthiness. We
use the Python NLTK toolkit [
Our semi-automated approach to named entity matching is based on a list of a priori known trusted and untrusted sources. We can either learn an entity list automatically using information theoretic weightings (i.e. TF-IDF) or create a list manually (i.e. using a journalists trusted source list). All news providers have long lists of trusted sources for different regions around the world so this information is readily available. For the MediaEval 2015 Verifying Multimedia Use task we created a list of candidate named entities by first running the regex patterns on the dataset. We then manually checked each entity via Google search (e.g. looking at Twitter profile pages). We removed any named entities which we considered a journalist would not have in a list of trusted or untrusted sources. We kept news organizations, respected journalists and well cited bloggers and experts. Creating these lists took under two hours (570 named entities checked, 60 accepted).
We chose these regex patterns based on the frequency of text
patterns for source attribution, fake and genuine claims in the
MediaEval-2015 devset. Other researchers have published
linguistic patterns used to detect rumours [
We assign a confidence value to each matched pattern based on its source trustworthiness level. Evidence from trusted authors is more trusted than evidence attributed to trusted authors, which is more trusted than other unattributed evidence. In a cross-check step we choose the most trustworthy claims to use for each image URI. If there is evidence for both a fake and genuine claim with an equal confidence we assume it is fake (i.e. any doubt = fake).
The MediaEval 2015 Verifying Multimedia Use task is to
classify tweets about images and videos as real, fake or unknown.
Details of the task datasets, ground truth and evaluation
methodology used can be found in [
Our first fully automated run used the 'faked & genuine' regex patterns applied to each tweet independently without lists of trusted
When it comes to verifying claims about suspicious images and videos our hypothesis is that the 'wisdom of the crowd' is not really wisdom at all and it is better to rank evidence from Twitter in order of the most trusted and credible sources. We have developed a semi-automated trust and credibility model based on this intuition and well known journalistic verification principles.
When applied to classifying tweets in isolation, our approach has a high precision and low recall, making it of limited value. When we cross-check tweets, ranking by trustworthiness and picking only the most trusted claims our approach is much more useful, with a high precision (0.94+) and average recall (0.43+). The ultimate goal of course is to classify images as fake (including use of image in the wrong context) or real not just the tweets that refer to them. Our classifier was able to classify 4-10% of fake images, getting it right 96-100% of the time. For the harder problem of classifying real images our approach was able to classify 19-23% of images, getting it right 62-95% of the time.
In the context of journalistic verification these results are promising. Given enough tweeted claims about an image or video we can rank the most trustworthy and provide a highly accurate classification result. This means that once images and videos, such as eyewitness content, go viral on twitter we will be able to provide a real-time view on their verification status. Our approach does not replace manual verification techniques - someone still needs to actually verify the content - but it can rapidly alert journalists to trustworthy reports of verification and/or debunking. This in turn should speed up the verification cycle and allow the 'time to publish' to be shortened.
This work is part of the research and development in the REVEAL project (grant agreement 610928), supported by the 7th Framework Program of the European Commission. The authors would like to thank journalists at Deutsche Welle for their valuable insights into the journalistic verification process. Conference on Web and Social Media (ICWSM-15). Oxford, UK