This paper describes the proposed solutions of the Texas State University Data Lab team to the MediaEval 2020 FakeNews benchmark. We have responded to the text-based and structural-based tasks with lexical, graph, and community labeling approaches. Our lexical analysis approach using logistic regression produces the best results at 0.43 MCC, we describe a promising community labeling model, and discuss our attempts to find predictive structural features in retweet graphs of conspiracy promoting tweets.
Our analysis assumes that people in the same social network community who agree on fake news also write in a similar style, discuss similar topics, produce similar content, and share similar values. We relate content of the tweets using lexical analysis, employ community discovery by building a network of re-tweets, and employ network analysis on structural data provided.
Analysis of tweet content spans from the use of Bag-Of-Words
features in classification model to capture most likely terms associated
with fake news [
Twitter restricts tweet content to 280 characters, a constraint that tends to influence a writing style that difers from that found in most corpora. To achieve brevity, users employ a lexicon that includes abbreviations, colloquialisms, hashtags, and emoticons. Tweets may also contain frequent misspellings. The context of a tweet is also richer as it resides in a rich network of retweets and replies. To this end, we employ lexical-based analysis and community analysis for tweet content and context.
Lexical Analysis Pipeline implements transformation of
twitter content, feature extraction, and modeling, to make predictions
for the NLP-based task [
Feature extraction in text can be accomplished by encoding terms as vectors representing either the occurrence of terms in text (Bag-Of-Words), or the impact of terms to a document in a corpus (TF-IDF). We employed a TF-IDF vectorizer to reduce the impact of repeated terms within individual tweets, but did not see improvements over the Bag-Of-Words model in our pipeline. We trained a set of classifiers (Naive Bayes, SVC, Random Forest, and Logistic Regression) on our extracted feature vector, and analysed the performance of both the fine-grained four class and coarse-grained two class classification sub-tasks. To account for imbalance in data, we have experimented with data augmentation, generating fake tweets using the most predictive or most common terms for each class. This approach led to overfitting of most classifiers, and we have dropped it early on. We have extended the feature set in the tweets using Optical Character Recognition of images embedded in tweets. We have also adjusted class weights to account for imbalanced data, when possible. Logistic regression showed superior performance in all our test runs, and in our submission we have included runs labeled LR (Logistic Regression) and LR-OCR (Logistic Regression with Optical Character Recognition text augmentation) in Table 1.
work [
Standard summary statistics for graphs are used here as feature vectors. We extract 19-dimensional feature vector from each provided graph adjacency matrix using ’igraph’ R package, and the dimensions represent: number of nodes, number of edges, diameter of the graph, mean distance, edge density, reciprocity, global transitivity, local transitivity, number of triangles, mean in-degree, maximum in-degree, minimum in-degree, mean out-degree, maximum out-degree, minimum in-degree, mean total degree, maximum total degree, and minimum total degree. Feature vectors are normalized and fed into a series of python scikit-learn classifiers: (1) Decision tree with no maximum depth and the Gini impurity or entropy criterion; (2) Linear discriminant analysis (LDA) with SVD, LSQR, and LSQR plus shrinkage, and (3) Naive Bayes. An 80/20 train/test split of the development data was used to train the models. Each classifier was trained for coarse classification and 4 way fine classification. 4
Text-Based Misinformation Detection: for each of the coarsegrained and fine-grained classifications the team has submitted one run of predictions from our community labeling model, two runs from our lexical analysis pipeline, and one run combining the two approaches, for a total of eight sets of predictions. Table 1 summarizes our returned test set results, and our own evaluations on the development set using the provided ground truth labels. Lexical Analysis Pipeline using logistic regression produces the highest MCC for both classification subtasks. Note that OCR text augmentation does not improve the MCC on test set, even as it shown improvements for multi-class subtask for development set.
high MCC on test set, but it does contribute to comparable MCC on test set (0.363), and higher precision, and comparable recall and accuracy on development set. We ran out of time to implement meaningful fusion of lexical and community runs. Internal analysis
Run Model showed that number of tweets that are isolated from the network degrades the performance of community based approach.
Run Model
Structure-Based Misinformation Detection runs are reported in Table 2, where the highest overall MCC on the test set was 0.0327 for LDA based coarse classifiers. There is not enough information to mine in the provided structure data to capture meaningful relations, as some of the runs produce MCC lower than random. Note that proposed community-based network produces higher MCC scores (Table 1) alone as it takes into account hashtags, retweets, and community discovery over larger corpora.
Lexical based analysis produced the highest MCC score on the test
set. Our community discovery method showed some promise in
the fusion approach with increased precision. Community-based
and structure-based methods will likely contribute more if we
consider conspiracy vs non-conspiracy classification, as recent work
has shown diferent dispersion patterns regardless of the
conspiracy topic [