In recent years, social media platforms have become leading channels for the exchange of knowledge, debates, and product or opinion advertising [PP10, WD07, Gly18, SKB12]. Social scientists routinely use data from social media platforms to survey public opinion on speci c topics [Mos13, CSPR16, HBK+17, BM18] and computer scientists use the data to improve the performance of state-of-the-art natural language processing (NLP) algorithms [CXHW17, ACCF16, GPCR18, ZWWL18].

Social media data, while abundant, pose many challenges in usage: 1) user demographics are rarely available; 2) posts are short and sometimes hard to understand without context, and 3) it is challenging to label millions of posts manually in short time. One may overcome the rst challenge by selecting only information from users where demographic information is available using multiple social platforms. However, this may bias the data. In order to solve the other two problems, we need systems that classify data into di erent opinion classes with limited human involvement. Numerous algorithms have been proposed to cope with large amounts of short text [ZZL15, ZQZ+16, LQH16, XC16, CSBL16, YYD+16, MGB+18]. All these algorithms are supervised in nature, and therefore, require hundreds of thousands of labels in order to achieve adequate performance levels. In the last two years, algorithms such as CoVe [MBXS17], ELMo [PNI+18], ULMFiT [HR18] and OpenAI GPT learning rate, slanted triangular learning rates are used for every layer to improve the accuracy of the model [HR18]. First, the learning rate sharply linearly increases so that the model can learn fast from the rst examples. Once learning rate achieves the L, it slowly linearly declines as shown in the top-right corner of Figure 2.

In the third step (Figure 2c), layers are trained gradually. First, only the top layer is trained with labeled data for one epoch while other layers are frozen. In each new epoch, the next frozen layer from the top is added to the training. 3

Experiments are conducted using Twitter data on USA midterm elections 2018 and news data from USA elections 2016.

Twitter data is collected by searching for posts published between November 4th and 7th 2018 which have one of the hashtags: "#vote", "#trump", "#election", "#midtermelection", "#democrats", "#republicans" and "#2018midterms". In total, we accrue 936,462 tweets. Most of the posts are retweets, which appear multiple times in the corpus. After retweets removal, 244,320 distinct posts remained, and we pre-process their text by removing all characters, except alphanumerics.

Out of those posts, we label 1,526 examples with 0, 1 or 2. Label 0 is assigned to examples that support or promote the left political spectrum or denounce the right point of view. Label 1 is given to politically neutral posts (e.g., posts that encouraged voting). Label 2 is assigned to examples that support or advertise the right political spectrum or condemn the left point of view. We discard 500 examples ( 25% of posts) because they are unrelated to elections.

News data is collected from six outlets that are perceived to have di erent political partisanship, ranging from the left-oriented to right-oriented outlets based on media bias fact check website (Table 1). Articles published between October 2015 and May 2017 that contain words "election", "ballot", "republican", "GOP", or "democrat" are selected. The news articles di er substantially in writing style, content diversity, bias, number of articles and number of words (Table 1). As with the tweets, news articles do not always discuss the U.S. elections. Sometimes, they debate Brexit or elections in France and other countries worldwide. In pre-processing, we remove all nonalphanumeric characters from news articles.

Experiments settings. We use the pre-trained WT103 token-vectors in the rst ULMFiT step. WT103 has 103 million tokens from Wikipedia texts for training, 217K tokens for validation and 245K tokens for testing [MXBS16]. Our system is trained using the architecture in Figure 2a. The vocabulary has 267K unique tokens. In this paper word and token have interchangeable meanings.

For the ne-tuning step, we explore ten di erent settings: 1) "all news" text with the data from all outlets + tweets text; 2) only the tweets; 3) text from "left-biased" outlets + tweets text; 4) text from "right-biased" outlets + tweets text. Remaining six experiments contain text from one outlet and tweets text. We randomly permute examples in a ne-tuning dataset before usage.

In the third step, experiments test two settings of labeled Twitter data. Mix 1 (balanced mix) contains 380 examples with label 0 (left), 323 examples with label 1 (neutral) and 323 examples with label 2 (right). Mix 2 (unbalanced mix) contains 380 examples with label 0 (left), 823 examples with label 1 (neutral) and 323 examples with label 2 (right). We randomly split labeled data into three disjoint parts: test (200 examples), validation (200 examples) and training (626 examples in Mix 1 and 1126 examples in Mix 2). Each experiment is repeated four times and accuracy mean, and the standard deviation is reported for each of the ten settings.

We do not clean Twitter, and news data of nonrelevant examples in order to emulate the real-world situation. The data retrieval process is intentionally simple to mirror the information extraction process often used in research papers [Mos13, CSPR16, HBK+17, BM18]. Those experiments test the robustness of the model to the bias and noise in data and robustness to the unbalanced classes. 4

We repeat each experiment four times, and we report the accuracy mean and standard deviation in Table 2. High standard deviation (1:2 5:3%) indicates the model's sensitivity to the order of examples in the netuning data and a need for more labeled examples.

Results provide evidence that the model is not robust to unbalanced datasets. When Mix 1 and Mix 2 results are compared, the model always achieved better results for Mix 2 (Table 2) which has 54% of neutral labels as compared to 31.5% of neutral labels in Mix 2 (380 : 823 : 323) 59:4 3:7% 66:6 2:5% 61:1 3:3% 63:0 3:2% 62:7 3:0% 60:7 1:4% 64:1 2:7% 64:2 1:8% 60:0 4:3% 61:9 3:3% Mix 1. As evident from Figure 4, 80 90% of predicted labels for Mix 2 are neutral. Therefore, better results for Mix 2 are achieved because the algorithm exaggerates the most frequent (neutral) label in the imbalanced dataset (which contains 54% of examples of that class).

The classi cation accuracy di erence between Mix 1 and 2 is the largest (11.9%) when "left-biased news" is used for ne-tuning. In this case, the accuracy on both Mix 1 and Mix 2 decreases compared to when "No news" is present. However, outlet bias has more in uence on the accuracy of Mix 1.

Figure 3 reveals that using "all news" data for netuning achieves the best balance among predicted labels for Mix 1. However, almost half of predicted labels are wrong, so accuracy is low.

Labeled Twitter data demonstrate diversity among posts with label "left". They often talk only about one particular issue and have fewer hashtags that support the left political spectrum. Additionally, the diversity of people and entities mentioned is more prominent in the posts labeled as "left" than those labeled "right" (which mainly mention president Trump). Hence, the best performance for Mix 1 is achieved when netuning with "CNN" data because the model is trained to focus more on left-relevant contexts.

The next best results for Mix 1 are achieved when ne-tuning with news articles from The Wall Street Journal because its articles often discuss both sides in detail (sometimes even in the same sentence). Hence, when the model is trained with data from this outlet, it understands relevant phrases and predicts "left" and "right" labels with higher accuracy. On the other hand, "The Wall Street Journal" ne-tuned experiment predicts much more often "right" label for "leftlabeled" example than the other experiments.

The confusion matrices created for each experiment and Figure 4 reveal that the algorithm recognizes the right label easier than the left label in Mix 2. A better understanding of the right label can be explained with the di erent writing style of left-labeled tweets, which re ects a more diverse set of topics and entities as discussed above. The best accuracy score for Mix 2 is achieved when "no news" data is used for the netuning process. Most of the labels are neutral, and news data is mainly left or right oriented/biased, so it in uences the accuracy negatively.

As hypothesized, results demonstrate that netuning with biased news datasets can in uence accuracy in contrasting ways. Di erent in uence of biased news is particularly visible in the results of Mix 1 where the di erence between the best and the worst accuracy for di erent ne-tuning settings is 9:5%. In Mix 2 this di erence is also notable, 7:2%. In uence of the bias is not uniform. While ne-tuning with "leftbiased news" gives the worst result for Mix 1, its performance for Mix 2 is average when compared to other experiments. On the other hand, ne-tuning with "all news" gives the worst results for Mix 2 and average results for Mix 1.

The size of the ne-tuning data does not seem to inuence the results. "Washington Post" has the largest amount of words, but it achieves average results in both mixes. "CNN" is the smallest dataset, but it achieves the best result for Mix 1. It is interesting to notice that "all news" achieves worse results than "no news" ne-tuning for both Mix 1 and Mix 2, even though in literature, training with more data often contributes to better results. This result suggests that the content (bias) of the ne-tuning dataset is more important than its size.

Accuracy behavior in many experiments requires further analysis in order to better understand the inuence of ne-tuning text characteristics on the performance. Additionally, the e ect of non-relevant text on the accuracy should be further tested since its frequency is high in both news and Twitter data. Since results clearly show that this model is not robust on bias and noise, other novel methods should be tested similarly. It is essential to create unbalanced and biased datasets for ne-tuning and testing of the future models to create robust methods that would be benecial to the real-world applications. 5

In this work we have shown that bias, noise and text properties need to be accounted for when constructing data for ne-tuning language models. Text size does not seem to be an important dimension. We performed experiments with data collected from Twitter and six news outlets using ULMFiT language model. Results show that the algorithm is not robust to noise in data, to bias in the ne-tuning dataset, or to the dataset imbalance.

While conducted experiments show weaknesses of the existing system, further work is needed to understand better the relationship between properties of ne-tuning data and speci c tasks. Additionally, better models are required that are more robust to bias and noise in order to be able to solve challenging realworld problems.

This research was supported in part by the NSF grants IIS-1842183. [BM18] [CSBL16] [CSPR16]