<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Method for identifying Twitter accounts that have changed their opinion about politicians</article-title>
      </title-group>
      <contrib-group>
        <aff id="aff0">
          <label>0</label>
          <institution>National Technical University of Ukraine “Igor Sikorsky Kyiv Polytechnic Institute”</institution>
          ,
          <addr-line>Kyiv</addr-line>
          ,
          <country country="UA">Ukraine</country>
        </aff>
      </contrib-group>
      <fpage>0000</fpage>
      <lpage>0001</lpage>
      <abstract>
        <p>Nowadays, social networks provide an opportunity to communicate with the world and share political views. Therefore, each politician has an account and many followers. Revealing political opinions on Twitter accounts can be a good alternative to regular social surveys. Identifying those who changed their opinions will help to understand the reasons for disappointment in a particular political figure. In this paper we introduce our approach to collect data from Twitter, detect accounts political views, opinion changes and the reasons people get frustrated. The experimental results showed that such research was successful in the analysis of Ukrainian President supporters.</p>
      </abstract>
      <kwd-group>
        <kwd>Twitter</kwd>
        <kwd>social network</kwd>
        <kwd>Ukraine</kwd>
        <kwd>politics</kwd>
        <kwd>machine learning</kwd>
        <kwd>natural language processing</kwd>
        <kwd>sentiment analysis</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>-</title>
      <p>Social networks have caught the attention of people around the world. It is hard to
imagine modern times without Twitter, Facebook or Instagram. While most users connect
with friends and use social networks for entertainment others see them as a tool to gain
authority or profit. The most obvious examples are politics and business. Every
politician has a page on at least one social network. This is a kind of close communication
with people, which sometimes even replaces social media. The approach has several
advantages such as speed, convenience, and a reputation as a trusted source. Among
the benefits for subscribers are analytical benefits. For example, using a business
Twitter account, you will know exactly how many times people have seen, retweeted, liked,
and replied to each tweet. Even more, you can export the data as a CSV file in order to
perform advanced analytics. Instagram Insights tool can show you when your audience
is online, which of your posts are most popular, impressions and reach of your account.
Also, you can compare your original post with a promoted version of the post to see if
your paid efforts are working. People increasingly tend to express their political
preferences in social networks.</p>
      <p>The automated approach has several advantages over traditional methods. Benefits
include speed, price and visibility.</p>
      <p>
        When we talk about social media as a political tool, the first thing that comes to mind
is Donald Trump and Twitter. Trump is successfully using social media to rally his
supporters and attract new ones [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ]. The number of monthly Twitter users worldwide
is growing continuously and as of the first quarter of 2020 has an average of 340 million
monthly active users [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ]. There is no doubt that politicians can influence large numbers
of people who use Twitter on a daily basis.
      </p>
      <p>Ukrainian president Volodymyr Zelenskyy won the presidency on April 21, 2019.
The Twitter page was created in April 2019 and the very first tweet was written on
April 25. Taking to account that Volodymyr Zelenskyy was chosen by 73.22 percent of
voters, support in those days was overwhelming. Twitter was not a part of the elections
campaign, but as we can see now, it is an important part of communicating with people.
Ukrainian social polls and the media regularly talk about the decline in the president’s
ratings. The question this paper is addressing is the following: can we collect data from
Twitter and use it as an alternative to conventional approaches to predicting election
results and understand the most important reasons for people’s choice? Is it possible to
find people who have changed their opinions about Volodymyr Zelenskyy and his
political party “Servant of the People”? Can we figure out which events affect people the
most?</p>
      <p>The plan of the paper is as follows. Section 2 presents the related paper and existing
solutions for similar problems. Section 3 explains our approach to finding accounts that
have changed their opinions about politicians. Starting with the identification of
supporters, and their classification according to opinions that change over time. The section
ends with an approach to identifying frustrated people through follower dynamics.
Section 4 explains in detail the research carried out and the experimental results. The last
section concludes the discussion of plans for future work.
2</p>
    </sec>
    <sec id="sec-2">
      <title>Related Paper</title>
      <p>
        The specificity of social media is manifested in the fact that, as a rule, only the most
active members of society declare their political preferences. Nevertheless, some
studies show that it is in social networks and largely due to the availability of its appropriate
tools (setting preferences, filters, search history, etc.) the so-called “echo chambers”
and “epistemic bubbles” are formed [
        <xref ref-type="bibr" rid="ref3 ref4">3, 4</xref>
        ]. And people who find the support of their
like-minded people in such communities tend to be less active in discussing their
political preferences offline, for example, in their workplace [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ]. In fact, data for only the
part of society that is most active on social media is available for research. But this data
can give us a correct picture of the mood in the whole society, provided that fake news
and accounts are weeded out [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ].
      </p>
      <p>Twitter is perhaps the most popular social networking site for research. This can be
explained by three reasons: 1) the small size of the tweets, which makes them easier to
collect and process; 2) active use of this social network by US President Donald Trump
and a number of other politicians; 3) availability of accessible API. Unfortunately, in
recent times, many of Twitter's API capabilities have been blocked or cut off.</p>
      <p>
        From the point of view of studying the political preferences of Twitter users, there
are two main approaches in the academic literature: based on sentiment analysis and
based on the study of account links [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ].
      </p>
      <p>
        Various sentiment analysis methods that have been proposed for eliciting political
opinions of Twitter users can be divided into three groups [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ]: untargeted/overall
sentiment, targeted sentiment, and stance detection methods.
      </p>
      <p>
        Untargeted methods are based on either hand-curated [
        <xref ref-type="bibr" rid="ref10 ref9">9, 10</xref>
        ] or supervised
learningselected [
        <xref ref-type="bibr" rid="ref11 ref12">11, 12</xref>
        ] lists of positive and negative words. The final rating of a tweet takes
into account the (possibly weighted) ratio of these two types of words.
      </p>
      <p>
        Unlike untargeted sentiment analysis methods, target methods have a predefined
person, entity or topic about which the sentiment of the author of the text should be
determined [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ]. In turn, stance detection methods differ from target sentiment analysis
methods in that the target is indicated in them, but implicitly [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ].
      </p>
      <p>
        Hence, the sentimental analysis approach has two significant drawbacks:
1) the vast majority of Twitter users write very little themselves; for example,
according to a 2018 Pew Research Center survey, 90% of American users
wrote two or fewer tweets per month [
        <xref ref-type="bibr" rid="ref15">15</xref>
        ];
2) back in 2013, it was discovered that discriminating values of features for
politically active and modest Twitter users should be different [
        <xref ref-type="bibr" rid="ref16">16</xref>
        ].
      </p>
      <p>
        Article [
        <xref ref-type="bibr" rid="ref17">17</xref>
        ] was a pioneering study showing the importance of exploring the
structure and internal links of social networks using the example of blogs. Different online
social networks have their own means of expressing structural links between their
members. However, in all of them, you can build social embedding either by specifying
friends, or by specifying users followed by a given account [
        <xref ref-type="bibr" rid="ref18">18</xref>
        ].
      </p>
      <p>
        Twitter has a number of additional features that can be viewed as associative links
between accounts, namely: each Twitter account can answer, cite, comment or retweet
a tweet from another one. A somewhat unexpected result was obtained in article [
        <xref ref-type="bibr" rid="ref19">19</xref>
        ]:
it turned out that for political debates, the study of retweets is more important than the
other “links” mentioned.
      </p>
      <p>
        We use a combination of two approaches, downloading not only tweets about a
specific topic but also account (profile) data. We download those who showed activity on
the page of Volodymyr Zelenskyy or subscribed to him. After that, we download as
much information as possible from the page of such users, which contains all the profile
data and all the tweets. Similar approach was used in [
        <xref ref-type="bibr" rid="ref20">20</xref>
        ] to detect phony accounts of
Ukrainian politicians on Facebook pages. This approach was based on finding groups
of accounts that leave mostly positive comments, but at the same time start writing
negative ones.
      </p>
      <p>One of the reasons for constraining research in this domain is the lack of publicly
available complete datasets containing the necessary information. This situation can be
explained, on the one hand, by the complexity of collecting relevant data, since social
networking APIs have a number of significant limitations. On the other hand, this is
due to the strict requirements of the legislation to protect the privacy of personal data.</p>
    </sec>
    <sec id="sec-3">
      <title>Method for finding accounts that have changed their opinion about a politician</title>
      <sec id="sec-3-1">
        <title>Problem formulation and solution</title>
        <p>In this work, we develop an automated approach to find Twitter accounts that first
supported politician but changed the opinion about him/her after some time. We assume
that, during or immediately after elections, people express support or hope for
politicians. After a while, frustration appears associated with unfulfilled promises or
unexpected actions. We are trying to find the accounts of such people on Twitter. The search
algorithm can be divided into several stages:
 Data collection. We collect as much data as possible about the politician and his/her
followers from Twitter pages.
 Pre-processing. All data is converted into a uniform format. We use various data
transformation techniques, such as converting date and time into one format,
discarding stop words, stemming and lemmatization. After that we create datasets for
further analysis.
 Analysis. In our case it is an iterative process of detection politician supporters,
analyzing their personal information, then tweets and classification account to one of
four groups: not active, do not show political views, supporters and haters. Iterations
can be repeated an infinite number of times as we expand our datasets and people
can change their opinions multiple times.
 Reporting. A summary of analysis results with visual representations.
3.2</p>
      </sec>
      <sec id="sec-3-2">
        <title>Advantages of automated approach</title>
        <p>Compared to traditional methods for determining the ratings of politicians, the
automated method has the following advantages:
 Speed. The approach allows you to track rating changes in real time and find the
reasons. Social polls are carried out with some frequency, but it takes time to process
the data, so for some time the public does not know the results and does not know
what could cause an increase or decrease in ratings.
 Price. The developed algorithms allow to reduce the number of people working for
the result. There is no need to interact directly with people and most of the work will
be done by machines.
 Visibility. A visual representation of the data allows an understanding of how many
people have changed their opinions and reasons.
3.3</p>
      </sec>
      <sec id="sec-3-3">
        <title>Twitter API and Selenium web driver</title>
        <p>Twitter has official API which allows to download personal information of accounts
and all tweets. To collect research data we used Python library tweepy which is based
on official Twitter API. Not all data can be fetched as API returns only fresh tweet
replies. For our research we need to collect replies left even more than one year before.
As such information is available on user interface then we can imitate browser behavior
and find necessary data. For such purpose we use Python Selenium web driver and
performed html parsing with downloaded data. Firstly, we download all Volodymyr
Zelenskyy tweet identifiers using tweepy. Then Selenium opens each tweet scrolls
down and extracts replies until the end of scroll line. All information stored to CSV
datasets in order to perform analysis. System diagram is described on Fig. 1. As input
user provides Tweeter account link and type of interested information. Based on inputs
Python controller decides download account personal information such as user name,
followers count, following count, location, description etc. or extract list of tweet
identifiers and download all replies including their authors. All datasets are stored in CSV
storage for advanced analysis.
To find accounts that have changed political positions, first of all we should find those
who supported the political leader or party. Volodymyr Zelenskyy won the presidential
elections in Ukraine on April 21, 2019. The Twitter page was created in April 2019 and
very firs tweet was written on April 25.</p>
        <p>The approach of finding Zelenskyy supporters can be described as follows:
 Collect replies to the very first tweet of the President of Ukrainian.
 Detect language of replies and filter out replies not in Ukrainian, Russian or English
as unpopular languages in Ukraine.
 Convert all messages to lower case, get rid of duplicated, stop words and punctuation
marks.
 Transform all words to the root form – perform lemmatization.
 Create a set of words that can be confidently attributed to the supporters of
Volodymyr Zelenskyy.
 Create word vectors for all tweets and the set of words.
 Measure how similar are tweet replies and our set of words using cosine similarity.
 Create a visual representation of classified messages.
3.5</p>
      </sec>
      <sec id="sec-3-4">
        <title>Identification of accounts that changed political views</title>
        <p>The next step is to find the accounts that have changed their opinion. In order to do this,
we should take a closer look into the tweets of Volodymyr Zelenskyy supporters and
find those who tweeted or retweeted some messages against the current president or his
political party “Servant of the People”. Different people can be affected by different
news, so the reasons can vary. Some news is very emotional and can affect people more
than others. In most cases, a person does not change his opinion immediately; this may
be the result of a series of unpleasant news. In this case, the last blow will be the latest
news before the change of opinion. The initial supporters of the president can be divided
into four groups, as shown in Fig. 2.
The search algorithm can be described as follows:
 Download all the tweets of detected supporters of Volodymyr Zelenskyy.
 Refer to a separate category those accounts that stopped tweeting on April or May
2019.
 By analogy with the previous algorithm, convert all messages to lower case, get rid
of duplicates, stop words, punctuation marks and perform lemmatization.
 Create set of words which can be confidently classified as belonging to politics.
 Among the political tweets, those can be detected that are against Volodymyr
Zelenskyy or “Servant of the People”, using separate set of words.
 Create a visual representation of results.
3.6
4
4.1</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>Experimental results</title>
      <sec id="sec-4-1">
        <title>Volodymyr Zelenskyy supporters</title>
      </sec>
      <sec id="sec-4-2">
        <title>The unsubscribed accounts</title>
        <p>Another way to find accounts that have changed their minds is to take a closer look at
the presidential followers. On Twitter, Volodymyr Zelenskyy always has the dynamics
of subscriptions. Some people do not show changes in their opinion but have stopped
watching the actions of the president. We assume that the person who is disappointed
by the incumbent president can unsubscribe from his page.</p>
        <p>First tweet contains 351 replies. Using Python library – fasttext, tweets were divided
into four parts: Ukrainian, Russian, English and other. The last part was filtered out as
it is very rare case. Tweets contain unnecessary objects like punctuation, stop words,
and mentions that can affect the performance of an algorithms. For each tweet we filter
common words that add noise or provide no value for analysis. Examples: the, a, an,
in. Python library – nltk contains stop words for all interested languages. Also,
punctuation and mentions were removed. Each word was converted to its base form with help
of nltk.</p>
        <p>Supporting tweets contain specific words. We build the set of words that can be
confidently classified as belonging to Volodymyr Zelenskyy supporters. Such sets of
words built for three interested languages. Examples of such words: congratulations,
hope, trust, expect, adore, bow, courage, at last, reliance, stick it.</p>
        <p>Classification performed by converting all words into vectors and measuring cosine
similarity between tweet and our set of words.</p>
        <p>As a result, 84 out of 351 accounts support Volodymyr Zelenskyy (see Fig. 3).</p>
      </sec>
      <sec id="sec-4-3">
        <title>Accounts that changed their political opinion</title>
        <p>130 416 tweets were collected from 84 accounts that initially supported Volodymyr
Zelenskyy. There is a group of accounts that became inactive after replies on the
Presidents page. They could be filtered out simply by date of last tweet. 29 such accounts
were removed from dataset as those who do not need additional analysis.</p>
        <p>Second category of accounts are those who do not tweet about politics. In order to
separate this group from those who require further investigation, we created a set of
words related to Ukrainian politics. Example of words: Zelenskyy, Tishchenko,
Poroshenko, Klitschko, Servant of the People, Fatherland, Opposition Platform – For Life,
European Solidarity, For the Future, Our Land. Set contains long and shortened names
of famous politicians and political parties. Same sets were created for Ukrainian,
Russian and English languages. As a result, 26 out of 84 accounts were not politically
active.</p>
        <p>To separate the last two categories, we have reduced the set of words to those related
to Volodymyr Zelenskyy and his party. In order to detect user opinion, we build a
mapping of most annoying news about the president and "Servant of the People" decisions.
Such as corruption, nepotism, lobbying for the interests of oligarchs and anti-Ukrainian
laws. If a person tweets or retweets such news, then most likely he is against the current
president. Using this approach, we found retweets of the following news:
 Despite promises, nepotism spread in the Servant of the People.
 Corruption scandal associated with the deputy Alexander Yurchenko, who is a
member of the Servant of the People party.
 Vladymyr Zelenskyy's interaction with Ukrainian oligarchs.</p>
        <p>This finding confirms the importance of the retweets described in section 2.</p>
        <p>The technique made it possible to identify those who were against the current
President of Ukraine. As a result, 6 accounts were discovered, and the remaining 23 still
support Volodymyr Zelenskyy.</p>
        <p>Percentage of political positions of accounts which initially supported president of
Ukraine is shown on Fig. 4.</p>
        <p>Fig. 4. Percentage of political positions of accounts which initially supported president of</p>
        <p>Ukraine
4.3</p>
      </sec>
      <sec id="sec-4-4">
        <title>Subscribers analysis</title>
        <p>To detect those accounts that are disappointed in Vladimir Zelenskyy, we download the
identifiers of all the president’s followers on Twitter for 28 days. The overall dynamics
show that the total number of subscribers is growing. This may be a result of the
growing popularity of Twitter. The rate of new subscriptions has slowed down a lot over
time. The more important fact is that people continuously unsubscribe with almost the
same velocity (see Fig. 5). Loss of interest from subscribers can be the result of
unjustified expectations. The following Table 1 gives a summary of subscriptions dynamics.</p>
        <p>New subscribers</p>
        <p>Unsubscribed
5472
2866
2514
It should be noted that for now period of time is pretty small and such accounts require
further investigations. In future work we are planning to download personal information
and all tweets of unsubscribed accounts.
5</p>
      </sec>
    </sec>
    <sec id="sec-5">
      <title>Conclusions</title>
      <p>In this research work, we have demonstrated an approach to collect Twitter dataset and
classify accounts accordingly to their opinion about Volodymyr Zelenskyy and his
political party "Servant of the People" during the period of his Twitter account creation to
the 29th of November. Proposed approach allowed to detect four groups of people who
initially were supporting Ukrainian president. Majority of accounts – 34.5 percent
stopped any activity right after they replied to Volodymyr Zelenskyy first posts. It
seems that they were created only with purpose of president popularization. Second
group – 31 percent became politically not active and we could not classify their opinion.
There is still big group of accounts – 27.4 percent which continue support of Volodymyr
Zelenskyy. The last group – 7.1 percent are those who completely changed opinion
about Ukrainian president which reflects results of local elections in Ukraine and social
polls.</p>
      <p>Such approach could be used not only for politics but in all fields where popularity
matters such as companies and brands.</p>
      <p>Another part of this research work is the analysis of the dynamics of subscriptions
on the page of Volodymyr Zelenskyy. At the moment we have identifiers of subscribers
for the last 4 weeks. In future work, we will extend the observation period. In addition,
we plan to download personal information and all tweets of unsubscribed accounts in
order to more accurately find out the reasons.</p>
      <p>Acknowledgment. This study is funded by the NATO SPS Project CyRADARS
(Cyber Rapid Analysis for Defense Awareness of Real-time Situation), Project SPS
G5286.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <surname>Ott</surname>
            ,
            <given-names>B. L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dickinson</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          :
          <article-title>The Twitter Presidency: Donald J. Trump and the Politics of White Rage</article-title>
          . Routledge, Taylor &amp; Francis, New York (
          <year>2019</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <surname>Hootsuite. DIGITAL</surname>
          </string-name>
          <year>2020</year>
          .
          <article-title>Global Digital Overview</article-title>
          , https://hootsuite.com/pages/digital2020, https://www.statista.com/statistics/282087/number-of
          <article-title>-monthly-active-twitter-users</article-title>
          ,
          <source>last accessed</source>
          <year>2020</year>
          /12/01.
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <surname>Dahlgren</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          <string-name>
            <surname>Media</surname>
          </string-name>
          ,
          <source>Knowledge and Trust: The Deepening Epistemic Crisis of Democracy. Javnost - The Public</source>
          <volume>25</volume>
          (
          <issue>1-2</issue>
          ),
          <fpage>20</fpage>
          -
          <lpage>27</lpage>
          (
          <year>2018</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <surname>Nguyen</surname>
            ,
            <given-names>C. T. Echo</given-names>
          </string-name>
          <string-name>
            <surname>Chambers</surname>
            and
            <given-names>Epistemic</given-names>
          </string-name>
          <string-name>
            <surname>Bubbles</surname>
          </string-name>
          . Episteme.
          <volume>17</volume>
          (
          <issue>2</issue>
          ),
          <fpage>141</fpage>
          -
          <lpage>161</lpage>
          (
          <year>2020</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <surname>Hampton</surname>
            ,
            <given-names>K. N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Shin</surname>
            ,
            <given-names>I.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lu</surname>
            ,
            <given-names>W.</given-names>
          </string-name>
          <article-title>Social media and political discussion: when online presence silences offline conversation</article-title>
          .
          <source>Information, Communication &amp; Society</source>
          ,
          <volume>20</volume>
          (
          <issue>7</issue>
          ),
          <fpage>1090</fpage>
          -
          <lpage>1107</lpage>
          (
          <year>2017</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <surname>Gurajala</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>White</surname>
            ,
            <given-names>J. S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hudson</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Matthews</surname>
            ,
            <given-names>J. N.</given-names>
          </string-name>
          <article-title>Fake Twitter accounts: Profile characteristics obtained using an activity-based pattern detection approach</article-title>
          .
          <source>In: SMSociety '15: International Conference on Social Media &amp; Society</source>
          , ACM, Toronto (
          <year>2015</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <surname>Gaumont</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Panahi</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Chavalarias</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          <article-title>Reconstruction of the socio-semantic dynamics of political activist Twitter networks-Method and application to the 2017 French presidential election</article-title>
          .
          <source>PLoS ONE</source>
          <volume>13</volume>
          (
          <issue>9</issue>
          ): e0201879 (
          <year>2018</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <surname>Sen</surname>
            ,
            <given-names>I.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Flöck</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wagner</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          <article-title>On the Reliability and Validity of Detecting Approval of Political Actors in Tweets</article-title>
          .
          <source>In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)</source>
          , pp.
          <fpage>1413</fpage>
          -
          <lpage>1426</lpage>
          . Association for Computational Linguistics (
          <year>2020</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <given-names>O</given-names>
            <surname>'Connor</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            ,
            <surname>Balasubramanyan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            ,
            <surname>Routledge</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B. R.</given-names>
            ,
            <surname>Smith</surname>
          </string-name>
          ,
          <string-name>
            <surname>N. A.</surname>
          </string-name>
          <article-title>From tweets to polls: Linking text sentiment to public opinion time series</article-title>
          .
          <source>In: Proceedings of the Fourth International Conference on Weblogs and Social Media (ICWSM</source>
          <year>2010</year>
          ), DBLP, Washington (
          <year>2010</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10.
          <string-name>
            <surname>Pasek</surname>
            ,
            <given-names>J</given-names>
          </string-name>
          , Yan,
          <string-name>
            <given-names>Y. Y.</given-names>
            ,
            <surname>Conrad</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F. G.</given-names>
            ,
            <surname>Newport</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            ,
            <surname>Marken</surname>
          </string-name>
          ,
          <string-name>
            <surname>S.</surname>
          </string-name>
          <article-title>The stability of economic correlations over time: Identifying conditions under which survey tracking polls and twitter sentiment yield similar conclusions</article-title>
          .
          <source>Public Opinion Quarterly</source>
          ,
          <volume>82</volume>
          (
          <issue>3</issue>
          ), pp.
          <fpage>470</fpage>
          -
          <lpage>492</lpage>
          (
          <year>2018</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <string-name>
            <surname>Socher</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Perelygin</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wu</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Chuang</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Manning</surname>
            ,
            <given-names>C. D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ng</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Potts</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          <article-title>Recursive deep models for semantic compositionality over a sentiment tree-bank</article-title>
          .
          <source>In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing</source>
          , pp.
          <fpage>1631</fpage>
          -
          <lpage>1642</lpage>
          , Seattle, Washington, Association for Computational Linguistics (
          <year>2013</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          12.
          <string-name>
            <surname>Thelwall</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          <article-title>The heart and soul of the web? Sentiment strength detection in the social web with SentiStrength</article-title>
          .
          <source>In: Holyst J. (eds) Cyberemotions. Understanding Complex Systems</source>
          , pp.
          <fpage>119</fpage>
          -
          <lpage>134</lpage>
          . Springer, Cham (
          <year>2017</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          13.
          <string-name>
            <surname>Tang</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Qin</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Liu</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          <article-title>Aspect level sentiment classification with deep memory network</article-title>
          .
          <source>In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing</source>
          , pp.
          <fpage>214</fpage>
          -
          <lpage>224</lpage>
          , Austin, Texas. Association for Computational Linguistics (
          <year>2016</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          14.
          <string-name>
            <surname>Mohammad</surname>
            ,
            <given-names>S. M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sobhani</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kiritchenko</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          <article-title>Stance and sentiment in tweets</article-title>
          .
          <source>ACM Transactions on Internet Technology</source>
          ,
          <volume>17</volume>
          (
          <issue>3</issue>
          ):26, pp.
          <fpage>1</fpage>
          -
          <lpage>23</lpage>
          (
          <year>2017</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          15. PEW RESEARCH CENTER.
          <article-title>Survey of U.S. adult Twitter users conducted</article-title>
          <source>Nov. 21 - Dec. 17</source>
          ,
          <year>2018</year>
          .
          <article-title>Data about respondents' Twitter activity collected via Twitter API. “Sizing Up Twitter Users"</article-title>
          , https://www.pewresearch.org/internet/2019/04/24/sizing-up
          <string-name>
            <surname>-</surname>
          </string-name>
          twitter-users/pdl_
          <fpage>04</fpage>
          -
          <lpage>24</lpage>
          -19_twitter_
          <fpage>users</fpage>
          -00-06/, last accessed
          <year>2020</year>
          /12/01.
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          16.
          <string-name>
            <surname>Cohen</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ruths</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          <article-title>Classifying Political Orientation on Twitter: It's Not Easy</article-title>
          ! In:
          <source>International AAAI Conference on Web and Social Media</source>
          , pp.
          <fpage>91</fpage>
          -
          <lpage>99</lpage>
          (
          <year>2013</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          17.
          <string-name>
            <surname>Adamic</surname>
            ,
            <given-names>L. A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Glance</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          <article-title>The Political Blogosphere and the 2004 U.S. Election: Divided They Blog</article-title>
          .
          <source>In: Proceedings of the 3rd international workshop on Link discovery (LinkKDD'05)</source>
          , ACM, pp.
          <fpage>36</fpage>
          -
          <lpage>43</lpage>
          (
          <year>2005</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          18.
          <string-name>
            <surname>Volkova</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bachrach</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Van Durme</surname>
            ,
            <given-names>B</given-names>
          </string-name>
          ,
          <article-title>Mining User Interests to Predict Perceived Psycho-Demographic Traits on Twitter</article-title>
          .
          <source>In: 2016 IEEE Second International Conference on Big Data Computing Service and Applications (BigDataService)</source>
          , Oxford, pp.
          <fpage>36</fpage>
          -
          <lpage>43</lpage>
          (
          <year>2016</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          19.
          <string-name>
            <surname>Garimella</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>De Francisci Morales</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gionis</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Mathioudakis</surname>
            ,
            <given-names>M. Quantifying</given-names>
          </string-name>
          <article-title>Controversy in Social Media</article-title>
          .
          <source>In: Proceedings of the Ninth ACM International Conference on Web Search and Data Mining (WSDM'16)</source>
          , pp.
          <fpage>33</fpage>
          -
          <lpage>42</lpage>
          . New York, ACM Press (
          <year>2016</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          20.
          <string-name>
            <surname>Chertov</surname>
            ,
            <given-names>O.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Rudnyk</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Palchenko</surname>
            ,
            <given-names>O.</given-names>
          </string-name>
          :
          <article-title>Search of phony accounts on Facebook. Ukrainian case</article-title>
          .
          <source>In: International Conference on Military Communications and Information Systems (ICMCIS)</source>
          , pp.
          <fpage>22</fpage>
          -
          <lpage>23</lpage>
          , Warsaw (
          <year>2018</year>
          ).
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>