<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Content analysis methods for estimating the dynamics of facebook groups</article-title>
      </title-group>
      <contrib-group>
        <aff id="aff0">
          <label>0</label>
          <institution>Rasa Kasperienė Faculty of humanities Vytautas Magnus univercity Kaunas</institution>
          ,
          <country country="LT">Lithuania</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>Tomas Krilavičius Faculty of informatics Vytautas Magnus univercity Kaunas</institution>
          ,
          <country country="LT">Lithuania</country>
        </aff>
      </contrib-group>
      <fpage>74</fpage>
      <lpage>79</lpage>
      <abstract>
        <p>- The relationship between the content that is generated by the users of social networks and their dynamics has been analyzed by many scholars. However, due to favorable data policies, the majority studies have been carried out by analyzing Twitter data. In addition, such research on Facebook (FB) groups (esp. political) is usually qualitative. The present study analyses the dynamics as well as topic dynamics of radical right political groups on FB by employing a quantitative research methodology. The current paper draws on a large data set that is comprised of posts from FB groups. Overall, there are 79 728 posts which are made up of more than 2 million words and were generated within the timespan ranging from 2010 to 2018. The experimental set up compares the general dynamics and the dynamics of activity on four topics in two radical right FB groups (i.e., pro-Russian and other radical right) in Lithuania. The results show that the year 2014 was important for the radical right FB groups in Lithuania. Newly created proRussian FB groups started growing rapidly, whereas the posting activity in other radical right FB groups started to decrease. The topic word Lithuania is relevant for the whole activity time when it comes to all the radical right FB groups. Such topic words as Russia and land correlate with national and international political crisis.</p>
      </abstract>
      <kwd-group>
        <kwd>Facebook groups</kwd>
        <kwd>radical right</kwd>
        <kwd>groups dynamics</kwd>
        <kwd>timestamp</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>I. INTRODUCTION</title>
      <p>In recent years, the European Union has been witnessing
the growth of radical political communities throughout
Europe, including Lithuania. Many European countries are
witnessing elections in which people vote for far-right and
nationalist parties, even though they are at the opposite ends
of a wide political spectrum. The migrant crisis accelerated a
backlash against the recent political balance, but the wave of
discontent also taps into long-standing fears about
globalization and dilution of national identity. The increase
in the percentage of radical wing voters substantially
surpasses the percentage of immigration inflow [1].</p>
      <p>The political radicals are more avid and enthusiastic to
adopt new technology and have thus found the virtual space
to be a uniquely useful place [2]. Through membership in
groups, one can define and confirm his/her values and beliefs
through incoming information or discussion. When members
of such groups face uncertain situations, they can gain
reassuring information about their problems and find security
in companionship [3]. It is also important to highlight the fact
that social media provides fertile ground for the
dissemination of propaganda and disinformation as well as
the manipulation of people’s perceptions and beliefs [4].
© 2019 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0)
Social networks can become a tool for manipulating the
masses and fighting wars with little to no cost.</p>
      <p>The present paper proposes a framework for carrying out
research on posts from Facebook (further FB) groups as a
means to reveal information dissemination and group
behavior patterns in communication by information
transmission dynamics in groups. In particular, the aim of this
study is to analyze the establishment of radical right FB
groups in relation to the political events of the time as well as
the dynamics of the most prominent themes by using the data
retrieved from FB groups and R toolset. This article
investigates the launch of Lithuanian radical right FB groups
in a wider political context. It is important to understand the
dynamics and the reasons behind the activity of such groups.
Another important issue is to pinpoint when the topics
discussed in the aforementioned FB groups become relevant
and no longer relevant. Finding the answers to these questions
can provide a deeper insight into the social processes of
radical right groups on FB.</p>
      <p>Such social networks as FB and Twitter have become the
most popular social networks in the world. In 2017, Twitter
had more than 330 million active users, whereas FB had more
than 2.13 billion monthly active users with a 14 per cent
increase every year [5]. This giant flow of information has
already shown to be useful for event detection [6], identifying
public health issues [7], behavioral information propagation
[1], community discovery [8], sentiment analysis [9],
identification of communication roles [9], and recently as a
means to aid political uprising [10] as well as a medium that
can help to pinpoint and analyze the act of triggering an
(upcoming) uprising [11].</p>
      <p>FB groups are the place for small group communication
and for people to share their common interests and express
their opinion. Such groups allow people to come together
around a common cause, issue or activity in order to mobilize,
express their objectives, discuss issues, post photos and share
related content [12]. All FB groups have a title and a group
description that indicate the common cause of group activity.
FB groups can be public or closed. In the first scenario, every
FB user can access group content. In the latter, content can be
accessed only with a permission given by the group
administrator. To comply with the ethical aspects of doing
research, the present study only reports on data that has been
retrieved from public FB groups.</p>
      <p>The data were downloaded by using the FB graph API
[13]. The Graph API is created to get data into and out of the
FB platform. This FB platform uses low-level HTTP-based
API access that can only be obtained by a user who is
registered as a FB developer. For API requests, it is necessary
to have the access token (app id) together with its app
password and the access token.</p>
      <p>FB API requests return the following group data [13]:
post author id (from_id) as numeric string, post author name
(from_name) as string, post text (message) as string, post
creation date (created_time) as string, post type (type) as
string, link in the post (link) as string, post id (id) as numeric
string, daily entry (story) as bool, likes count (likes_count) as
number, comment counts (comments) as number, shares
count (shares_count) as number.</p>
      <p>To analyze the posts of FB groups as a means of
information dissemination together with the patterns of group
behavior in terms of communication by information
transmission dynamics in groups, the following subset of data
was used:
post text (message) as string, post creation date
(created_time) as string, post id (id) as numeric string.
To handle the large dataset more efficiently, fingerprint of
each FB group was created, and it contains the names and ids
as well as the names of the dataset that come from the FB
groups in focus.</p>
      <p>The radical right groups on social network FB were
identified through the Facebook search engine. The
supporters of radical right diverge from other individuals
though manifestation of nationalism, strong nation [14] and
xenophobic ideology [15]. Nationalistic ideology relates to
ethnocentrism and Euroscepticism. Xenophobic ideology
relates to anti-immigration policy, hostility to ethnical
minorities, and intolerance to sexual minorities. To identify
radical FB groups by using the FB search engine, their most
prominent characteristics were taken into consideration, and
based on that, the following keyword list was compiled:
Lithuania, Lithuanians, land sale, European Union, NATO,
refugees, refugee crisis, Muslims, Jewish restitution, Jew,
Russian, Roma tabor, gay pride, gay mountaineering. More
than 20 most recent posts in each group that match the
keywords were analyzed. After the analysis that aimed to
pinpoint the FB groups which openly exhibit radical
ideology, only 10 groups that proved to endorse radical
ideology were chosen for a more in-depth analysis. The FB
group selection criteria were the following: the presence of
radical left ideological features on group titles, description
and latest posts, the size of the group (more than 100
members), activity – the most recent post published at least 2
days prior to the analysis.</p>
      <p>The data retrieved from FB groups were divided into
two datasets, pro-Russian and other radical right groups. The
analysis reveals that some radical right groups in addition to
the nationalistic ideology manifest pro-Russian and
prosocialism ideology. Even though in some cases the titles and
descriptions of the group’s manifest nationalism and the idea
of strong Lithuania, there was also support for Russian
politics or a sense of nostalgia for the Soviet Union.
Each data set is comprised of five FB groups. As was
previously indicated, to be able to handle such large amounts
of data, the datasets were supplemented with additional
records, i.e., the group and cluster ids. The dataset of
proRussian FB groups consists of more than 70 150 posts. The
second dataset, i.e. that of the other radical right groups, is
comprised of 9 578 posts. The former dataset of groups has
13 940 members, whereas the latter has 6 126.</p>
      <p>
        Short data info
Posts published period 4th of Ma
        <xref ref-type="bibr" rid="ref20">rch 2010</xref>
        – 1st of January
2018
Number of posts 79728
Download date 12th of February 2018
Max length of word 15 symbols
Min length of word 1 symbol
      </p>
      <p>Lithuanian is a highly inflectional language, i.e. there
are two grammatical genders for nouns and there are three
genders for adjectives, numerals, participles, and pronouns.
Every word must follow the gender and the number of the
noun. All these features produce a substantial number of
inflective forms of lemma. To avoid any loss of data, the
lemmatization of the texts in FB posts was not used.</p>
    </sec>
    <sec id="sec-2">
      <title>III. METHODS</title>
      <p>To analyze the dynamics of the topics discussed in
groups, the most frequent words were employed as features
[16]–[18]. In addition, social networks post timestamp
modelling was applied to analyze the behavior of online users
[19], [20]. This paper proposes to study the posts from FB
groups as a means of information dissemination and group
behavior patterns in communication by information
transmission dynamics in groups. The proposed approach is
based on the following observation: the amount of
information passed from one period to another in the social
network may be quantified in different ways. For example, in
the dataset of FB groups, the amount of information can be
quantified by the time that passed from one post’s appearance
to other. The quantity of published group posts in a social
network by looking at the time frame can show group
behavior.</p>
      <p>To grasp the information transmission when it comes to
the group dynamics, the datasets of FB groups were expanded
by adding fingerprints entries. Let a pro-Russian FB groups
dataset be denoted by D1 and another radical right group
dataset be denoted by D2. W represents time window (W = 6
months). Denote each Facebook post as eij, where i =1
represents that a post belongs to D1 and i=2 represents that
post belongs to D2;  = 1;   where ni is the number of posts
in group Di. Each post eij consists of pij, tij, gij. Each post pij,
consists of a set of words pij = (wij1, wij2 … wijk), where k is
the number of words in pij.</p>
      <p>Fig. 1. Datasets of Facebook groups with expanded fingerprints entries
To compare the dynamics of the users in the two
datasets, the transformed dates were stored from string to
POSIXct objects. To transform the dates, Lubridate [21]
package for R was used. In order to visualize the distribution
of groups’ activities through time, ggplot2 [22]] package for
R was used. It helps to visualize the distribution of a single
continuous variable by dividing the x-axis into bins and
counting the number of observations in each bin. To make the
text of the post tidy and the datasets lighter, Tidytext [23] and
Stringr [24] packages for R were employed. By using these
packages, English and Lithuanian stop words were removed.
To estimate the dynamics of the topics in the collected posts,
each entry (in form of sentences) was split into words. Once
again, to keep track of data, every split word was
supplemented with a post and dataset id, group name, and
timestamp entries.</p>
    </sec>
    <sec id="sec-3">
      <title>IV. EXPERIMENT</title>
      <p>
        The preliminary analysis identified two types of radical
right ideology in FB groups under investigation. The
visualization in “Fig. 2” compares the dynamics of
proRussian and other radical right groups’ activity. It includes
the posts (message) of both groups’ members and post
creation time (created_time). It also shows the peak activity
periods that can be noticed in the datasets (within a time
window of six months).
The experiment shows that the activity of radical
        <xref ref-type="bibr" rid="ref20">right
FB groups starts in 2010</xref>
        , whereas pro-Russian groups
emerge on FB four ye
        <xref ref-type="bibr" rid="ref10">ars later, in 2014</xref>
        . The pro-Russian
groups that were created on the same year reached three times
greater activity compared to other radic
        <xref ref-type="bibr" rid="ref10">al right groups on FB.
From 2014</xref>
        to 2017, the activity of pro-Russian groups has
been increasingly growing. The activity has reached the
maximum peak in 2017 with 23 413 posts per year “table 2”.
Until 2014, the radical right groups were witnessing the
growth of posting activity every ye
        <xref ref-type="bibr" rid="ref10">ar, too. The year 2014</xref>
        was
important for the radical right FB groups as new
ideologyfollowing radical right groups started appearing and rapidly
growing. After the appearance of pro-Russian groups on FB,
the data spread in other radical right groups started
decreasing, but the activity of pro-Russian groups on FB
increased each year. T
        <xref ref-type="bibr" rid="ref25">his is evident because in 2015</xref>
        , the
activity of pro-Russian groups on FB was 61 per cent greater
than in previous year. Finally, in 2017, the posting activity in
pro-Russian FB groups is 44 per cent gre
        <xref ref-type="bibr" rid="ref10">ater than it was
initially in 2014</xref>
        .
58 311 604 1416 2997 791 1914 1739
Total 58 311 604 1416 13444 17482 21117 25215
      </p>
      <p>
        During the course of the Ukrainian crisis, the role of
actual military interventions has remained low in comparison
to different tools of asymmetric warfare (e.g., information
warfare, economic measures, cyber war, and psychological
war on all levels), often referred to as hybrid warfare [25].
This cyber war passed national or post-Soviet Union borders
more widely and the spread of fake news reached the western
world. The conflict in the Ukraine re-awakened Russian
propaganda. For example, Twitter analyst Lawrence
Alexander has identified an increase in bot registration
coinciding with the start of the Euromaidan protests on
2013/2014 year in Ukraine and subsequent armed uprisings
by pro-Russian militants in Eastern Ukr
        <xref ref-type="bibr" rid="ref10">aine in early spring of
2014</xref>
        [26]. Lawrence’s investigation corelates with rise of
pro-Russi
        <xref ref-type="bibr" rid="ref10">an Facebook groups in 2014</xref>
        . Prior to 2014, on FB
there were only radical right groups with low
        <xref ref-type="bibr" rid="ref10">activity, but
after 2014</xref>
        , the situation has changed. The activity of the
newly created pro-Russian groups started rapidly growing.
According to NATO Strategic Communications Centre of
Excellence, some techniques, such as Russian propaganda
techniques in particular, are used for achieving psychological
influence and manipulation on social media [27]. One of such
techniques is the mass-generated content which is used in
order to spread manipulative messages and minimize
alternative voices.
      </p>
      <p>To analyze the dynamics of the most relevant topics in
the groups, four keywords were chosen, namely, Lithuania,
Russia, land, and sky. The words Lithuania, land and Russia
were chosen for this experiment based on the previously
defined most prominent characteristics of radical right
groups. The word Russia also was chosen in order to assess
and compare the dynamics of topics discussed by pro-Russian
and other radical right in relation to the country. The neutral
word sky was chosen to reveal whether there is any space for
neutral topics in the datasets of radical right groups.</p>
      <p>TABLE III.</p>
      <p>Radica
l right
groups</p>
      <p>ProRussian</p>
      <p>Other
Total
The dynamics of the word Lithuania in the posts of radical right
groups on Facebook</p>
      <p>Year</p>
      <p>The topic word Lithuania is relevant for all the radical
right groups “Fig. 3”. This word in the posts of FB groups
appears more than 22 300 times throughout the whole period
of groups’ activity “table 3”. In 2014, the topics that
mentioned the word Lithuania were mostly discussed by
newly created pro-Russian groups rather than by other radical
right groups. In 2017, both types of radical right FB groups
mentioned Lithuania in the content of their posts the most
frequently if compared to the previous years. Lithuania
appears 5 302 times in pro-Russian groups and 857 times in
radical right groups.
words shows that from the beginning to the end of 2018, the
words Russia appeared 16 times more than in other radical
right groups. The word count indicates that the word appeared
2 864 times in pro-Russian and 178 times in other radical
right groups “table 4”.</p>
      <p>
        As was previously mentioned, the increased instances
of mentioning Lithuania were the most prominent in
proRussian groups. NATO Strategic Communication Centre of
Excellence claims that in the period ranging from 1
Novem
        <xref ref-type="bibr" rid="ref1">ber 2017</xref>
        to 31 January 2018, the proportion of bot
activity in Twitter was relatively high, with 62 per cent of all
tweets mentioning NATO and Lithuania [31]. In other radical
right FB groups, Lithuania is mentioned less often as opposed
to the pro-Russian groups. The data in the NATO report
correlate with the experimental results. The Russian hybrid
troll or bot activity campaign has reached the users of social
networks in Lithuania, and the experiment shows that this
campaign is still being successfully implemented. According
to NATO Hybrid trolls (as we have labelled hired,
proRussian trolls), communicate a particular ideology and, most
importantly, operate under the direction and orders of a
particular state or state institution. In the context of the
Ukraine crisis, the aim of hybrid trolls has been to promote
the Kremlin’s interests and portray Russia as a positive force
against the ‘rotten West’ and the US hegemony[28].
      </p>
      <p>Russia-related topics seem to be more important to
proRussian groups than other radical right FB groups (Fig. 4).
The word analysis of the FB groups’ posts that were split to</p>
      <p>
        The word Russia in the topics of pro-Russian groups
w
        <xref ref-type="bibr" rid="ref10">as most frequently used in 2014</xref>
        <xref ref-type="bibr" rid="ref26 ref33">and 2015</xref>
        . This data
correlate with Russia’s policy and international political
crises of 2013
        <xref ref-type="bibr" rid="ref26 ref33">and 2015</xref>
        – after Russian military intervention
to Ukraine, various sanctions were imposed on Russia by
the United States, the European Union (EU) and other
countries as well as intern
        <xref ref-type="bibr" rid="ref26 ref33">ational organizations. In 2015</xref>
        ,
Russia intervened to Syrian civil war (30 September 2015 –
February 2016) and this event correlates with the dynamics
of the topics on Russia in pro-Russian FB groups Russian.
The members of other radical right groups show no attention
to this international crisis, the Russian topic in their FB posts
is irrelevant.
      </p>
      <p>Creating ‘noise’ or ‘informational fog’ around a topic is
a strategy used to distract attention from more strategically
important events. An important example of this has been the
case of the downing of Malaysian air flight MH17. Russian
media channels and social media distributed a large volume
of messages offering numerous explanations for why the
plane crashed. Another bot campaign was launched to distract
the public by offering an alternative explanation of the
murder of the Russian politician Boris Nemtsov, claiming
that he was killed by jealous Ukrainians. Such ‘news’ were
published just a few hours after the attack [1]. The experiment
shows that the word Russia in the pro-Russian groups became
more actively used during the turmoil caused by Russia’s
policy. This could have affected the results of the trending
topics in order to make ‘noise’ or ‘informational fog’ around
any given topic.</p>
      <p>
        The themes related to land are more relevant to the
members of both groups. The word count estimations show
that f
        <xref ref-type="bibr" rid="ref20">rom 2010</xref>
        to 2017, the word land appeared 110 times in
pro-Russian groups and 141 times in other radical groups
“table 5”. The assessment of the thematic dynamics of the
groups indic
        <xref ref-type="bibr" rid="ref10">ate that in 2014</xref>
        , the word land was more popular
in the posts of other radical right groups than in what was
posted by pro-Russian users (Fig. 5).
0 0 0 0 46 15 32 17
0 0 3 41 94 1 1 1
0 0 3 41 140 16 33 18
      </p>
      <p>
        The word correlates with the Lithuanian land-related
political crisis related to the restrictions imposed on
foreigners who want to purchase land for agricultural
purposes in Lithuania. The referendum by the Republic of
Lithu
        <xref ref-type="bibr" rid="ref10">ania held on 2014</xref>
        July was related to the
abovementioned restrictions. Prior to the referendum, there
were many protests and a rally against land purchase
restrictions. These events also ignited debates in the virtual
space and affected the topics that were generated in the
radical right FB groups.
      </p>
      <p>In order to compare the content of the posts in radical
right FB groups, a neutral keyword sky was chosen. The
assessment of dynamics show that the word sky did not
appear in the content produced by the radical right FB groups
Radica
l right 201 201 201 201 201 201 201 201
groups 0 1 2 3 4 5 6 7</p>
      <p>ProRussian 0 0 0 0 1 5 4 13
Other 0 0 0 0 0 0 0 1
Total 0 0 0 0 1 5 4 14</p>
      <p>This indicates that the content generated by the
members of the radical right groups is similar to the political
background. As Veronika Solovian, the administrator of the
popular Finnish-Russian website russia.fi, admits, the trolls
are commenting on political topics. They are able to attract
other participants into arguments, and other users do not
necessarily immediately identify them as trolls [29]. The
experiment reveals that political topics are indeed relevant for
radical right-wing political groups on Facebook. The largest
part of the generated political content could be generated by
trolls or bots. Therefore, social media provides fertile ground
for the dissemination of propaganda and disinformation. The
latter indicates that social media can be an effective tool to
manipulate people’s mind and influence their decisions. Ease
of Use</p>
    </sec>
    <sec id="sec-4">
      <title>V. CONCLUSIONS AND FUTURE WORK</title>
      <p>Facebook developer acc with API requests and R tools
set (Lubridate, Tidytext, ggplot2) can help to analyze radical
right FB groups establishment and themes dynamics. For
social and political scientists, the most important result is that
in Lithuania radical right groups on Facebook posts together
with nationalism, strong nation and xenophobic ideology also
appears topics related to the support for the Russian policy
and former communist ideology. The analysis reveals that
some radical right groups in addition to the nationalistic
ideology manifest pro-Russian and pro-socialism ideology.</p>
      <p>
        Radical right groups on Facebook sta
        <xref ref-type="bibr" rid="ref20">rted to appear in
2010</xref>
        , but the ye
        <xref ref-type="bibr" rid="ref10">ar 2014</xref>
        was important for the radical right
FB groups as new ideology-following radical right groups
appeared and was rapidly growing each year. Experiment
data correlates with the awakening of Russian propaganda on
social media.
      </p>
      <p>
        The topic word Lithuania is relevant for all the radical
right groups. This word in the posts of FB groups appears
more than 22 300 times throughout the whole period of
groups’ activity. The increased instances of mentioning
Lithuania were the most prominent in pro-Russian groups.
Russia-related topics seem to be more important to
proRussian groups than other radical right FB groups and
landrelated topics is more important to other radical right groups.
These topics activity correlates with national or international
political crisis: the land-related topics activity reaches its
maximum before referendum related to the restrictions
imposed on foreigners who want to purchase land for
agricultural purposes in Lithuania, the word Russia in the
topics of pro-Russian groups w
        <xref ref-type="bibr" rid="ref10">as most frequently used in
2014</xref>
        <xref ref-type="bibr" rid="ref26 ref33">and 2015</xref>
        while after Russian military intervention to
Ukraine, various sanctions were imposed on Russia by
the United States, the European Union (EU) and other
countries as well as international organizations. The
assessment of dynamics show that the word sky did not
appear in the content produced by the radical right FB groups.
This indicates that the content generated by the members of
the radical right groups is similar to the political background.
      </p>
      <p>Future plans are to make different kind of radical right
groups generated content most frequency words estimations
and analyze it dynamics. Future work is also to analyze
dynamics of FB groups incoming information and the posting
dynamics of most active groups’ members.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          <string-name>
            <given-names>B.</given-names>
            <surname>Podobnik</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Jusup</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Kovac</surname>
          </string-name>
          , and
          <string-name>
            <given-names>H. E.</given-names>
            <surname>Stanley</surname>
          </string-name>
          , “
          <article-title>Predicting the Rise of EU Right-Wing Populism in Response to Unbalanced Immigration,” Complexity</article-title>
          , vol.
          <year>2017</year>
          , pp.
          <fpage>1</fpage>
          -
          <lpage>12</lpage>
          , Aug.
          <year>2017</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          <string-name>
            <given-names>J.</given-names>
            <surname>Bartlett</surname>
          </string-name>
          , “
          <article-title>From hope to hate: how the early internet fed the far right | World news | The Guardian</article-title>
          ,”
          <year>2017</year>
          . [Online]. Available: https://www.theguardian.com/world/2017/aug/31/far-right
          <article-title>-altright-white-supremacists-rise-online</article-title>
          . [Accessed:
          <fpage>03</fpage>
          -Mar-2019].
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          <string-name>
            <given-names>D. R.</given-names>
            <surname>Forsyth</surname>
          </string-name>
          and
          <string-name>
            <given-names>D. R.</given-names>
            <surname>Forsyth</surname>
          </string-name>
          , Group dynamics, 2nd ed.
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          <string-name>
            <given-names>Pacific</given-names>
            <surname>Grove Calif</surname>
          </string-name>
          .: Brooks/Cole Pub. Co,
          <year>1990</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          <source>NATO strategic communications centre of excellence, “Robotrolling</source>
          <year>2018</year>
          /1 | StratCom,”
          <year>2018</year>
          . [Online]. Available: https://www.stratcomcoe.org/robotrolling-20181. [Accessed:
          <fpage>03</fpage>
          - Mar-2019].
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          <source>“Top 20 Facebook Statistics - Updated March</source>
          <year>2019</year>
          .” [Online].
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          Available: https://zephoria.com/top-15
          <string-name>
            <surname>-</surname>
          </string-name>
          valuable-facebookstatistics/. [Accessed:
          <fpage>03</fpage>
          -Mar-2019].
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          <string-name>
            <given-names>J.</given-names>
            <surname>Weng</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Yao</surname>
          </string-name>
          , E. Leonardi, and
          <string-name>
            <given-names>F.</given-names>
            <surname>Lee</surname>
          </string-name>
          , “Event Detection in Twitter,”
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          <string-name>
            <surname>M. J. Paul</surname>
            and
            <given-names>M.</given-names>
          </string-name>
          <string-name>
            <surname>Dredze</surname>
          </string-name>
          , “
          <article-title>You Are What You Tweet: Analyzing Twitter for Public Health</article-title>
          .”
          <string-name>
            <given-names>A.</given-names>
            <surname>Java</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Song</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Finin</surname>
          </string-name>
          , and
          <string-name>
            <given-names>B.</given-names>
            <surname>Tseng</surname>
          </string-name>
          , “
          <article-title>Why We Twitter: Understanding Microblogging Usage</article-title>
          and Communities,”
          <year>2007</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          <string-name>
            <given-names>A.</given-names>
            <surname>Ortigosa</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J. M.</given-names>
            <surname>Martín</surname>
          </string-name>
          , and
          <string-name>
            <given-names>R. M.</given-names>
            <surname>Carro</surname>
          </string-name>
          , “
          <article-title>Sentiment analysis in Facebook and its application to e-learning,”</article-title>
          <string-name>
            <given-names>Comput. Human</given-names>
            <surname>Behav</surname>
          </string-name>
          .,
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          <string-name>
            <given-names>J.</given-names>
            <surname>Park</surname>
          </string-name>
          , W. L.
          <article-title>-P. of the fifth international A</article-title>
          . conference,
          <source>and undefined</source>
          <year>2011</year>
          , “
          <article-title>Revolution 2.0 in Tunisia and Egypt: Reactions and sentiments in the online world,” icwsm</article-title>
          .org.
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          <string-name>
            <given-names>K.</given-names>
            <surname>Alaimo</surname>
          </string-name>
          , “
          <article-title>How the Facebook Arabic Page 'We Are All Khaled Said' Helped Promote the Egyptian Revolution</article-title>
          ,
          <source>” Soc. Media + Soc.</source>
          , vol.
          <volume>1</volume>
          , no.
          <issue>2</issue>
          , p.
          <fpage>205630511560485</fpage>
          ,
          <string-name>
            <surname>Sep</surname>
          </string-name>
          .
          <year>2015</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          <string-name>
            <given-names>Matt</given-names>
            <surname>Hicks</surname>
          </string-name>
          , “
          <article-title>(18) Facebook Tips: What's the Difference between a Facebook Page</article-title>
          and Group? | Facebook,”
          <year>2010</year>
          . [Online].
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          Available: https://www.facebook.com/notes/facebook/facebooktips
          <article-title>-whats-the-difference-between-a-facebook-page-andgroup/324706977130/</article-title>
          . [Accessed:
          <fpage>03</fpage>
          -Mar-2019].
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          <string-name>
            <given-names>“Graph</given-names>
            <surname>API</surname>
          </string-name>
          .” [Online]. Available: https://developers.facebook.com/docs/graph-api. [Accessed:
          <fpage>03</fpage>
          - Mar-2019].
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          <string-name>
            <given-names>K.</given-names>
            <surname>Arzheimer</surname>
          </string-name>
          , “Electoral Sociology:
          <article-title>Who Votes for the Extreme Right and why-and when?” C. Mudde, The ideology of the extreme right</article-title>
          .
          <year>2018</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          <string-name>
            <surname>J. B.-L</surname>
            . and
            <given-names>L.</given-names>
          </string-name>
          <string-name>
            <surname>Computing</surname>
          </string-name>
          and undefined
          <year>1992</year>
          , “
          <article-title>Not unles you ask nicely: The interpretative nexus between analysis and information</article-title>
          ,” academic.oup.com.
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          <string-name>
            <surname>D. H.-L</surname>
            . and
            <given-names>L.</given-names>
          </string-name>
          <string-name>
            <surname>Computing</surname>
            and
            <given-names>U.</given-names>
          </string-name>
          <year>2004</year>
          , “Delta prime?,” academic.oup.com.
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          <string-name>
            <given-names>M.</given-names>
            <surname>Eder</surname>
          </string-name>
          ,
          <string-name>
            <surname>J. R.-L.</surname>
          </string-name>
          and
          <string-name>
            <given-names>L.</given-names>
            <surname>Computing</surname>
          </string-name>
          , and undefined
          <year>2012</year>
          , “
          <article-title>Do birds of a feather really flock together, or how to choose training samples for authorship attribution</article-title>
          ,” academic.oup.com.
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          <string-name>
            <given-names>R.</given-names>
            <surname>Perera</surname>
          </string-name>
          , … S. A.-2010
          <string-name>
            <surname>-M</surname>
          </string-name>
          .
          <year>2010</year>
          , and undefined
          <year>2010</year>
          , “
          <article-title>Twitter analytics: Architecture, tools and analysis,” ieeexplore</article-title>
          .ieee.org.
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          <string-name>
            <given-names>Q.</given-names>
            <surname>Zhao</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Tian</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Q.</given-names>
            <surname>He</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Oliver</surname>
          </string-name>
          , … R. J.-
          <source>P. of the 19th, and undefined</source>
          <year>2010</year>
          , “
          <article-title>Communication motifs: a tool to characterize social communications,” dl</article-title>
          .acm.org.
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          <string-name>
            <given-names>G.</given-names>
            <surname>Grolemund</surname>
          </string-name>
          , H. W.-
          <source>J. of S. Software, and undefined</source>
          <year>2011</year>
          , “
          <article-title>Dates and times made easy with lubridate,” academia</article-title>
          .edu.
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          <string-name>
            <given-names>L.</given-names>
            <surname>Wilkinson</surname>
          </string-name>
          , “ggplot2:
          <article-title>Elegant Graphics for Data Analysis by WICKHAM, H</article-title>
          .,” Biometrics, vol.
          <volume>67</volume>
          , no.
          <issue>2</issue>
          , pp.
          <fpage>678</fpage>
          -
          <lpage>679</lpage>
          , Jun.
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          <string-name>
            <given-names>J.</given-names>
            <surname>Silge</surname>
          </string-name>
          ,
          <string-name>
            <surname>D.</surname>
          </string-name>
          R.-
          <source>T. J. of O. S. Software, and undefined</source>
          <year>2016</year>
          ,
          <article-title>“tidytext: Text mining and analysis using tidy data principles in r,” theoj</article-title>
          .org.
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          <string-name>
            <given-names>H.</given-names>
            <surname>Wickham</surname>
          </string-name>
          , “Simple,
          <source>Consistent Wrappers for Common String Operations [R package stringr version 1.4</source>
          .0].”
          <string-name>
            <given-names>V.</given-names>
            <surname>Sazonov</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Molder</surname>
          </string-name>
          , and Muur Kristiina, “
          <article-title>COMBINED ANALYSIS PREPARED BY THE NATO STRATEGIC COMMUNICATIONS CENTRE OF EXCELLENCE RUSSIAN INFORMATION CAMPAIGN AGAINST THE UKRAINIAN STATE</article-title>
          AND DEFENCE FORCES,”
          <year>2015</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>
          <string-name>
            <given-names>A.</given-names>
            <surname>Lawrence</surname>
          </string-name>
          , “
          <article-title>Social Network Analysis Reveals Full Scale of Kremlin's Twitter Bot Campaign · Global Voices</article-title>
          ,”
          <year>2015</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref27">
        <mixed-citation>
          [Online]. Available: https://globalvoices.org/
          <year>2015</year>
          /04/02/analyzing-kremlin-twitterbots/. [Accessed:
          <fpage>03</fpage>
          -Mar-2019].
        </mixed-citation>
      </ref>
      <ref id="ref28">
        <mixed-citation>
          <string-name>
            <given-names>Svetoka</given-names>
            <surname>Sanda</surname>
          </string-name>
          , “
          <article-title>Social Media as a Tool of Hybrid Warfare | StratCom</article-title>
          ,”
          <year>2016</year>
          . [Online]. Available: https://www.stratcomcoe.
          <article-title>org/social-media-tool-hybrid-warfare.</article-title>
        </mixed-citation>
      </ref>
      <ref id="ref29">
        <mixed-citation>
          [Accessed:
          <fpage>03</fpage>
          -Mar-2019].
        </mixed-citation>
      </ref>
      <ref id="ref30">
        <mixed-citation>
          <string-name>
            <surname>NATO</surname>
          </string-name>
          , “
          <article-title>Internet Trolling as a hybrid warfare tool: the case of Latvia | StratCom</article-title>
          ,”
          <year>2016</year>
          . [Online]. Available: https://www.stratcomcoe.
          <article-title>org/internet-trolling-hybrid-warfaretool-case-latvia-0</article-title>
          . [Accessed:
          <fpage>03</fpage>
          -Mar-2019].
        </mixed-citation>
      </ref>
      <ref id="ref31">
        <mixed-citation>
          <string-name>
            <given-names>J.</given-names>
            <surname>Aro</surname>
          </string-name>
          and
          <string-name>
            <given-names>Y.</given-names>
            <surname>Kioski</surname>
          </string-name>
          , “
          <article-title>This is How Pro-Russia Trolls Manipulate Finns Online - Check the List of Forums Favored by Propagandists</article-title>
          ,”
          <year>2015</year>
          . [Online]. Available: https://www.stopfake.org/en/this-is
          <article-title>-how-pro-russia-trollsmanipulate-finns-online-check-the-list-of-forums-favored-bypropagandists/</article-title>
          . [Accessed:
          <fpage>03</fpage>
          -Mar-2019].
        </mixed-citation>
      </ref>
      <ref id="ref32">
        <mixed-citation>
          <string-name>
            <surname>Wozniak</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Polap</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Borowik</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          , &amp;
          <string-name>
            <surname>Napoli</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          (
          <year>2015</year>
          , July).
        </mixed-citation>
      </ref>
      <ref id="ref33">
        <mixed-citation>
          <article-title>A first attempt to cloud-based user verification in distributed system</article-title>
          .
          <source>In 2015 Asia-Pacific Conference on Computer Aided System Engineering</source>
          (pp.
          <fpage>226</fpage>
          -
          <lpage>231</lpage>
          ). IEEE.
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>