<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Is “manovra” really “del popolo”? Linguistic Insights into Twitter Re- actions to the Annual Italian Budget Law1</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Claudia Roberta Combei</string-name>
          <email>claudia.combei2@unibo.it</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>University of Bologna</institution>
        </aff>
      </contrib-group>
      <abstract>
        <p>English. Relying on linguistic cues obtained by means of structural topic modelling as well as descriptive lexical analyses, this study contributes to the general understanding of the Twitter users' response to the annual Italian budget law approved at the end of December 2018. Some topics contained in the dataset of tweets are procedural or generic, but besides those, it often emerges that Twitter users expressed their concern with respect to the provisions of this law. Supportive attitudes seem to be less frequent. This paper also advocates that findings from inductive studies on Twitter data should be interpreted with caution, since the nature of tweets might not be adequate for drawing far-reaching generalisations.</p>
      </abstract>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>-</title>
      <p>In the last decade, Internet has revolutionized
human communication and interaction. And among
all forms of digitally-mediated communication,
social media stand out as one of the most
effective. As Boulianne (2017) points out, the effects
of social media depend on their nature of use (e.g.
source of information;
one-to-one/one-tomany/many-to-many communication; networking
and relationship-building; expression of opinions;
etc.).</p>
      <p>Nowadays, potentially everyone with a
computer or a mobile device having access to the
internet can write and share contents which may be
viewed and debated immediately by other people.
The impact of a social media post may be huge,
and unlike other prior forms of communication, it
can easily cross borders in just a few seconds. In
fact, social media make things happen faster than
ever before. For instance, Facebook and Twitter
were crucial in allowing the Arab uprisings or the
Romanian anti-corruption protests to happen
more efficiently and on a larger scale.
2</p>
    </sec>
    <sec id="sec-2">
      <title>Tweets and politics</title>
      <p>
        Besides their essential role in information
dissemination, networking, and people mobilization,
social media are also important indicators and
predictors of their users’ opinions, sentiments and
attitudes. In fact, various studies have explored
people’s reactions towards social, economic, and
political issues, by analysing social media posts
        <xref ref-type="bibr" rid="ref10 ref19 ref23 ref6">(e.g.
Burnap et al., 2014; Gaspar et al., 2016; Nesi et
al., 2018)</xref>
        , especially tweets, since they are easily
retrievable by means of APIs.
      </p>
      <p>
        With over 6,000 tweets posted every second,
corresponding to roughly 350,000 per minute, 500
million per day, and around 200 billion per year,
Twitter has become one of the main tools of
communication worldwide
        <xref ref-type="bibr" rid="ref13">(Internet Live Stats, 2019)</xref>
        .
The number of tweets written daily seems to be
correlated to things happening in the real world,
and, as a matter of fact, it was shown that
important events generate high number of tweets
        <xref ref-type="bibr" rid="ref18">(cf.
Hughes and Palen, 2009)</xref>
        , something that is
generally reflected also on the Twitter “trends”.
Based on
        <xref ref-type="bibr" rid="ref12">Hootsuite’s (2019</xref>
        ) report, each month,
in Italy there are almost 2.5 million active users2
of Twitter, a datum that confirms the popularity of
this network among various layers of Italian
audience.
      </p>
      <p>
        This means that Twitter may represent an easily
exploitable opportunity for politicians in their
attempt to reinforce communication with potential
voters in what might be defined as a permanent
digitally-mediated electoral campaign.
Additionally, it has been suggested that Twitter could be
used to model and predict public opinion and
behaviour regarding political events, such as
electoral campaigns
        <xref ref-type="bibr" rid="ref15 ref7">(e.g. Coletto et al., 2015;
Kalampokis et al., 2017)</xref>
        . In fact, Ott (2017: 59)
claims that Twitter may be the ideal tool for the
afore-mentioned purposes since, it “privileges
discourse that is simple, impulsive, and uncivil.”
      </p>
      <p>
        While indeed tweets have been widely used to
analyse public opinion and political discussions in
all its forms, several methodological
considerations are dutiful. First of all, Twitter users do not
represent an optimal sample for public opinion or
voting population, especially due to their higher
than average level of education and political
sophistication, as well as a generally younger age
        <xref ref-type="bibr" rid="ref11">(cf. Gayo-Avello, 2013; Barberá et al., 2015)</xref>
        . As
a matter of fact, we believe it is more accurate to
define Twitter users as a potential share of
electorate. Secondly, the language of tweets is
characterised by succinctness and sometimes
informality, colloquialism, irony, and susceptibility to
rumour, all of which are aspects that render the
results of large-scale analyses hard to interpret and
generalise.
3
      </p>
    </sec>
    <sec id="sec-3">
      <title>Aims and motivations</title>
      <p>
        Acknowledging all the limitations mentioned
above, this inductive exploratory study aims to
contribute to the growing body of literature
examining Twitter and its increasingly prominent role
in online communication by studying its
application in the context of political discourse. In
particular, the linguistic approach presented here is
providing insights into tweets regarding the
discussion and the approval of the annual Italian
budget law (in Italian “legge finanziaria” and/or
“legge di bilancio”). This law was also often
labelled as “the manoeuvre” (in Italian “la
manovra”) and “the people’s manoeuvre” (in
Italian “la manovra del popolo”) by its proponents –
in particular Movimento 5 Stelle (abbreviated
M5S) –, mainly due to some of its populist
provisions (e.g. the citizen's basic income and pension).
3 The full text of the annual Italian budget law
        <xref ref-type="bibr" rid="ref8">(Legge
30 dicembre 2018, n. 145 – Bilancio di previsione dello
Stato per l'anno finanziario 2019 e bilancio
pluriennale per il triennio 2019-2021)</xref>
        was published on the
Official Gazette of the Italian Republic (GU n.302
31By means of structural topic modelling
        <xref ref-type="bibr" rid="ref23">(cf.
Roberts et al., 2014)</xref>
        and descriptive analyses (i.e.
terminology extraction of multi-keywords and
word sketches), we are interested in grasping the
Twitter users’ attitudes towards the budget law in
a significant moment for the first populist
Government in the eurozone, namely the coalition formed
by Lega and M5S.
      </p>
      <p>This topic is worth studying since the two
parties displayed differences in economic, fiscal,
infrastructural, and social policies both in the
electoral campaign for the 2018 general elections as
well as during the first months of government. For
instance, Lega supported the flat taxation on
incomes, while M5S the citizen's basic income
(“reddito di cittadinanza” in Italian). However,
these measures, although slightly modified, as
well as the amendment to the 2011 pensions
reform (“quota 100” in Italian) were included in the
coalition agreement and subsequently in the draft
for the annual budget law. The bill also contained
various other economic and fiscal provisions (e.g.
taxes on digital services; new VAT rates; reducing
military expenses and the Italian contribution to
United Nations; new labour measures;
environmental incentives; etc.)3.</p>
      <p>We believe that the textual material contained
in tweets may be promising in providing hints on
how Twitter users – a fraction of the Italian voters
– reacted to the provisions of the budget law.
Linguistic insights into tweets might be able to guide
us in understanding whether the so-called
“manovra del popolo” was perceived by Twitter
user as representing indeed the people’s interest.
4</p>
    </sec>
    <sec id="sec-4">
      <title>Data</title>
      <p>
        Although in the Western world there are three
mainstream social media networks (i.e. Facebook,
Instagram, and Twitter), in this paper we analyse
Twitter posts, primarily as a consequence of data
availability. Indeed, unlike other tools for social
media, Twitter APIs for R
        <xref ref-type="bibr" rid="ref21">(R Core Team, 2018)</xref>
        allow scholars to collect large quantities of tweets
and their related metadata in a rather effortless
way.
      </p>
      <p>
        Using the rtweet package
        <xref ref-type="bibr" rid="ref17">(Kearney, 2019)</xref>
        for
R and Twitter’s developer account, we collected a
dataset of 167,259 Twitter posts, for a total of 6.5
million tokens, consisting in tweets and retweets
12-2018 - Suppl. Ordinario n. 62) and it is available
online at this webpage:
https://www.gazzettaufficiale.it/atto/stampa/serie_generale/originario
(accessed on the 1st of June 2019).
related to the Italian budget law. Moreover, we
extracted 88 metadata describing the tweet (i.e.
character length, device used, number of retweets,
etc.) and the user (i.e. username, location, gender,
etc.). In order to capture the most important
phases of the Twitter discussion about the annual
budget law and considering the one-week rate
limit for tweets extraction imposed by the
Standard Search API4, the data were collected weekly
from the 27th of November 2018 through the 8th of
January 2019, for a total of 43 consecutive days.
The hashtags used as keywords in the queries
represented all the names given to the budget bill by
Italian political actors, the press, and the public
opinion: “#leggedibilancio”, “#leggefinanziaria”,
“#manovra”, “#manovradibilancio”,
“#manovraeconomica”, “#manovradelpopolo”, and
“#manovrafinanziaria”. This guaranteed a large
coverage of Twitter users and tweet typologies.
Some of the afore-mentioned hashtags (e.g.
“#manovra”, “#manovradelpopolo”) were also
trending at the end of December.
      </p>
      <p>To avoid duplicates, we discarded all retweets
and all posts that contained quotes of other tweets.
The removal process was obtained by filtering the
dataset, thus selecting only tweets whose values
for “is_retweet” and “is_quote” corresponded to
“FALSE”. Duplicates other than retweets and
quotes were removed with R’s base functions
duplicated – which identified duplicated tweets –
and unique – which extracted unique tweets. Since
the aim of this study is to uncover the reactions of
the Italian voters active on Twitter, we removed
the tweets written by political actors. To do so, we
defined a list containing the Twitter usernames of
the members of the Italian Parliament, as well as
those of the official national and local party
profiles; this list was used to automatically filter and
remove tweets published by the unwanted
profiles. We decided to keep tweets from news
agencies, online newspapers, and television channels,
since they could represent vectors of information
exchange regarding the topic analysed in this
study. The final dataset contained 20,891 tweets.</p>
      <p>Tokens 701,986</p>
      <sec id="sec-4-1">
        <title>Words</title>
      </sec>
      <sec id="sec-4-2">
        <title>Types</title>
        <p>414,803
75,485</p>
        <p>Lemmas 31,947
Table 1: Dataset statistics.
4.1</p>
        <sec id="sec-4-2-1">
          <title>Pre-processing</title>
          <p>
            Since the tweets and their metadata would have
been used for lexical analyses and structural topic
modelling5, we performed several pre-processing
steps: defining a “stop words” list for Italian
consisting of roughly 1,000 lexically empty or
uninformative words (i.e. prepositions, conjunctions,
auxiliary verbs, etc.); uniformizing, normalising
and cleaning the texts with various corpus
processing functions available on the R packages
quanteda
            <xref ref-type="bibr" rid="ref2">(Benoit et al., 2018)</xref>
            , tm
            <xref ref-type="bibr" rid="ref9">(Feinerer,
Hornik, and Meyer, 2008)</xref>
            , and qdapRegex
            <xref ref-type="bibr" rid="ref22">(Rinker, 2017)</xref>
            . Hashtags at the beginning and
inside the tweet sentences were kept and
decomposed into words (i.e. from “#trasportipubblici”
to “trasporti pubblici”), while those after the final
point were removed, since most of the times they
represented one of the keywords used for
extracting tweets. Numbers, punctuation, sequences
made up of a single character, and excessive white
spaces were removed as well. In order to further
use temporal metadata as a covariate for the
topical prevalence, the “created_at” metadatum was
divided it into date and hour.
5
          </p>
        </sec>
      </sec>
    </sec>
    <sec id="sec-5">
      <title>Analyses and results</title>
      <p>
        As a result of the ever-growing interest and
availability of text data – often unstructured –, various
statistical and machine-assisted approaches for
the analysis of textual material have been
proposed. In this paper we are employing the
Structural Topic Model (STM) – a generative model of
word counts –
        <xref ref-type="bibr" rid="ref23">(cf. Roberts et al., 2014)</xref>
        in R to
discover topics from tweets on the annual Italian
budget bill and to estimate their relationship to
temporal metadata.
      </p>
      <p>
        Similarly to Latent Dirichlet Allocation
        <xref ref-type="bibr" rid="ref4">(cf.
Blei, Ng, and Jordan, 2003)</xref>
        and Correlated Topic
Model
        <xref ref-type="bibr" rid="ref3">(cf. Blei and Lafferty, 2007)</xref>
        , in the STM
approach, a topic represents a mixture over words
where each word has a probability of pertaining to
a topic, whilst a document is a mixture over
topics, therefore a specific document can consist of
various topics. The sum of the topic proportions
across topics for a specific document as well as
the sum of word probabilities for a given topic
both qual to 1. The main innovation of STM is the
possibility to model topical prevalence and topical
content6 as a function of metadata. Here we are
4 A description of the Standard Search API for Twitter
is available at this webpage:
https://developer.twitter.com/en/docs/tweets/search/api-reference/getsearch-tweets.html (accessed on the 1st of June 2019).
5 Considering the scope of this paper and the analyses
proposed, emoticons and emojis were left out.
6 The topical prevalence shows the frequency with
which a specific topic is discussed, while the topical
using the date covariate to explain topical
prevalence over time.
5.1
      </p>
      <sec id="sec-5-1">
        <title>Topics</title>
        <p>
          After having employed the STM’s searchK
function to perform several tests, such as held-out
likelihood and residual analysis, the ideal number of
topics seemed to be between 10 and 14.
Additionally, STM gave the possibility to set the type of
initialization, so here the spectral one was chosen,
since previous studies had proven its stability and
consistence
          <xref ref-type="bibr" rid="ref10 ref24 ref8">(cf. Roberts, Stewart, and Tingley,
2016)</xref>
          . All results presented in this paragraph are
based on a K of 10. The date of the tweet was used
as a prevalence covariate; as a word profile we
opted for the highest probability. We did not use
the stemming function on STM since it did not
perform well on Italian.
        </p>
        <p>
          Figure 1 in Appendix shows the topics related
to the annual Italian budget law as they emerged
from the analysis of tweets. Each topic was further
classified into one category (i.e. EU &amp;
Confidence, Main Measures, Criticism &amp; Concern,
Government vs. Opposition, Procedures –
Generic, Support). This classification was based on
the correlations obtained from a hierarchical
clustering representation performed with the plot
function of the stmCorrViz package
          <xref ref-type="bibr" rid="ref8">(Coppola et
al., 2016)</xref>
          , on the review of the most
characterising words, and on the examination of the most
exemplar documents, namely the tweets that had the
highest proportion of words associated with the
topic.
        </p>
        <p>
          Although we do not claim to model public
opinion from tweets, interestingly, the topics
managed to echo various issues regarding the
budget law. Judging by the expected topic
proportions, one could order the most prevalent topics as
follows: Topics 9, 8, and 3 (sum of topic
proportions: 0.29) reflect disapproval and doubts
towards the provisions of the budget law; Topics 1
and 7 (sum of topic proportions: 0.22) describe the
difficult negotiation with the European Union
(EU) and the threat of an infringement procedure;
Topics 10 and 2 (sum of topic proportions: 0.19)
depict the main measures contained in the budget
bill; Topic 6 (topic proportion: 0.13) illustrates the
support to the budget bill and to the Government;
Topic 5 (topic proportion: 0.11) refers to the
procedures regarding the discussion, the vote, and the
approval of the budget law; and Topic 4 (topic
proportion: 0.06) reveals the conflict between the
content represents the words used to discuss about that
topic
          <xref ref-type="bibr" rid="ref23">(cf. Roberts et al., 2014: 1068)</xref>
          .
        </p>
        <p>Government and the oppositions on the provisions
of the law.</p>
        <p>After having calculated the estimated effects of
the temporal covariate on topical prevalence, a
plot displaying this variation was created. Figure
2 in Appendix shows how the afore-mentioned
topics varied over the 43 days considered. Topics
are ordered as a function of their expected
proportions.</p>
        <p>Firstly, there emerged that the variation was not
particularly strong, except for some topics. For
instance, Topic 9 had a peak at the end of
December/the beginning of January, suggesting that
Twitter users might have written tweets of
concern soon after the approval of the annual Italian
budget law. On the other hand, Topic 6, which
contained mostly tweets of support towards the
measures of the budget bill seemed to be prevalent
primarily at the end of November and in
mid-December. The procedural topic was generally
prevalent at the end of December, a timeframe
corresponding to the vote and approval of the law. The
two topics summarising the negotiations with the
EU, the confidence, and the possible infringement
procedure were pervasive during the entire period
considered, with some peaks in early- and
midDecember. Topic 4 that regarded the
disagreement between the Government and the opposition
was constant over time, and so were the topics
delineating the main measures of the law.
5.2</p>
      </sec>
      <sec id="sec-5-2">
        <title>Descriptive lexical analyses</title>
        <p>
          We were also interested in performing descriptive
lexical analyses on tweets. First of all, with the
terminology extraction tool on Sketch Engine
          <xref ref-type="bibr" rid="ref16">(Kilgarriff et al., 2014)</xref>
          we obtained
multi-keywords – able to convey more insights than single
words on the issues examined – that appear more
frequently in our dataset than in the reference
corpus
          <xref ref-type="bibr" rid="ref24 ref8">(i.e. Italian Web 2016 – itTenTen16, cf.
Jakubíček et al., 2013, for TenTen corpora)</xref>
          . If we
exclude the hashtags used as keywords for tweets
extraction, these are the 30 most representative
syntagmas in our dataset:
        </p>
        <p>Syntagma
reddito di
cittadi</p>
        <p>nanza
procedura di
infra</p>
        <p>zione
clausole di
salvaguardia</p>
      </sec>
      <sec id="sec-5-3">
        <title>Translation into English</title>
        <p>the citizen’s basic</p>
        <p>income
infringement
pro</p>
        <p>cedure
safeguard clauses
voto di fiducia
blocco assunzioni
professioni
sanitarie senza titolo</p>
        <p>flat tax
commissione
bilan</p>
        <p>cio
gilet azzurri
taglio pensioni
scatoletta di tonno
governi precedenti
pensioni minime
scatola chiusa</p>
        <p>nuove tasse
promesse elettorali</p>
        <p>fasce deboli
deficit strutturale
accordo tecnico
braccio di ferro
appalti senza gara</p>
        <p>assurdità totale
terrorismo
media</p>
        <p>tico
auto inquinanti
più tasse
confidence vote</p>
        <p>hiring freeze
health professions
without a degree</p>
        <p>flat tax
budget committee</p>
        <p>blue vests
pension cuts</p>
        <p>tuna can
previous
govern</p>
        <p>ments
minimum pensions
black box
new taxes
campaign promises
vulnerable citizens
structural deficit
technical
arrange</p>
        <p>ment
trial of strength
no-bid contracts
total nonsense
media terrorism
polluting cars</p>
        <p>more taxes
sovereignist
govgoverno sovranista ernment
manovra contro il manoeuvre against
popolo the people
false promesse false promises
IVA sui tartufi VAT for truffles
popolo italiano Italian people
Table 2: The most representative syntagmas in
the dataset.</p>
        <p>It is clear that various multi-word expressions
referred to procedural aspects, such as those
reflecting the vote and the approval of the budget
law (e.g. “confidence vote”), while others were
used to list its measures, especially fiscal and
economic policies (e.g. “the citizen’s basic income”,
“flat tax”, etc.). Nevertheless, various syntagmas
seemed to express doubts with respect to the
provisions of this law. In fact, often, the words chosen
by many Twitter users to express their criticism
were rather strong (e.g. “total nonsense”, “black
box”, “sovereignist government”, etc.).</p>
        <p>These concerns and rather negative reactions to
the budget bill were reflected also in the word
sketches (i.e. visual representations of
collocations and word combinations obtained on Sketch
Engine) for the words “manovra” and “legge”.
Generally, three different scenarios are
distinguishable.</p>
        <p>First of all, there were several neutral verbs,
nouns, and modifiers associated to the budget law,
most of which regarding its procedural aspects.
The most frequent (i.e. frequency ≥ 10.81 per
million) are listed below:</p>
      </sec>
      <sec id="sec-5-4">
        <title>Translation into</title>
        <p>Word/Syntagma English
scrivere write
cambiare change
modificare modify
discutere discuss
approvare approve
contenere contain
prevedere consist
varare launch
votare vote
passare pass
riscrivere rewrite
promulgare promulgate
gialloverde yellow-green
economica economic
finanziaria financial
populista populist
discussione discussion
commissione commission
bilancio budget</p>
        <p>Table 3: Neutral associations.</p>
        <p>Next, some positive evaluations of the budget
law emerged. The most frequent (i.e. frequency ≥
10.81 per million) are listed below:</p>
        <p>Word/Syntagma TraEnnsglalitsiohn into
favorire (l’innova- favour
(innovazione) tion)
grande big
buona good
bella beautiful
significativa significant
del popolo of the people
del cambiamento of the change
per i cittadini for the citizens
per la crescita for the growth</p>
        <p>Table 4: Positive associations.</p>
        <p>Nonetheless, several word associations seemed
to suggest negative reactions to the budget law.
The most frequent (i.e. frequency ≥ 10.81 per
million) are shown below:</p>
      </sec>
      <sec id="sec-5-5">
        <title>Word/Syntagma</title>
        <p>recessiva
piena di errori</p>
      </sec>
      <sec id="sec-5-6">
        <title>Translation into English</title>
        <p>recessive
full of errors
This paper explored the Twitter users’ reactions to
the annual Italian budget bill. STM outputs and
descriptive lexical analyses showed that tweets
concerned various aspects associated to the object
of this study. Apart from talking about procedural
and generic issues, users expressed their doubts
and disapproval with respect to the measures of
the budget law. Generally, tweets supporting this
law were less frequent. The findings of this study,
although preliminary, might be seen as indicators
of what subsequently turned out to be a failure for
the first Conte government. Still, as reiterated
throughout the paper, the results might not reflect
the real attitudes of the Italian voting population,
since Twitter users tend to be younger and to have
an above the average level of education and
political sophistication (cf. Barberá et al., 2015).
Moreover, tweets, by nature, might not be suitable
for drawing steady generalizations, even if the
prospects they offer for content and discourse
analysis are indeed significant. Further research
on this topic might include the investigation of
Twitter user’s reactions by means of sentiment
analysis.</p>
      </sec>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          <string-name>
            <given-names>Pablo</given-names>
            <surname>Barberá</surname>
          </string-name>
          , John T. Jost, Jonathan Nagler, Joshua A.
          <string-name>
            <surname>Tucker</surname>
          </string-name>
          , and Richard Bonneau.
          <year>2015</year>
          .
          <article-title>Tweeting from Left to Right</article-title>
          . Psychological Science,
          <volume>26</volume>
          (
          <issue>10</issue>
          ):
          <fpage>1531</fpage>
          -
          <lpage>1542</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          <string-name>
            <given-names>Kenneth</given-names>
            <surname>Benoit</surname>
          </string-name>
          , Kohei Watanabe,
          <string-name>
            <given-names>Haiyan</given-names>
            <surname>Wang</surname>
          </string-name>
          , Paul Nulty, Adam Obeng, Stefan Müller, and
          <string-name>
            <given-names>Akitaka</given-names>
            <surname>Matsuo</surname>
          </string-name>
          .
          <year>2018</year>
          .
          <article-title>quanteda: An R package for the quantitative analysis of textual data</article-title>
          .
          <source>Journal of Open Source Software</source>
          ,
          <volume>3</volume>
          (
          <issue>30</issue>
          ):
          <fpage>774</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          <string-name>
            <given-names>David M.</given-names>
            <surname>Blei</surname>
          </string-name>
          and
          <string-name>
            <given-names>John D.</given-names>
            <surname>Lafferty</surname>
          </string-name>
          .
          <year>2007</year>
          .
          <article-title>A correlated topic model of Science</article-title>
          .
          <source>The Annals of Applied Statistics</source>
          ,
          <volume>1</volume>
          (
          <issue>1</issue>
          ):
          <fpage>17</fpage>
          -
          <lpage>35</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          <string-name>
            <given-names>David M.</given-names>
            <surname>Blei</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Andrew Y.</given-names>
            <surname>Ng</surname>
          </string-name>
          , and
          <string-name>
            <given-names>Michael I.</given-names>
            <surname>Jordan</surname>
          </string-name>
          .
          <year>2003</year>
          .
          <article-title>Latent Dirichlet Allocation</article-title>
          .
          <source>Journal of Machine Learning Research</source>
          ,
          <volume>3</volume>
          (
          <issue>3</issue>
          ):
          <fpage>993</fpage>
          -
          <lpage>1022</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          <string-name>
            <given-names>Shelley</given-names>
            <surname>Boulianne</surname>
          </string-name>
          .
          <year>2017</year>
          .
          <article-title>Revolution in the making? Social media effects across the globe</article-title>
          .
          <source>Information, Communication &amp; Society</source>
          ,
          <volume>22</volume>
          (
          <issue>1</issue>
          ):
          <fpage>39</fpage>
          -
          <lpage>54</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          <string-name>
            <given-names>Pete</given-names>
            <surname>Burnap</surname>
          </string-name>
          ,
          <string-name>
            <surname>Matthew L. Williams</surname>
            , Luke Sloan, Omer Rana,
            <given-names>William</given-names>
          </string-name>
          <string-name>
            <surname>Housley</surname>
            , Adam Edwards, Vincent Knight, Rob Procter, and
            <given-names>Alex</given-names>
          </string-name>
          <string-name>
            <surname>Voss</surname>
          </string-name>
          .
          <year>2014</year>
          .
          <article-title>Tweeting the terror: modelling the social media reaction to the Woolwich terrorist attack</article-title>
          .
          <source>Social Network Analysis and Mining</source>
          ,
          <volume>4</volume>
          (
          <issue>206</issue>
          ):
          <fpage>1</fpage>
          -
          <lpage>14</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          <string-name>
            <given-names>Mauro</given-names>
            <surname>Coletto</surname>
          </string-name>
          , Claudio Lucchese, Salvatore Orlando, and
          <string-name>
            <given-names>Raffaele</given-names>
            <surname>Perego</surname>
          </string-name>
          .
          <year>2015</year>
          . Italian Information Retrieval Workshop - IIR
          <year>2015</year>
          .
          <source>In Proceedings of the 6th Italian Information Retrieval Workshop</source>
          , Cagliari. CEUR Workshop Proceedings.
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          <string-name>
            <given-names>Antonio</given-names>
            <surname>Coppola</surname>
          </string-name>
          , Margaret E. Roberts,
          <string-name>
            <surname>Brandon M. Stewart</surname>
            , and
            <given-names>Dustin</given-names>
          </string-name>
          <string-name>
            <surname>Tingley</surname>
          </string-name>
          .
          <year>2016</year>
          .
          <article-title>stmCorrViz: A Tool for Structural Topic Model Visualizations</article-title>
          .
          <source>R package version 1</source>
          .3. Retrieved from https://cran.rproject.org/web/packages/stmCorrViz/index.html
          <source>/ (accessed on the 1st of June</source>
          <year>2019</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          <string-name>
            <given-names>Ingo</given-names>
            <surname>Feinerer</surname>
          </string-name>
          , Kurt Hornik, and David Meyer.
          <year>2008</year>
          .
          <article-title>Text Mining Infrastructure in R</article-title>
          .
          <source>Journal of Statistical Software</source>
          ,
          <volume>25</volume>
          (
          <issue>5</issue>
          ):
          <fpage>1</fpage>
          -
          <lpage>54</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          <string-name>
            <given-names>Rui</given-names>
            <surname>Gaspar</surname>
          </string-name>
          , Cláudia Pedro, Panos Panagiotopoulos, and
          <string-name>
            <given-names>Beate</given-names>
            <surname>Seibt</surname>
          </string-name>
          .
          <year>2016</year>
          .
          <article-title>Beyond positive or negative: Qualitative sentiment analysis of social media reactions to unexpected stressful events</article-title>
          .
          <source>Computers in Human Behavior</source>
          ,
          <volume>56</volume>
          :
          <fpage>179</fpage>
          -
          <lpage>191</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          <string-name>
            <surname>Daniel</surname>
          </string-name>
          Gayo-Avello.
          <year>2013</year>
          .
          <article-title>A Meta-Analysis of Stateof-the-Art Electoral Prediction from Twitter Data</article-title>
          . Social Science Computer Review,
          <volume>31</volume>
          (
          <issue>6</issue>
          ):
          <fpage>649</fpage>
          -
          <lpage>679</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          <string-name>
            <given-names>Hootsuite</given-names>
            <surname>Media Inc</surname>
          </string-name>
          .
          <year>2019</year>
          . Digital in
          <year>2019</year>
          . Retrieved from https://hootsuite.com/it/risorse/digital-in2019
          <source>-italy (accessed on the 1st of June</source>
          <year>2019</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          <string-name>
            <given-names>Internet</given-names>
            <surname>Live Stats</surname>
          </string-name>
          .
          <year>2019</year>
          . Twitter Usage Statistics. Retrieved from https://www.internetlivestats.com/twitter-statistics
          <source>/ (accessed on the 1st of June</source>
          <year>2019</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          <string-name>
            <given-names>Miloš</given-names>
            <surname>Jakubíček</surname>
          </string-name>
          , Adam Kilgarriff, Vojtěch Kovář, Pavel Rychly, and
          <string-name>
            <given-names>Vít</given-names>
            <surname>Suchomel</surname>
          </string-name>
          .
          <year>2013</year>
          .
          <article-title>The TenTen Corpus Family</article-title>
          .
          <source>In 7th International Corpus Linguistics Conference CL 2013. Lancaster</source>
          ,
          <volume>125</volume>
          -
          <fpage>127</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          <string-name>
            <given-names>Evangelos</given-names>
            <surname>Kalampokis</surname>
          </string-name>
          , Areti Karamanou, Efthimios Tambouris, and
          <string-name>
            <given-names>Konstantinos</given-names>
            <surname>Tarabanis</surname>
          </string-name>
          .
          <year>2017</year>
          .
          <article-title>On Predicting Election Results using Twitter and Linked Open Data: The Case of the UK 2010 Election</article-title>
          .
          <source>Journal of Universal Computer Science</source>
          ,
          <volume>23</volume>
          (
          <issue>3</issue>
          ):
          <fpage>280</fpage>
          -
          <lpage>303</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          <string-name>
            <given-names>Adam</given-names>
            <surname>Kilgarriff</surname>
          </string-name>
          , Vít Baisa, Jan Bušta, Miloš Jakubíček, Vojtěch Kovář, Jan Michelfeit, Pavel Rychlý, and
          <string-name>
            <given-names>Vít</given-names>
            <surname>Suchomel</surname>
          </string-name>
          .
          <year>2014</year>
          .
          <article-title>The Sketch Engine: ten years on</article-title>
          .
          <source>Lexicography</source>
          ,
          <volume>1</volume>
          (
          <issue>1</issue>
          ):
          <fpage>7</fpage>
          -
          <lpage>36</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          <string-name>
            <given-names>Michael W.</given-names>
            <surname>Kearney</surname>
          </string-name>
          .
          <year>2019</year>
          .
          <article-title>rtweet: Collecting Twitter Data</article-title>
          .
          <source>R package version 0.6</source>
          .9 Retrieved from https://cran.r-project.
          <source>org/package=rtweet (accessed on the 1st of June</source>
          <year>2019</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          <string-name>
            <given-names>Amanda</given-names>
            <surname>Lee</surname>
          </string-name>
          Hughes and
          <string-name>
            <given-names>Leysia</given-names>
            <surname>Palen</surname>
          </string-name>
          .
          <year>2009</year>
          .
          <article-title>Twitter adoption and use in mass convergence and emergency events</article-title>
          .
          <source>International Journal of Emergency Management</source>
          ,
          <volume>6</volume>
          (
          <issue>3</issue>
          /4):
          <fpage>248</fpage>
          -
          <lpage>260</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          <string-name>
            <given-names>Paolo</given-names>
            <surname>Nesi</surname>
          </string-name>
          , Gianni Pantaleo, Irene Paoli, and
          <string-name>
            <given-names>Imad</given-names>
            <surname>Zaza</surname>
          </string-name>
          .
          <year>2018</year>
          .
          <article-title>Assessing the reTweet proneness of tweets: predictive models for retweeting</article-title>
          .
          <source>Multimedia Tools and Applications</source>
          ,
          <volume>77</volume>
          (
          <issue>20</issue>
          ):
          <fpage>26371</fpage>
          -
          <lpage>26396</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          <string-name>
            <given-names>Brian L.</given-names>
            <surname>Ott</surname>
          </string-name>
          .
          <year>2017</year>
          .
          <article-title>The age of Twitter: Donald J. Trump and the politics of debasement</article-title>
          .
          <source>Critical Studies in Media Communication</source>
          ,
          <volume>34</volume>
          (
          <issue>1</issue>
          ):
          <fpage>59</fpage>
          -
          <lpage>68</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          <string-name>
            <given-names>R Core</given-names>
            <surname>Team</surname>
          </string-name>
          .
          <year>2018</year>
          .
          <article-title>R: A language and environment for statistical computing</article-title>
          .
          <source>R Foundation for Statistical Computing</source>
          , Vienna, Austria. Retrieved from http://www.R-project.
          <source>org/ (accessed on the 1st of June</source>
          <year>2019</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          <string-name>
            <given-names>Tyler W.</given-names>
            <surname>Rinker</surname>
          </string-name>
          .
          <year>2017</year>
          . qdapRegex: Regular Expression Removal, Extraction, and
          <string-name>
            <given-names>Replacement</given-names>
            <surname>Tools</surname>
          </string-name>
          .
          <source>R package version 0.7</source>
          .2. University at Buffalo. Buffalo, New York. Retrieved from http://github.com/trinker/qdapRegex/ (
          <source>accessed on the 1st of June</source>
          <year>2019</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          <string-name>
            <given-names>Margaret E.</given-names>
            <surname>Roberts</surname>
          </string-name>
          ,
          <string-name>
            <surname>Brandon M. Stewart</surname>
            , Dustin Tingley, Christopher Lucas, Jetson Leder-Luis, Shana Kushner Gadarian, Bethany Albertson, and
            <given-names>David G.</given-names>
          </string-name>
          <string-name>
            <surname>Rand</surname>
          </string-name>
          .
          <year>2014</year>
          .
          <article-title>Structural Topic Models for OpenEnded Survey Responses</article-title>
          .
          <source>American Journal of Political Science</source>
          ,
          <volume>58</volume>
          (
          <issue>4</issue>
          ):
          <fpage>1064</fpage>
          -
          <lpage>1082</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          <string-name>
            <given-names>Margaret E.</given-names>
            <surname>Roberts</surname>
          </string-name>
          ,
          <string-name>
            <surname>Brandon M. Stewart</surname>
            , and
            <given-names>Dustin</given-names>
          </string-name>
          <string-name>
            <surname>Tingley</surname>
          </string-name>
          .
          <year>2016</year>
          .
          <article-title>Navigating the Local Modes of Big Data: The Case of Topic Models</article-title>
          . In R. Michael Alvarez (editor),
          <source>Computational Social Science: Discovery and Prediction (Analytical Methods for Social Research)</source>
          ,
          <fpage>51</fpage>
          -
          <lpage>97</lpage>
          . Cambridge University Press., Cambridge.
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>