<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Felicitta: Visualizing and Estimating Happiness in Italian Cities from Geotagged Tweets</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Leonardo Allisio</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Valeria Mussa</string-name>
          <email>valeria.mussag@studenti.unito.it</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Cristina Bosco</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Viviana Patti</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Giancarlo Ru o</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Universita degli Studi di Torino Dipartimento di Informatica c.</institution>
          <addr-line>so Svizzera 185, I-10149 Torino</addr-line>
          ,
          <country country="IT">Italy</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>Felicitta1 is an online platform for estimating happiness in the Italian cities, which uses Twitter as data source and combines sentiment analysis and visualization techniques in order to provide users with an interactive interface for data exploration. In particular, Felicitta daily analyzes Twitter posts and exploits temporal and geo-spatial information related to Tweets in order to easy the summarization of sentiment analysis outcomes and the exploration of the Twitter data. By interactive maps it provides users with the possibility to have a comprehensive overview of the sentiment analysis results about the main Italian cities, and with the opportunity to zoom-in to a speci c region to visualize a ne-grained map of the city or district as well as the location of the individual sentiment-labeled Tweets. The platform allow users to tune their view on such huge amount of information and to interactively reduce the inherent complexity, possibly providing an hint for nding meaningful patterns, and correlations between moods and events.</p>
      </abstract>
      <kwd-group>
        <kwd>Data Analysis and Visualization</kwd>
        <kwd>Twitter</kwd>
        <kwd>Sentiment Analysis</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>-</title>
      <p>The huge amount of information streaming from online social networking and
micro-blogging platforms such as Twitter, are increasingly attracting the
attention of many kinds of researchers and practitioners, such as sociologists,
psychologists, communication and political scientists, as well as data journalists
and computational linguistics scholars. From di erent perspectives, each
observer looks for relationships from massively user generated data, in order to get
insights that can lead to new conjectures, correlations, and causalities.</p>
      <p>Even if the debate on how the social networking users generated content can
be representative of the human behavior in everyday life is still open, a
computational framework that analyzes and visualizes information at customizing scales
1 The name Felicitta is a fusion of two Italian words, \felice" (happy) and \citta"
(which corresponds to both city and town).
of summarization is a valuable tool for experts with di erent backgrounds that
do not want to deal directly with raw data or abstract models; in fact,
statistical tools, machine learning techniques, large-scale network analysis and natural
language modeling are hard to be applied by many skilled sociologists and
communication scientists. On the other hand, a lot of quantitative research has been
conducted on data, but the connection with theories that would qualitatively
explain the observed and modeled phenomena is often missing.</p>
      <p>In this paper, we introduce Felicitta, an online platform that allows the user
to explore the result of sentiment analysis performed over geotagged Tweets. The
contribution is two-fold. First, we propose a fully implemented visualization
system to estimate the level of happiness in a given geographical area based on
geotagged Tweets, that has been engineered in a modular way, and can overlay
di erent analysis engines. Second, we describe a particular instantiation of the
system, where a sentiment analysis engine for detecting Italian Tweets'
sentiment polarity has been developed and employed in order to estimate happiness
in Italian cities. Visualization techniques are adopted to support researchers
and practitioners to explore the data about happiness in Italian cities. Maps,
plots, tag clouds and other charts can be interactively requested on demand
for giving more contextual and quantitative details. The paper is organized as
follows: Section 2 contains a brief overview of related work. Section 3 describes
the modules of the computational framework devoted to Tweets' retrieval and
sentiment analysis. The front-end module of our online service, which provides
summarization and visualization of user sentiments in Italian cities, is outlined
in Section 4.Brief conclusions end the paper.
2</p>
    </sec>
    <sec id="sec-2">
      <title>Related Work</title>
      <p>Relevant contributions for the issues addressed in Felicitta can be found in a
wide range of disciplines, ranging from computational linguistics and sociology
to advanced interfaces for big data exploration.</p>
      <p>
        The investigation of correlations between Twitter geotagged data (expressing
in real-time sentiment of individuals) and emotional, geographic, demographic,
and health characteristics has been matter on an interesting recent work [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ],
where a team of researchers analyzed ten million Tweets to map happiness in
the United States and to create a sort of geography of happiness. The
happiness score, computed by applying sentiment analysis techniques, is measuring
something di erent but perhaps complementary to traditional survey-based
techniques. Based on the Tweet analysis happiness maps of the United States are
produced, but they are not web accessible neither inspectable in an interactive
way. On the same line, an ongoing study is focussing on the geography of hate,
by analyzing geotagged hateful Tweets in the United States. The resulting hate
heat map shows where racist, homophobic tweets come from, and is available
on-line: http://users.humboldt.edu/mstephens/hate/hate_map.html.
      </p>
      <p>In Italy, the issue of measuring the happiness in the cities by analyzing Italian
Tweets has been addressed recently in the context of the Voices from the Blog
project2 and lead to the development of an iPhone/iPad app, called iHappy. In
this application, results of the sentiment analysis on Tweets are summarized and
visualized to users mainly by static maps of happiness, by showing happiness
indicators related to Italian cities, provinces or region in a seven-day time window,
and by providing daily (or weekly) region or city happiness rankings.</p>
      <p>
        The main contribution of Felicitta w.r.t. to the above mentioned systems
is two-fold. On the one hand, the visualization component of Felicitta
(Section 4) provides introduces more zoom &amp; ltering tools and details-on-demand
capabilities, that support users in the activity of interactively inspecting
Tweetgenerated happiness maps. On the other hand, our platform has been engineered
to be modular w.r.t. Tweet-based analysis engines, and, in principle, can overlay
sentiment analyzers specialized for di erent languages and domains.The issue of
identifying key features of an information visualization tool has been matter of
a pioneering work of the information visualization researcher Ben Schneiderman
[
        <xref ref-type="bibr" rid="ref13">13</xref>
        ]. An information visualization tool should allow users \to gain an overview of
the data under study, provide zoom and ltering capabilities, item-level
detailson-demand, allow users to see relationships among items in a collection, and
extract target data about speci c subsets within the collection" [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ]. On this
line, recent works deal with the visualization of Twitter data [
        <xref ref-type="bibr" rid="ref11 ref6 ref7">6, 7, 11</xref>
        ], but
without a speci c interest on the estimation of happiness in given areas, which is the
focus of the present work.
      </p>
      <p>
        For what concerns the technical sentiment analysis task, comprehensive
surveys are available in literature [
        <xref ref-type="bibr" rid="ref16 ref3 ref5">3, 5, 16</xref>
        ]. Only very recently some works focussed
on analyzing the sentiment in Italian Tweets [
        <xref ref-type="bibr" rid="ref1 ref2 ref9">1, 2, 9</xref>
        ], let us mention, among
others, the work in [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ] which focusses on irony detection, or [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ], which addresses the
task of modeling political disa ection in Italy. The approach used in Felicitta is
lexicon-based. The evaluation of the polarity of a Tweet is based on the word's
polarity, and supported by state-of-the-art lexical and a ective resources, i.e.
MultiWordNet and WordNet-A ect [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ], as we will explain below.
3
      </p>
    </sec>
    <sec id="sec-3">
      <title>The Heart of Felicitta: the Sentiment Analyzer</title>
      <p>Felicitta aims at automatic mining and estimating happiness of people living in a
given location from the Tweets posted in that geographic area. The architecture
is sketched in Fig. 1. Geotagged information is retrieved from social media APIs.
Such data can be enriched and contextualized with other information accessible
from a wide range of data sources, such as o cial statistics produced to o er
services to citizens and policy makers. For example, ISTAT (Italian National
Institute of Statistics) is the main producer of socio demographic information,
and their data can be directly accessed. At the core of our architecture there is
the analyzer that receives as input the data gathered from di erent sources and
produces an estimation of the general sentiment at di erent geographical and
temporal scales. The analysis layer communicates with other services through
2 http://www.blogsvoices.unimi.it/
an API, keeping modules logically separated and easy to be modi ed or
substituted without a ecting other layers. On top of the architecture we have our
visualization framework that is presented in the next section.</p>
      <p>We focused on the main Italian towns, i.e. the 110 administrative Italian
centers. By exploiting Twitter's APIs, the system collects daily all the Tweets
freely downloadable (450,000) geolocated in these towns, and performs three
steps of analysis for each Tweet3 in order to classify it as positive or negative. At
the end, it aggregates the polarity of all the Tweets according to their geotagging
and thus evaluates the happiness of each town and region.</p>
      <p>The pipeline of Felicitta's analyzer includes four steps (Fig. 2). The rst one
is the collection of the Tweets to be analyzed: assuming that each town T can
be identi ed by latitude and longitude values, all the messages posted in a range
of 10 km from the center of T the previous day are daily collected, by applying
the Twitter's search function, and geolocated. In the second step the collected
Tweets are cleaned by deleting emoticons, links, mentions of other users and
redundant punctuation. Emoticons which are clearly expressing some emotion
are substituted by the more similar emotional words, in order to maintain their
a ective value. Mentions, usually preceded by `@', are instead substituted by
the word `user'. Table 1 shows some example.</p>
      <p>The third step consists in parsing the cleaned Tweets by Freeling4, an open
source tool for morpho-syntactic analysis of Italian and other languages,
developed at the University of Catalunya (Spain). In particular, the grammatical
category of each word is recognized allowing for the association with a lemma to
be searched in the a ective lexicon in the next step, e.g. the word \odio" (hate)
is recognized as a Verb and associated with the lemma \odiare" (to hate). The
3 In this paper, we capitalize the T in Tweet and Twitter as requested in the Twitter</p>
      <p>Trademark and Content Display Policy at https://twitter.com/logo.
4 http://nlp.lsi.upc.edu/freeling/
Regular Expression Interpretation Substitution
:[-]?[D]+ :D gioia (joy)
[:;][-]?[Pp]+ :P ;P ironia (irony)
[;T][_]+[;T] ;_; T_T tristezza (sadness)
&gt;[_]+&lt; &gt;_&lt; rabbia (anger)
(mw|MW)*[hH]*([aAeEiI]+[hH]+)+[aAeEiI]* ahah mwahah eheh risata (laugh)
recognition of lemmas is especially needed for morphologically rich languages
like Italian where the phenomenon of in ection, even if more notable in the case
of Verbs, a ects a large variety of grammatical categories where the agreement
is required in the most of cases5.</p>
      <p>
        In the fourth step the sentiment analysis is applied on Tweets. First, all the
content words of each Tweet, i.e. words carrying semantic content, and
therefore useful for the detection of the a ective meaning, are extracted. They are
Nouns, Verbs, Adverbs and Adjectives associated to lemmas in the previous
step. Second, for each extracted lemma a query is done on a lexical database in
order to nd its meaning(s) that can be related to some a ective concept. The
resources developed for Italian we exploited for this task are MultiWordNet6
and WordNet-A ect [
        <xref ref-type="bibr" rid="ref10 ref14">10, 14</xref>
        ]. The former is a lexicon built at the FBK
(Fondazione Bruno Kessler, Trento, Italy) for Italian and other languages, aligned
with the WordNet database developed for English at Princeton University [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ].
It is organized according to the grammatical category of words and for each
lemma it includes the meaning(s) that the lemma can assume. WordNet{A ect
is instead the a ective extension of WordNet domains, aligned with
MultiWord5 Think for instance to Adjectives and Determiners that agree with Nouns or Verbs
in participle (past and present) which agree with subject Nouns or ProNouns.
6 http://multiwordnet.fbk.eu/english/home.php
Net, that associates to some lemmas a ective concepts. If a lemma L occurs in
MultiWordNet, the query results in a list of one or more meanings:
      </p>
      <p>L =&lt; m1:::mn &gt;
Each mi is then searched in WordNet{A ect and when the association with an
a ective concept is found, an a ective evaluation is expressed using one polarity
label7 among 'negative' ( 1), 'neutral' (0), or 'positive' (+1):
p(mi) =
1=0= + 1
The polarity of the lemma is then described by the following formula:
8 1
p(L) = &lt; 0
: 1
if Pm2L p(m) &lt; 0
if Pm2L p(m) = 0
otherwise
On the one hand, p(L) is an indication of the prevailing polarity of the mi
associated by WordNet{A ect to L. Moreover, it should be observed that in
each Tweet for each L, also associated with several meanings, only one single
mi is realized.</p>
      <p>While the above described part of sentiment analysis is referred to single
words only, the last part of the process concerns each full Tweet T considered
as a bag of lemmas. The polarity of a Tweet T is described by the following
formula:</p>
      <p>8 1
p(T ) = &lt; 0
: 1
if PL&lt;2TPp(L)=m
if L2T p(L)=m &lt;
otherwise
where m is the number of lemmas of T , and is an empirical estimable small
value constant that may allow to extend the range of Tweets to be labeled as
neutral (in our platform, we set to a temporary value of 0, because we need a
more accurate evaluation in order to make such estimation). The polarity of a
city/region is de ned as the rate of positive Tweets geolocated in the city/region
w.r.t. the total amount of Tweets geolocated in that area.</p>
      <p>It should moreover be observed that because of the size of the WordNet{
A ect lexicon, which includes around 4,700 words, the a ective polarity can be
detected only in a limited part of the Tweets collected every day by Felicitta.
The results are based on around 35,000 out of the 450,000 daily collected Tweets.
7 For the purpose of the present work, exploiting the hierarchical organization of the
database, we have limited the granularity of the a ective concepts included in the
WordNet{A ect classifying with these three labels only a broad notion of polarity.
3.1</p>
      <sec id="sec-3-1">
        <title>An Example</title>
        <p>We conclude by showing the result of each step above described on a Tweet of
our corpus. In Fig. 3, rst, you can see the original Tweet (I HATE that horrible
place, I will be satis ed when it will be razed. &gt;_&lt; ). Then, it is shown the
substitution of the emoticon &gt;_&lt; in the pre{parsing step, and how the Freeling
parser analyzes and split the Tweet in columns in order to associate to each word
the lemma and the morpho-syntactic features. We can see the result of the query
on MultiWordNet and WordNet-A ect: two lemmas of the Tweet are detected as
a ective, that is \odiare" (to hate) and \rabbia" (rage), the former with a single
meaning and the latter with two meanings, all with negative polarity. Finally,
the polarity evaluation for the Tweet is reported.</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>Visualizing and Interacting with Estimated Happiness</title>
      <p>In this section we introduce the interactive user interface designed for
geotemporal visualizations of happiness in Twitter. The results of Felicitta are
displayed to the user according to di erent perspectives, as you can see at http:
//www.felicitta.net. The main views available to the user on the web site are
three: Citta (Towns), Regioni (Regions) and Top10.</p>
      <p>The view Citta (see Fig 4) is a map of Italy where the user can nd a round
marker for each town, which assumes di erent colors ranging from blue to red
to represent the a ective status, and di erent size to display the larger/smaller
amount of messages posted by the users of the town. Also an ordered score of all
the Italian towns is shown for each given day. By selecting a town on the map
the user can see more details about the evaluation expressed by the system and
also the Tweets posted there to test the reliability of the system evaluation.</p>
      <p>The view Regioni (see Fig 5) shows instead the a ective status of full regions
calculated on the basis of the results obtained for all the towns of the region
itself. This can be displayed in two di erent ways, a map of Italy where the
regions are colored according to their a ective status, as for towns, or in a score
of the regions from the happiest to the less happy. The last view, i.e. Top10,
displays the Tweets as geolocated on Italy. Also this view includes a map and
a score. The map shows all the Tweets positioned within the area where they
have been posted. By moving on the map, the user can zoom in to nd the
exact position of Tweets on the map and also to read them. Each Tweet is here
represented by a marker colored according to its detected a ective polarity (see
Fig. 6). Nevertheless, only posts which are associated with a precise geolocation
can be seen in the Top10 view, while several others, for which the geolocation is
only referred to a town, are exploited in the computation of the a ective polarity
of a geographic area, but cannot be seen in this view by the user.
4.1</p>
      <sec id="sec-4-1">
        <title>Looking for Relationships and Details on Demand</title>
        <p>
          According to the visualization features identi ed by Schneiderman in [
          <xref ref-type="bibr" rid="ref13">13</xref>
          ], our
interface includes several ways to expand other details, that may help the
observer to look for hidden relationships and correlations. If we focus on a given
city (for example: Turin), we can browse the Tweets that have been geotagged
there using the rightmost column displayed in the city map view (See Fig. 4).
However, we can further expand our search for information on a temporal
dimension. In fact, we can view the number of Tweets that have been analyzed in
a period of time (e.g., August 2013), as in Fig. 7.
        </p>
        <p>If we want to observe how the level of happiness varies in time, we can open
quarterly and daily based views. In the given example, and according to the
sample of Twitter users we analyzed, August has been the happiest summer
month in Turin (Fig. 8a). However, the distribution plotted on a daily basis
(Fig. 8b), shows that happiness does not exhibit a regular behavior, and we
have an happiest day (08/09) and a saddest one (24/09).</p>
        <p>We decided to display in our framework also tag-clouds, a visual
representation useful for quickly perceiving the most prominent terms involved in analyzed
Tweets. For example, in Fig. 9a and 9b we show some of the words used in Turin
during the happiest and saddest days of August 2013. Words are displayed with
di erent sizes according to the number of their occurrences in the Tweets posted
during those days. Quite interestingly, on 08/24, the soccer player Carlos Tevez
made his rst appearance with Juventus (one of the teams quartered in Turin),
scoring the winning goal and beating U.C. Sampdoria 1-0 in their opening match
of the 2013/14 season. It worths to be noted that Tevez wears the shirt number
10, i.e., \maglia numero 10" (see 9b). Maybe, the general bad mood re ected
in Tweets posted in Turin in that day can be explained by the disappointment
of the many citizens that do not support Juventus or by the anxiety (\ansia")
resulted by the long-awaited season opening. However, the given interpretation
is much beyond the scope of our tool, and the empirical co-occurrence of
particular words in charts and a corresponding polarity evaluation can be used only
to raise issues for further investigation.</p>
        <p>Fig. 6: Felicitta view Top10</p>
      </sec>
    </sec>
    <sec id="sec-5">
      <title>Conclusion and Future Work</title>
      <p>The paper describes an online platform for estimating happiness in the Italian
cities, which uses Twitter as data source and combines sentiment analysis and
advanced visualization techniques in order to provide users with an interactive
interface for the exploration of the resulted data.</p>
      <p>For what concerns the visualization module of Felicitta, we aim at improving
the browsing experience of the user as regards time-oriented information, by
allowing users to visualize data within the speci ed window or time period,
rather than within a single day as in the actual implementation. Moreover, we are
studying the embedding of tools that enable users to personalize (and export) the
views, the graphs and the statistics o ered by Felicitta, according to their needs
and goals, with the main aim to o er a richer support to sociologists, researchers,
(a) Quarterly happiness. Period: June{
September 2013.</p>
      <p>(b) Daily happiness. Month: August
2013. Happiest day: 08/09. Saddest
day: 08/24.
journalists, common users in detecting meaningful patterns, possible correlations
and so on. For instance, it could be interesting to have the possibility to compare
the results of Felicitta's estimation of happiness in a given district of a city and
in a given time period, with other information, e.g. quality of public services in
the area, events occurred in the area in the same time window, in order to give
users food for thought about possible correlations between happiness expressed
by Tweets and living in di erent districts of the same city.</p>
      <p>For what concerns the sentiment analysis task, we currently rely on a simple
lexicon-based approach, where the positive and negative polarity of a Tweet is
calculated based on a dictionary of Italian words annotated with the word's
polarity. In this work our rst focus was on the sentiment visualization and
summarization issue, but we are currently working to improve the analyzer and
provide a reliable evaluation of it. For this purpose, we are developing a gold
standard corpus of manually annotated Tweets that can be used as a testbed
for evaluation and comparison with other systems.</p>
      <p>
        Another interesting challenge to address is to apply emotion detection
techniques in order to classify Tweets according to di erent emotions (e.g. the
Ekman's basic emotions used in [
        <xref ref-type="bibr" rid="ref12 ref2">12, 2</xref>
        ], or the emotional categories from the
Plutchik's model used in [
        <xref ref-type="bibr" rid="ref15">15</xref>
        ]) and to provide a sort of geography of emotions.
      </p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <given-names>V.</given-names>
            <surname>Basile</surname>
          </string-name>
          and
          <string-name>
            <given-names>M.</given-names>
            <surname>Nissim</surname>
          </string-name>
          .
          <article-title>Sentiment analysis on Italian tweets</article-title>
          .
          <source>In Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis</source>
          , pages
          <volume>100</volume>
          {
          <fpage>107</fpage>
          ,
          <string-name>
            <surname>Atlanta</surname>
          </string-name>
          ,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <given-names>C.</given-names>
            <surname>Bosco</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            <surname>Patti</surname>
          </string-name>
          ,
          <article-title>and</article-title>
          <string-name>
            <given-names>A.</given-names>
            <surname>Bolioli</surname>
          </string-name>
          .
          <article-title>Developing corpora for sentiment analysis: The case of irony and Senti-TUT</article-title>
          .
          <source>IEEE Intelligent Systems</source>
          ,
          <volume>28</volume>
          (
          <issue>2</issue>
          ):
          <volume>55</volume>
          {
          <fpage>63</fpage>
          ,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <given-names>E.</given-names>
            <surname>Cambria</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Schuller</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Xia</surname>
          </string-name>
          , and
          <string-name>
            <given-names>C.</given-names>
            <surname>Havasi</surname>
          </string-name>
          .
          <article-title>New avenues in opinion mining and sentiment analysis</article-title>
          .
          <source>IEEE Intelligent Systems</source>
          ,
          <volume>28</volume>
          (
          <issue>2</issue>
          ):
          <volume>15</volume>
          {
          <fpage>21</fpage>
          ,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4. C. Fellbaum, editor.
          <source>WordNet: An Electronic Lexical Database</source>
          . MIT Press,
          <year>1998</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <given-names>P.</given-names>
            <surname>Goncalves</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Araujo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Benevenuto</surname>
          </string-name>
          , and
          <string-name>
            <given-names>M.</given-names>
            <surname>Cha</surname>
          </string-name>
          .
          <article-title>Comparing and combining sentiment analysis methods</article-title>
          .
          <source>In Proc. of the 1st ACM Conf. on Online Social Networks</source>
          , pages
          <volume>27</volume>
          {
          <fpage>38</fpage>
          . ACM,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <given-names>S.</given-names>
            <surname>Kumar</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Morstatter</surname>
          </string-name>
          , and
          <string-name>
            <given-names>H.</given-names>
            <surname>Liu</surname>
          </string-name>
          .
          <source>Twitter Data Analytics</source>
          . Springer, New York, NY, USA,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <given-names>K.</given-names>
            <surname>McKelvey</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Rudnick</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Conover</surname>
          </string-name>
          , and
          <string-name>
            <given-names>F.</given-names>
            <surname>Menczer</surname>
          </string-name>
          .
          <article-title>Visualizing communication on social media: Making big data accessible</article-title>
          .
          <source>In Proc. of CSCW Workshop on Collective Intelligence as Community Discourse and Action</source>
          ,
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8. L. Mitchell,
          <string-name>
            <given-names>M. R.</given-names>
            <surname>Frank</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K. D.</given-names>
            <surname>Harris</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P. S.</given-names>
            <surname>Dodds</surname>
          </string-name>
          , and
          <string-name>
            <given-names>C. M.</given-names>
            <surname>Danforth</surname>
          </string-name>
          .
          <article-title>The geography of happiness: Connecting Twitter sentiment and expression, demographics, and objective characteristics of place</article-title>
          .
          <source>PLoS ONE</source>
          ,
          <volume>8</volume>
          (
          <issue>5</issue>
          ),
          <year>05 2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <given-names>C.</given-names>
            <surname>Monti</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Rozza</surname>
          </string-name>
          , G. Zappella,
          <string-name>
            <given-names>M.</given-names>
            <surname>Zignani</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Arvidsson</surname>
          </string-name>
          , and
          <string-name>
            <given-names>E.</given-names>
            <surname>Colleoni</surname>
          </string-name>
          .
          <article-title>Modelling political disa ection from twitter data</article-title>
          .
          <source>In Proc. of Workshop on Issues of Sentiment Discovery and Opinion Mining</source>
          ,
          <source>WISDOM'13</source>
          , pages
          <issue>3:1</issue>
          {
          <issue>3</issue>
          :
          <issue>9</issue>
          ,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10. E.
          <string-name>
            <surname>Pianta</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          <string-name>
            <surname>Bentivogli</surname>
            , and
            <given-names>C.</given-names>
          </string-name>
          <string-name>
            <surname>Girardi</surname>
          </string-name>
          .
          <article-title>MultiWordNet: developing an aligned multilingual database</article-title>
          .
          <source>In Proc. of Int. Conference on Global WordNet</source>
          ,
          <year>2002</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <string-name>
            <surname>J. Ratkiewicz</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          <string-name>
            <surname>Conover</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          <string-name>
            <surname>Meiss</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          and
          <string-name>
            <surname>Goncalves</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          <string-name>
            <surname>Patil</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          <string-name>
            <surname>Flammini</surname>
            , and
            <given-names>F.</given-names>
          </string-name>
          <string-name>
            <surname>Menczer</surname>
          </string-name>
          .
          <article-title>Truthy: mapping the spread of astroturf in microblog streams</article-title>
          .
          <source>In Proceedings of the 20th International Conference on World Wide Web, WWW 2011 (Companion Volume)</source>
          , pages
          <fpage>249</fpage>
          {
          <fpage>252</fpage>
          , New York, NY, USA,
          <year>2011</year>
          . ACM.
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          12.
          <string-name>
            <given-names>K.</given-names>
            <surname>Roberts</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. A.</given-names>
            <surname>Roach</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Johnson</surname>
          </string-name>
          , J. Guthrie, and
          <string-name>
            <given-names>S. M.</given-names>
            <surname>Harabagiu</surname>
          </string-name>
          .
          <article-title>EmpaTweet: Annotating and detecting emotions on Twitter</article-title>
          .
          <source>In Proc. of the 8th Language Resources and evaluation Conference</source>
          ,
          <source>LREC'12</source>
          , pages
          <fpage>3806</fpage>
          {
          <fpage>3813</fpage>
          ,
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          13.
          <string-name>
            <given-names>B.</given-names>
            <surname>Shneiderman</surname>
          </string-name>
          .
          <article-title>The eyes have it: A task by data type taxonomy for information visualizations</article-title>
          .
          <source>In In IEEE Symposium on Visual Languages</source>
          , pages
          <volume>336</volume>
          {
          <fpage>343</fpage>
          ,
          <year>1996</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          14.
          <string-name>
            <given-names>C.</given-names>
            <surname>Strapparava</surname>
          </string-name>
          and
          <string-name>
            <given-names>A.</given-names>
            <surname>Valitutti</surname>
          </string-name>
          .
          <article-title>WordNet-A ect: an a ective extension of WordNet</article-title>
          .
          <source>In Proc. of the 4th Language Resources and evaluation Conference</source>
          ,
          <source>LREC'04</source>
          , volume
          <volume>4</volume>
          , pages
          <fpage>1083</fpage>
          {
          <fpage>1086</fpage>
          .
          <string-name>
            <surname>ELRA</surname>
          </string-name>
          ,
          <year>2004</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          15.
          <string-name>
            <given-names>J.</given-names>
            <surname>Suttles</surname>
          </string-name>
          and
          <string-name>
            <given-names>N.</given-names>
            <surname>Ide</surname>
          </string-name>
          .
          <article-title>Distant supervision for emotion classi cation with discrete binary values</article-title>
          .
          <source>In Computational Linguistics and Intelligent Text Processing, CICLing</source>
          <year>2013</year>
          , volume
          <volume>7817</volume>
          <source>of LNCS</source>
          , pages
          <volume>121</volume>
          {
          <fpage>136</fpage>
          . Springer,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          16.
          <string-name>
            <surname>M. Taboada</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          <string-name>
            <surname>Brooke</surname>
          </string-name>
          , M. To loski, K. D.
          <string-name>
            <surname>Voll</surname>
            , and
            <given-names>M.</given-names>
          </string-name>
          <string-name>
            <surname>Stede</surname>
          </string-name>
          .
          <article-title>Lexicon-based methods for sentiment analysis</article-title>
          .
          <source>Computational Linguistics</source>
          ,
          <volume>37</volume>
          (
          <issue>2</issue>
          ):
          <volume>267</volume>
          {
          <fpage>307</fpage>
          ,
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>