Advancements in mobile technology and the proliferation of social media platforms have made it possible for individuals to stay constantly connected with friends and family. This has provided new opportunities to the emergency response domain, where the information shared by individuals in crisis can provide invaluable insight into the situation on the ground. Information shared on social media is highly dynamic, heterogeneous, large scale, geographically distributed and multilingual. Moreover, the context of such information is mostly relevant for a very short period of time and the information can be very subjective, embedded in personal feelings. This is a signi cant challenge for the emergency response domain, where critical decisions need to be made quickly on the basis of the users' situation awareness. We propose to address this issue using visual analytics techniques to facilitate browsing and understanding of topicality and feelings in social media. Our approach is twofold- rstly, we enrich social media posts by adding semantics to facilitate browsing and sentiment in order to gauge the emotions behind individual posts. Secondly, we combine two paradigms of data browsing - topical and temporal into a real-time dynamic visualisation of social media messages.
Social networking and media-sharing platforms such as Twitter, Facebook and
Flickr has seen a large scale adoption by individuals and communities across the
web over the past few years. Cheaper mobile phones and internet-enabled
devices have made it possible for individuals to stay connected with such platforms,
thereby enabling people to share their experiences at all times. This amass of
data continually generated has become a major source of information for
organizations, user groups and communities to understand important situations and
events. Such data streams are particularly useful in the case of emergencies and
crises - where highly critical and potentially life-saving decisions need to be taken
in a short amount of time. Social media data have become a natural port of call
to achieve Situational Awareness, i.e. accurate, complete and real-time
information about an incident [
The goal of our research is to boost the users' understanding of interrelations
between topicality and sentiments by the timely visual analysis of large-scale
social streams (aggregated from Twitter, Facebook and Flickr). To this end, we
have designed and developed VisIn uence, a new visualisation approach that
leverages the semantic value of the information, to create contextual enrichment
for visually browsing information. While e ectively ltering dynamic social data
is a signi cant challenge, our focus in this work is on exploring the sentiments
associated with social media. Our present work is an extension of a previous
work in visualising in uence in social media, where we focussed on visualising
in uential users[
In the following sections we present related work focused on the use of social media data for situation awareness and on visual analytics in emergency response, This is followed by a description of our approach for obtaining and visualising the data, with a focus on the new visualisation approach we designed. Details about the implementation are then presented, followed by Conclusions and Future Work.
In this section we present related work in two areas : Situational awareness and
Visual Analytics in emergency response. Situational Awareness in times of
emergency is paramount to deliver a timely and e ective response [
Visual Analytics techniques have been proposed to represent and lter social
media at di erent levels of speci city [
Whilst most social media visualisation approaches rely on geographical and
temporal features TwitterReporter [
The rst step in our approach is enriching social media posts by adding semantics to facilitate browsing and sentiment to gauge the emotions behind individual posts. Social Media messages are typically composed of metadata (e.g. about the users, the device used to post, the location, etc.) and content. Both metadata and content can be analysed to extract information (e.g. keywords, terms, named entities, events, etc.) and to establish semantic data. In our implementation, we have used four processes to extract information from the messages - a tag processor to extract user-generated tags, a user detail extractor to extract metadata from the posts (such as user names, pro les and so on), an entity extractor and a sentiment analyser. The entity extractor and sentiment analyser are modules that use the Alchemy API3 to extract di erent entities and sentiments of the messages. Our approach is based on two sub-steps: (1) Content analysis to identify related concepts that are later fed into the topical navigation widget; and (2) Sentiment analysis, to understand the mood of users and their in uence on a topic. We illustrate this using the following example, Alyssa Milano @Alyssa_Milano: I am so blown away by the police officers and all 1st responders in Boston. Awesome bravery. I salute you! #BostonStrong
The user name (Alyssa Milano) and the user ID (@Alyssa Milano) is initially stored along with the user-generated hashtag BostonStrong. The text of the tweet is then passed to the Alchemy API to extract relevant concepts (Tags: Awesome Bravery, police o cers, responders) and sentiment (0.146867). We combine this information as shown below and store it within a local data store, to be queried later. The triples are formalised based on a visin uence vocabulary as well as several other well-known vocabularies such as dcterms4, sioc5 and geonames6. While our visin uence vocabulary is rather simplistic, our focus in this paper is on visual analytics instead of formalising the domain. @prefix vin: <http://dcs.shef.ac.uk/visinfluence/> <http://dcs.shef.ac.uk/visinfluence/status/325418041229848577> rdf:type sioc:Post ; sioc:hasCreator <http://twitter.com/Alyssa_Milano> ; dcterms:created "2013-4-20 02:18:01" ; dcterms:subject "BostonStrong" ; vin:City <vin:Boston> ; vin:Tag "Awesome Bravery" ; vin:Tag "police officers" ; vin:hasSentiment 0.146867 ;
When all the tweets are processed, the system can search for tweets that are relevant to topics as well as created within speci c temporal durations. The retrieved set of tweets is then `bundled' into one single representation in the visualisation. In our current implementation, the sentiment score for a set of
tweets is calculated as the mean of their individual scores. In the near future, we will investigate other ways of generating combined sentiment scores for a group of tweets. 4
We build our approach on the classical visual information - seeking mantra
proposed by Shneiderman - Overview rst, zoom and lter, then details-on-demand
[
Being a time-critical application domain, users need to be able to quickly communicate what they are interested in. This calls for easy means to identify their topics of interest, and then subsequently add them to their list of topics to monitor. In our present implementation, topics can be of two types - user-generated hashtags or entities (extracted from the text). Upon initialisation, users are presented with a tag cloud that represents a summary of all the hashtags that are posted from the time the data is collected. The tag cloud is also presented with a drop-down menu showing di erent types of entities that have been identi ed while processing the posts. Additionally, the system provides users with a topical search facility to add their own search terms. In the next stages, the tweets relevant to the identi ed topics will only be considered, as shown in Figure 1. The nal set of topics are shown to the user as a list, which can be edited as desired before proceeding to the next step of exploration. This stage is a notable change from the original metaphor of visual information seeking proposed by Shneiderman, where the author proposed that the rst step would be to provide an overview of everything, and then provide ltering operations. We argue that in order to make quick and important decisions, users need to rst identify the topics that are most relevant to them. This saves critical time and can help users guage situations that are of utmost importance to them. The set of topics identi ed by the user are then used by VisIn uence to render the nal visualisations. The visualisation space consists of individual horizontal panels for each topic, and each panel presents the dynamic information collected about the topic. The panels are divided into several vertical time frames (600 time frames in our present implementation), each time frame indicates a duration of time. Figure 2 shows one such example, where each time frame represents 10 seconds - this means that every 10 second, a new reading will be requested from the backend, where the tweets collected over the last 10 seconds will be processed to calculate a combined sentiment score.
The backend extractors harvest tweets related to the individual topics, based on its own search criteria. The visualisation interface then queries the backend for the latest readings of the collective sentiments for all the topics, which are then rendered as a bar on the panel. The bar is color-coded to indicate a positive or a negative sentiment (positive displayed as a green bar, while negative as a red). The length of the bar indicates the extent of the general positive or negative feeling associated with the topic at the given time frame.
Users can control how often the visualisations are updated using a slider on the top of the interface, thereby choosing how quickly new data is visualised in the system. When a new time frame is initiated, all the readings for the previous timeframes are shifted toward the left, and the new readings are plotted on the far right. The oldest reading from the left most timeframe is deleted from the system. This enables users to have a historical perspective on the sentiments associated with the topic and follow how it develops as a situation evolves - however, ensuring that the users donot get overloaded by the amount of information that is presented to them.
The system also allows users to add new topics of interest at run-time, while the visualisations are being updated. The users can use the search box on the top right side of the interface (Figure 2) to enter terms they are interested in, and the topics get added to their list of topics to monitor. This is to accommodate for new topics of interest that they observe from their monitoring activities. The new topics get stacked on top of the previous topic panels - e.g. \999" and \911" panels are added by the user after they observe that these topics can be relevant as shown in Figure 2. 4.3
The users interact with the interface to explore the data to a greater extent. For example, hovering over the panels show a scale that encapsulate all the other panels, and show the readings for each panel at that time frame. In addition to the visual indication of the computed sentiments being rendered as charts, the scale shows the real values of the sentiments to enable a more ne-grained comparison. Users can also visualise the number of tweets posted as well as the number of unique users posting tweets that were relevant to the topic at the speci c time frame. Users select from a drop-down menu the data they would like to visualise and the queries to the back end extractors would then be modi ed based on the facet of the data they would like to visualise. The tweet count and unique user count is presented in the same interface, similar to the sentiment plots where bars represent the number of tweets or users. Unlike sentiment plots, however, these plots are not color-coded, and merely represent how many tweets have contributed toward the nal sentiment scores. Figure 3 shows an example where the tweet count has been selected as the facet to be visualised for the topic 911 - the gure shows the rising amount of the tweets over the latter half of the exploration session. The number of tweets had suddenly spiked, and dropped at the end of the session.
We believe that this step is similar to the `overview' step in Shneiderman's information seeking paradigm, since this allows the users to have a holistic view of the data. However, our extension for this step is to facilitate an exploratory paradigm where the user can be provided with means to decide which facet of the data is of interest to them. The nal step of our approach is similar to Shneiderman's - where we provide means to reach the details of the data. It is extremely important to make this as seamless as possible, and make it easy for the user to understand the data being presented. Users can access details during their exploratory session by hovering their mouse (and consequently, the scale) to view the scores at the speci c time frame and clicking on the plot area. The exact position of the mouse click is then calculated to identify the time frame and the topic that the user is interested in. At this stage, we provide users with views over multiple aspects of the data. The most important facet that users are presented with are the individual instances of the tweets. This assists the analyst in understanding why the sentiments around a given topic at one particular time frame generates its scores. The tweets are ranked on the basis of their sentiments - users can chose to view the tweets with the most negative scores on top, and the most positive scores at the bottom and vice versa. The second facet is the list of most in uential users for the topic at the chosen time frame. The list of users is also ranked on the basis of their contribution toward the sentiment score. The user with most tweets that are negative with the greatest margin is placed on top of the list, while the user with the most positive tweets are placed at the bottom of the list. 5
We discuss the implementation of VisIn uence in two phases - data processing and visual representation. The rst step requires streaming Tweets using Twitter API - this process introduces a few challenges - searching the public timeline for tweets would result in large number of tweets that are irrelevant. Hence, as a starting point, VisIn uence is fed a few search terms to gather an initial set of data. This set of search terms is directly related to the domain - for example, a user monitoring a music festival would prefer using terms relevant to the festival, while an emergency personnel may prefer using generic terms such as `emergency', `disaster' and so on. This initial set of tweets collected every 15 minutes are then fed into four extractors - a user detail extractor, tag processor, entity extractor and sentiment extractor. The entity extractor identi es the various types of entities found in the tweet - these entities can be of several types such as persons, countries, cities, tags and so on. The sentiment analyser computes the sentiments associated with each tweet. The entity extractor and sentiment analyser are essentially scripts that invoke the language processing service Alchemy API7. The responses from the Alchemy API are parsed and stored locally, available to be queried at later stages.
The other two modules, Tag Processor and User Detail Extractor extracts the hashtags and relevant user information such as user IDs, user names and locations. The extracted information is then stored in a local triplestore, via a backend controller. The user-facing part of the system, or the front end constitutes of a visualisation interface built on Processing.js8. The processing modules are displayed on an HTML page, along with several other Javascript-based tools such as JQueryTagCloud9 for rendering the tag cloud and jQuery to handle AJAX queries to the backend. Each interaction with the user results in queries that are then sent to the backend controller. The queries are then forwarded to the triplestore, and the results retrieved are returned to the relevant callback functions. The responses are then parsed using Javascript, and rendered accordingly. 6
The development of the VisIn uence system has followed an iterative usercentered design process, where users have been presented with incremental versions of di erent features. Another recently developed system, the Context and
8 http://processingjs.org/
9 http://www.jondev.net/articles/jQuery Tag Cloud Example
Hierarchy chain [
One of the primary challenges that the system faces is limited computational resources. Processing thousands or millions of tweets at near real-time is highly challenging and computationally expensive. This results in processing delays, that can introduce a lag in the system. While such delays are expected, a time-critical domain such as emergency response requires careful consideration of how signi cant and what is the possible impact of such delays. Indeed, our approach of lter- rst can reduce unnecessary and irrelevant information to be discarded, a helpful addition would be e cient spam-detection techniques. Identifying spammers and spam content before as a pre-processing step would signi cantly reduce the amount of information to be processed, thereby reducing the possible delays.
Another set of issues associated with the system is the limits introduced by services - Twitter and Alchemy are both services that impose certain restrictions in terms of number of API calls. In our implementation, we have had to introduce delays to ensure that calls to such services respected these restrictions. These delays have also added to the overall delay in the system. However, such issues can be mostly addressed by paid or premium accounts with such services. In the near future we plan two major modi cations in the system - improve the backend processing and provide a way to improve the ltering mechanisms. With these nal set of updates, we plan to evaluate the system with real users from the emergency response domain. We aim to validate our four-step approach and understand how can such technologies be used to compliment more traditional techniques.