D.2 [Software Engineering]: Design Tools and Techniques Flow charts, Object-oriented design methods, User interfaces.
How can visualizations of massive amounts of information be made more useful for data journalists? The availability of large amounts of publicly available user generated content is opening new opportunities to study social, cultural, and communications phenomenon. Computer assisted analysis now makes it possible to explore the relationship between nodes and text without having to choose between data size and depth. To create a visualization technique that would allowed us to reveal the network of actors and the main themes hidden in a large dataset, we had to work in a method of inquiry for social sciences. Based on the actor-network theory (ANT) we explored a dataset extracted from Twitter in order to map relationships and indicate new possibilities for journalists by discovering main themes around a hashtag, this way we interpret a layer of text multiple times, analyzing the nodes in its many attributes. Beyond the boundaries of 140 characters, this approach can succeed as it reproduces and reveals the dynamic connections contained in a collective phenomenon. In the last section, we demonstrate a prototype visualization that reveals behaviors and discourses within the large sample datasets. . We use the D3 visualization library to overlap related links and nodes to produce a comprehensible interactive visualization. Our model is interactive and allows us to identify part and whole pattern relationships constant with the three principles of information visualization: overview first, zoom and filter, then details on demand. This paper analyses networks from the perspective of ANT in order to create a visualization ready to support users when telling a story with data.
D.3.3 [Programming Languages]: Language Constructs and Features – abstract data types, polymorphism, control structures.
A fair number of events and social phenomenon find themselves
connected; they are caused by a range of parts of a complex
puzzle interacting to each other. As a society [
The project began as a conversation about how to visualize large quantities of data and how this process could support data driven information. We made the decision to focus on hashtag and the Twitter conversations surrounding these hashtags. The conversation led a set of agreed upon features that are represented in the original sketch. The tool had to do two simple things: visualize the frequency of hashtags in a data set and allow the user to click on specific hashtags and read the tweets associated with them. Every other feature we incorporated into the visualization serves one of these two purposes.
The tweets themselves were extracted using the Twarc1 tweet scrapper. Twarc is a command line tool that takes a single search term (in this case the string 'rob ford'), queries the twitter API (Application Programming Interface), and the downloads all of the metadata associated with whatever tweets it finds. However, Twarc alone produces a large amount of unnecessary data. For every 140-character tweet that Twarc downloads, approximately five thousand characters worth of metadata is received. All told, we collected about twenty gigabytes of twitter data. The next step was to filter this data, for that it was built another scrapper, also in python, that would search this data and return in csv format all of the information needed. In this case hashtags, but many other attributes such as: geolocation, mentions, and url are also available. This dataset returned approximately one gigabyte of data. In order to filter the data further, we used an R script to split the csv files, format character codes and time stamps, as well as filter out every tweet that does not contain a hashtag. This reduced the dataset to two hundred megabytes. We then uploaded the entire remaining dataset to a MySQL database through a PHP script. The final step was to query this database for visualization in a JavaScript library: D3. For reaching out a visualization dashboard that could provide interactive information, D3 proved to be extremely useful. All told we employed seven different programs across six different programming languages in order to pre-process the data.
This paper situates the debate and challenges posed by the large
amount information available online. In this matter, we begin with
a context of critical questions on Big Data. Mathematicians,
philosophers, sociologists, and many scholars from different fields
of study are claiming “for access to the massive quantities of
information produced by and about people, things, and their
interactions.”[
Social network sites like Facebook, Instagram, and Twitter
became a central component of sociability in our contemporary
society. User generated content is a way to measure qualitative
data, from the metrics on the success of a product inside the
market to tracing the news about a natural disaster, social media
delivers a massive amount of information everyday. In studies of
network analysis, Twitter has become a broad database for
quantitative and qualitative scholarly analysis. With user
generated content and the flow of information, the microblog is
the virtual space for peoples perspectives online [
The mapping controversies technique is a successful method to
trace digital data. Cartography of controversies is a method
created by Bruno Latour [
A brief background about the case that explains the choice for data: starting in May 16, 2013, a series of reports about a video supposedly showing the Toronto mayor smoking from a glass pipe ends up circulating on the U.S media. Subsequently, media outlet Toronto Star also spread the news about a man their reporters claim in a video smoking crack. This is enough for the long controversy to begin. Since May, from denying allegations to new videos emerging from time to time in several news media, Rob Ford is an ongoing conversation on Twitter.
Building up from the theoretical references exposed above, we
needed a dataset big enough to challenge us within the limits of
back end and front end work with Big Data. With different themes
underlying the discussion on Canadian and Toronto politics, the
dataset extracted from Twitter around Rob Ford elaborates on
how citizens are expressing their concerns on social, economics,
and political issues in the society. The Rob Ford tweets set us up
with long tail of conversations to follow, presenting us with a
scenario demanding of critical thinking about information
visualization. Moretti[
In this section we demonstrate a prototype visualization that
reveals behaviours and discourses within the large sample
datasets. Our model is interactive and allows us to identify part
and whole pattern relationships constant with the principles of
Shneiderman’s[
http://analytics.artsrn.ualberta.ca/viz/hashtag.html 3 D3.js is a JavaScript library for manipulating documents based on data. D3 helps you bring data to life using HTML, SVG and CSS http://d3js.org/ down such as username and keyword search would then provide the viewer greater detail after. Highlighted hashtag occurrence over time, in the context of how often it appears, provides a macro view of a conversation arc over a given period. We also chose to highlight outliers — hashtags that only appear once in the data set — reasoning that they might provide a unique perspective from outside of occurring trends and patterns. After the broad strokes of the overview, the user can explore the data more closely. The ‘zoom’ Schneiderman referred to typically means changes in the scale of magnification — in TweetViz, it is semantic in nature. The user can move from a macro reading of the data to closer examinations of the text. In the original sketches, this is accomplished by either a small word cloud generated for a given hashtag each day, or in the tweets themselves in a second panel. Filtering is achieved with a date-range selector and a username and keyword search. A significant design concern for large data sets is dealing with occlusion: ensuring that the design inhibits visual elements overlapping as much as possible. In the early sketches, we designed a division between the 10 most commonly occurring hashtags and the rest of the hashtags in order to minimize overlap: when the slider bar is raised, the user can see all but the top 10 occurring hashtags in their relative (and often occluded) arrangement; when the bar is lowered, greater vertical space lessens overlap for the top 10.
The current visualization offers a toggle between relative and absolute views of the top 10 hashtags, and uses jitter — the slight, irregular movement of overlapping hashtags — to reveal overlapped elements at minute intervals. In the next paragraphs, we engage in the process of untangling the "Hairball" by building our own tool. The visualization dashboard consists of two screens. The first is a visualization of the relative frequency of each hashtag in the data. The larger the percentage of tweets that that hashtag gets used in the higher it appears on in the chart. Secondly, we also wanted to visualize the contents of these tweets. This is done in two ways. Firstly, the original design had a word cloud associated with each node, this word cloud is designed to offer an ‘at a glance’ insight into the content of the tweets represented by a hashtag. Secondly the user can click on a node and transfer the tweets in that node to the second screen. The viewer is simply a widget that allows the user to sort, filter, and read individual tweets. Due to the incredible quantity of tweets that twitter processes on a daily basis, the unique identification numbers assigned to each tweet was massive. Unfortunately, not every program handles large numbers in the same way, and due to the large assortment of programs in use, not all of this data was translated between languages perfectly. Another problem encountered is due to character encoding. Because twitter is an international platform, it is extremely lenient in which characters it allows. Unfortunately, due to the large amount programs and data formats used, not all of which allow by default the entire unicode character set, certain characters needed to be removed from the set (notably all newlines, carriage returns, and some foreign symbols I could not identify) and certain characters were lost in translation. An example of where this problem appears is in the ‘t’ hashtag in the rob ford data set. Unfortunately, ‘t’ is only a small part of the hashtag itself, but the rest does not render properly. Beyond the prototype stage a better solution to this project needs to be addressed. Request size also proved to be a problem. Javascript is a client side service, and in order for it to visualize properly the entire data set needs to be processed and transferred to the user computer. Unfortunately, due to the size of the project, these requests tended to overwhelm the earlier versions of the project. Early versions of the twitter viewer actually fetched the full text of every tweet it was analyzing. This was necessary because it was the easiest way to generate the word clouds dynamically. However, this soon proved to be too much for JavaScript to handle. Instead, we needed to preprocess all of the data on the server. Unfortunately, this meant that the word clouds needed to be generated outside of D3. Due to the difficulties to visualize, it was decided to cut the world clouds and only visualize the content through the tweet reader.
Due to the incredible quantity of tweets that twitter processes on a daily basis, the unique identification numbers assigned to each tweet was massive. Unfortunately, not every program handles large numbers in the same way, and due to the large assortment of programs in use, not all of this data was translated between languages perfectly. Another problem encountered is due to character encoding. Because twitter is an international platform, it is extremely lenient in which characters it allows. Unfortunately, due to the large amount programs and data formats used, not all of which allow by default the entire unicode character set, certain characters needed to be removed from the set (notably all newlines, carriage returns, and some foreign symbols I could not identify) and certain characters were lost in translation. An example of where this problem appears is in the ‘t’ hashtag in the rob ford data set. Unfortunately, ‘t’ is only a small part of the hashtag itself, but the rest does not render properly. Beyond the prototype stage a better solution to this project needs to be addressed. Request size also proved to be a problem. JavaScript is a client side service, and in order for it to visualize properly the entire data set needs to be processed and transferred to the user computer. Unfortunately, due to the size of the project, these requests tended to overwhelm the earlier versions of the project. Early versions of the twitter viewer actually fetched the full text of every tweet it was analyzing. This was necessary because it was the easiest way to generate the word clouds dynamically. However, this soon proved to be too much for JavaScript to handle. Instead, we needed to preprocess all of the data on the server. Unfortunately, this meant that the word clouds needed to be generated outside of D3. Due to the difficulties to visualize, it was decided to cut the world clouds and only visualize the content through the tweet reader.
In terms of visualization, crowding turned out to be the biggest problem in the visualization itself. Once the initial prototype was built on a small subset of the data, it became immediately apparent that some of the assumptions made in the original design were not true. The first assumption was that spacing between the top few hashtags would be relatively even. We could visualize the top hashtags as a relative percentage and use a slider bar to 'squish' all of the lower hashtags down allowing us to push them out of the way and focus on the higher percentage hashtags. In the Rob Ford data set, this is false, and in fact, the opposite is true. The top hashtags are completely dominant, and only the top three or so are actually visible on a relative scale with everything else squishing into the bottom. Instead, of using a slider bar to push the lower less important hashtags out of the way, it became apparent that we needed a way to focus in on the lesser hashtags and push the dominant ones out of the way.
In short, our research builds up from a solid theoretical reference to visualize relationships in a network, the alliance between computing methods and the humanities consider qualiquantitative techniques that means more than overlapping statistical resources and ethnographic approach. We believe that information is only visible when the user can have the opportunity to click on, explore, discover, and share new findings. Data analysis can serve as technique to reveal the different structures of the same story and to provide new lens to see levels of information. When journalists use data to do their jobs they shift from being the first one to communicate to being the ones telling people what a certain progress of an event may actually mean. This tool can be appropriate by for journalists trying to visualize news and events, using data to transform something abstract into something everyone can understand and relate to the real events. With the curiosity to continue to think critically on how to display digital information and to explore data, for the future work we hope to overcome the issues with data encoding and crowding in our tool.
Funding for the project generously supplied by Just What do They Do (JWDTD), Implementing New Knowledge Environments (INKE), and Social Science and Humanities Research Council of Canada (SSHRC).