With the increasing volume of data, there is a growing need for dynamic data visualization to help reveal instant changes in data patterns. There exist many commercial visualization tools, but traditional scholars are often disengaged from the tool development process; thus, the choice of functionalities is contingent upon tool developers whose choice may not fit the end-users. This collaboration, however, has a potential in bridging the gap between traditional scholars, who are more interested in sense-making of the text than in the tools, and the data scientists, who are more interested in the tools than in the substance, but must still contextualize the outcomes. Until recently, this collaborative process was hindered by the complexity of customization procedures and technological hurdles imposed on users with new installations. With the advent of reactive web frameworks, such as Shiny, the user-driven customization becomes not only feasible, but also essential to advance scientific research. In this paper, we demonstrate a collaborative e↵ort between learned scholars and tool developers, allowing for a computational and humanistic fusion.
In the last decade, the volumes of data collections have
grown so “large and complex that it becomes dicult to
process using on-hand databases, management tools or
traditional data processing applications” [
Until recently, this collaboration was unfeasible. Software
not only necessitates a team of software engineers and
designers, but also requires installation and consistent updates,
which is a technical hurdle for users. Furthermore, the
design of collaborative visualization has been commonly
described as a grand challenge for visualization research [
With the advent of reactive web applications, such as Shiny, the user-driven tool customization becomes a reality. First, these applications require no installation and are accessible from any web browser, which enables a direct testing of new functionalities by users. Second, the reactive framework allows for a creation of highly dynamic tools with minimum knowledge of web development. Finally, Shiny is a web application developed for R, which is an open source language with a large library for data visualization.
In this paper, we describe our current collaborative
research on text mining and visualization customization. Our
goal is to assist scholars in their process of ingestion
(‘reading’), digestion (analyzing and sense-making), and egestion
(through the creation of new learned texts via queries). Our
workflow is illustrated in Figure 1.
Our initial stage begins with the current version of
Interactive Text Mining Suite,1 a Shiny web application, developed
to test various text mining and visualization techniques for
digital humanities scholars [
This paper is organized as follows. In section 2 we introduce Shiny, a reactive web framework. We then describe Interactive Text Mining Suite and its current functionalities in section 3. Section 4 and 5 will overview the development of customized functionalities for scholarly research, followed by conclusions and future directions presented in section 6. 2. 2.1
Traditional imperative web framework model was
developed by Trygve Reenskaug in 1979 and followed a
threecomponent structure: model, view, and control [
Interactive Text Mining Suite (henceforth, ITMS) is
designed to assist humanities scholars in the discovery of new
insights and patterns within large digital collections, and
to provide access to natural language processing techniques
with a user-friendly design. Its major strength is the ability
to work with data in various formats, PDF and text formats,
as well as CSV, JSON, and XML, as shown in Figure 2.
In contrast, many existing text mining tools are limited to
specific importing formats. Additionally, ITMS performs
a wide range of common preprocessing tasks, allowing for
maximum flexibility and user control, illustrated in Figure
3 (for a more detailed description, see [
As mentioned in 2.2, ITMS was designed as a digital humanities tool suitable for performing common text mining tasks and visualization methods. That is, it was built for scholars, but not by humanities scholars. However, there exists a gap between scholars, who have been doing more qualitative text-based research for public and government sectors, and data scientist/computational linguistics scholars, who work on theoretical text-mining research.2 To bridge this gap, we have developed a collaborative communication between these two communities (a.k.a. end-users and developers). Instead of a typical github environment for reporting progress and issues, we chose a social collaborative platform rizzoma.3 Rizzoma is built as knowledge-management and discussion platform allowing for real-time team communication and multimedia support. Figure 4 illustrates our collaborative project structure. In the following sections, we describe our workflow and design considerations based on this collaboration. 4.
Based on the previous work with existing text mining tools, it was determined that the main pollutant for scholarly research is the inability to pre-define text excerpts within the text collection. It appears that stopwords filtering and text preprocessing were not sucient to obtain intuitive data interpretation for qualitative scholarly studies. Collaboratively, we have developed and tested the following algorithm (see also Figure 5): 1. Parse document collection 2. Scan every document for a specific term defined by the user (e.g., “security”) or two terms (e.g., “influenc*” within 10 words of “Europ*”) 3. Define a window around these terms (e.g., 10 words to the left and 10 words to the right) In addition, scholarly research collections are often stored and accessed via bibliographic management systems (e.g., Zotero, Mendeley, and Endnote). While most of these systems do not perform text mining analysis, the Zotero plugin, namely Paper Machine,4 o↵ers a wide range of interactive visualization for document collections. Nevertheless, the user cannot control text segmentation, which yields very broad topic and metadata visualizations. Given that Zotero is the main bibliographic management system in our collaborative project, data import from Zotero into ITMS became one of the most essential primary tasks for our team. Several options exist for exporting library collections from Zotero, namely rdf and csv formats. However, a few issues were discovered during the exploratory phase: 1) csv and rdf files only contain local paths to actual document articles (see Figure 6); 2) local paths cannot be accessed directly from a remote web application; 3) running ITMS locally would require R installation and some programming knowledge, thus generating technological hurdles for end-users. 4. Include only the extracted segments into data analysis and visualization lowing criteria: 1) functionality and 2) the level of complexity. The first approach is the development of a small Shiny application installed locally that would process rdf library collections, export pdf files, convert them into text files, and place them into one directory, which can be accessed from our web application (see Figure 7). This application is used only once and has a low level of complexity, yet the functionality is less user-friendly, as it creates an additional directory with extracted files from Zotero. These files can then be imported into ITMS. The second approach is suggested by the end-users: export zotero library as a csv file, run a local script to extract all pdf files, and add them into the CSV file as a plain text. While the functionality is high, the level of complexity is much higher. 5.
There is no doubt that visual analytics facilitates
analytical reasoning [
The ability to perform dynamic and interactive visualization is one of the strengths in reactive applications. While there are many R libraries implementing various types of interactive visualization, we decided to examine two packages, namely plot.ly and googleViz. Comparison and parallel testing feed our decision to implement their functionalities into ITMS. Table 5.2 presents our current summary. Types Stepped Area chart Bubble chart Gauge Intensity Map Geo Chart Table with pages Tree Map Annotation chart Sankey chart Calendar chart Timeline chart Merging charts Flash charts Annotated time line chart Chord diagram Filled Chord diagram k-means clustering Stream Graph PCA Hierarchical Clustering Doughnut Chart
NA NA NA NA NA GoogleViz
Plot.ly NA NA NA NA After identifying their functionalities, our next step is to determine the best fit via our collaborative feedback. 5.3
Topic modeling is a statistical model used in machine
learning and natural language processing for discovering
abstract topics that occur in a collection of documents. This
analysis assists in “classification, novelty detection,
summarization, and similarity and relevance judgements” [
By comparing other software with their unique options for topic representation, two candidates for ITMS were identified: topic bubbling and topic coupling from MALLET, a topic modeling package. The goal of topic bubbling is to compare the relative importance of all the topics; the size of a topic bubble is the accumulated size of all word bubbles within that topic. In contrast, topic coupling reveals the relations between the topics based on their associated words. In this representation, topics are shown as a network of terms (nodes) linked by their interaction with other topics.
In recent years, we have seen growing interest in the use of data visualization tools in the humanities fields. However, many of the existing tools are unable to integrate the humanistic component of exploratory research. Thus, the overarching goal of the current work on ITMS is to bridge the gap between tool-developers and learned scholars by adding a user-customization component. In addition, the social interaction between scholars and data scientists has a strong potential to promote text mining methods among humanities as well as to enhance capabilities and functionality of visualization tools. We have also shown that a recent development of reactive Shiny framework has facilitated the task of user-customization: On the one hand, a wide range of open source R libraries and its overall simplicity for deployment made the Shiny framework very accessible to non-experienced programmers. On the other hand, Shiny is user-friendly web application, where users are not constrained by limitations of their local computer memory and platform dependency, as compared to other software tools. While this project only focuses on a collaboration between political/social science scholars, this idea can be extended to other fields. Below we summarize some of the possible implementations for future research: 1. Teaching tool: The web application is developed in the conjunction with the lesson plans, for example statistics modules. The collaboration can also be expanded by including students into the development and testing phases. 2. Digital Humanities: Based on individual research, the web application can be augmented with additional visualization types, for example spatial or chronological maps for literature analysis. 3. Social Science: The user could specify additional media for research and customize their appearance, e.g. tweets, blogs, or photos.
All these considerations and scholarly collaboration also present
new opportunities for the field of data visualization and
analytics, advancing our understanding of computation and
human nature, namely “synthesis of computational and
humanistic modes of inquiry” [