=Paper=
{{Paper
|id=Vol-2068/esida1
|storemode=property
|title=PIHVI: Online Forum Posting Analysis with Interactive Hierarchical Visualisation
|pdfUrl=https://ceur-ws.org/Vol-2068/esida1.pdf
|volume=Vol-2068
|authors=Jaakko Peltonen,Ziyuan Lin,Kalervo Jaäarvelin,Jyrki Nummenmaa
|dblpUrl=https://dblp.org/rec/conf/iui/PeltonenLJN18
}}
==PIHVI: Online Forum Posting Analysis with Interactive Hierarchical Visualisation ==
https://ceur-ws.org/Vol-2068/esida1.pdf
PIHVI: Online Forum Posting Analysis with Interactive
Hierarchical Visualization
Jaakko Peltonen∗ , Ziyuan Lin∗ , Kalervo Järvelin∗ , and Jyrki Nummenmaa∗
jaakko.peltonen@uta.fi, ziyuan.lin@uta.fi, kalervo.jarvelin@uta.fi, jyrki.nummenmaa@uta.fi
ABSTRACT or administrators of the venue. Online forums are big data:
We introduce PIHVI: a novel interactive system for visualizing as a case study we use a popular Finnish discussion forum
and exploring a large hierarchical text corpus of online forum “Suomi24” (www.suomi24.fi) spanning 16 years and 6.5 mil-
postings. The main view of the visual interface shows a large- lion threads. Hierarchically organized discussion occurs also
scale scatter plot, created by flexible nonlinear dimensionality for example in user commentaries on news sites whose news
reduction based on text contents of the postings, and we couple have a hierarchy, in online reviews of products that have a hi-
it with a coloring optimized by a second dimensionality reduc- erarchy, and so on; our work is applicable to such hierarchies
tion to represent the forum hierarchy. We exploit the hierarchy as well but we focus on the discussion forum case.
to provide data-driven summaries of plot areas at multiple lev-
In discussion forums, sections and their hierarchy are created
els of detail, allowing the user to quickly see and compare both
by administrators as a simplified division of discussions in-
the content-based similarity of groups of posts and how near
tended to represent a subset of prototypical user interests, and
they arise in the forum hierarchy. A user can move between
such a division does not suffice to describe the true variety of
hierarchy levels, mark posts or spots of interest, filter posts by
semantic content in online discussion. Our target users are ad-
content similarity and by location within the hierarchy, and
ministrators and analysts of the forums: if the hierarchy does
inspect post contents. Experiments show the interface can
not support the content, forum owners can add subsections or
reveal hidden semantic relationships between postings that
shortcuts to promote content, aiming to increase user engage-
would be hard to find based on the known hierarchy alone.
ment with the venue. An important task in data analytics of
ACM Classification Keywords online forums is to perform visual analytics of how discus-
D.2.2 Software Engineering: Design Tools and Techniques sions and their underlying topics spread across the hierarchy.
User interfaces; H.5.2 Information Interfaces and Presenta- When the data and the hierarchy are large this becomes a hard
tion: User Interfaces User-centered design; I.2.7 Artificial task, as exhaustive inspection of all data or comparisons across
Intelligence: Natural Language Processing Text analysis all sections become unfeasible: computational support is then
needed. However, most visual analytics tools for social media
Author Keywords discussion focus on other aspects such as temporal evolution
user interaction; visual analytics; text analysis. or networks of conversation participants.
We propose a new system for analysis of online conversation in
INTRODUCTION hierarchical forums in a large scale, by an interactive interface
Online discussion often takes place in venues that have a hier- that scales to large data (Figure 1; details in later sections). We
archical organization, such as large message boards organized focus on analysis of content variation and its relationship to the
in sections, subsection, sub-sub-sections and so on. While hierarchical sections; we do not focus on e.g. temporal filter-
some venues such as specialized message boards may use a ing or other such aspects which are extensively studied in other
relatively flat organization having a small number of sections systems and can easily be integrated as additional views for
directly under a root, other venues such as general-interest our system. Main novelties: first work to use dimensionality
message boards have an extensive multilevel hierarchy form- reduction based linking of views through optimized coloring;
ing a deep tree of sections. Users post new threads, and post among the first to use large-scale dimensionality reduction in
comments and replies under threads, but the hierarchy of sec- visual analytics of discussion venues; novel multi-level inspec-
tions is typically designed by owners of the discussion venue tion of a large-scale scatterplot; novel use case - analysis of
* All the authors are affiliated with University of Tampere, Finland. suitability of existing hierarchy in a venue to describe content
The work was supported by Academy of Finland decisions 295694 of the venue. We next discuss related work, introduce the
and 313748 and by Aalto Science-IT. system, denoted PIHVI - Posting Analysis with Interactive Hi-
erarchical Visualization, then describe our machine learning
solution, describe a user study and discuss conclusions.
RELATED WORK
Visual analytics for social media has been an active research
topic, as social media data are rich and heterogeneous in struc-
©2018. Copyright for the individual papers remains with the authors. ture and thus difficult to model. A recent survey [2] taxono-
Copying permitted for private and academic purposes. mized methodologies on the field by Data sources – Twitter,
ESIDA ’18, March 11, 2018, Tokyo, Japan
Facebook, blogs, online forums, etc.; Entities of data – net- various visual analytics for inspecting online forums. Sim-
work, geographic information, or text content; Goal of visual- ple text search does not suffice when users aim to broadly
ization – pattern extraction, visual monitor, anomaly detection, comprehend the information space with multiple exploratory
etc.; Visualization techniques – scatter plots, trees, node-link searches and information needs [17]. For example, a set of
diagrams, flows, etc.; Target users – general public or analysts; visualization designs showing different aspects from MOOC
Applications – journalism, politics, finance, sports, advertise- forums, including posts, users, and threads at different scales
ments, etc. Our PIHVI system can be categorized as using is considered, with an aim to support online education with the
interactive hierarchical scatter plots to analyze text content of forums [5]. A recent work [19] focuses on revealing anoma-
general online forums for analysts, with the goals of extracting lous information spreading occurring in social media along the
interesting patterns of threads, detecting outliers of threads time-line with coordinated views, which particularly contains
like spam or novel topics, and advertising, to name a few. Be- a scatter plot of threads created from Multidimensional scaling
low we summarize related work on using other visualization (MDS) [1] without the hierarchical functionality. We provide
techniques to analyze aspects of online discussions. views that help gain comprehensive understanding of ongoing
discussion and how it relates to a multi-level hierarchy of the
Semantic features of threads in online forums. Bag-of- whole discussion venue, rather than on relationships within
word (BoW) vectors like TF-IDF are typical to represent text individual threads; moreover our focus is on mostly anony-
content for analysis [9, 13]. Topic facets to extract sentence mous discussion where most users are unregistered so user or
level sub-topics are considered [6], The facets are shown to user-group views are less useful.
reflect programming concepts for questions in programming
discussion forums, visualized in a prototypical system. An
alternative hierarchical semantic model for sentences incor- OVERVIEW OF THE SYSTEM: PIHVI - POSTING ANALYSIS
porating discrete Fourier transform with a mixture of topic WITH INTERACTIVE HIERARCHICAL VISUALIZATION
models was proposed [4]. In this first publication on PIHVI, We now describe the proposed system PIHVI - Posting Analy-
we focus on the main goal of studying content relationships sis with Interactive Hierarchical Visualization, on an overview
to the known hierarchy within the venue, and in order not to level, describing the design principles and how the system
muddle that focus we use simple BoW-based features which operates. In the next section we describe how the system is
already work well in capturing thread similarities in our experi- implemented through dimensionality reduction methods.
ments. Other derived features like topical features are possible,
Design Principles. The design principles of the PIHVI sys-
the rest of our method works for any choice of features.
tem are: 1. The system should illustrate, in a compact way,
Showing posts from individual threads. In PIHVI, we em- the overall variety of discussion over the online forum, both
phasize exploration based on the content-based hierarchical in terms of semantic content, and the hierarchical variety of
scatter plot and the related tasks, thus we use a simple detail discussion sections where threads can be posted. This answers
view showing plain texts from selected threads. However, we the question: “Which kinds of topics are users discussing,
are aware of works that visually display thread content. This and in which sections is the discussion happening?” 2. The
line of research usually focuses on visualizing the structure system should let analysts efficiently browse semantic content
of the thread under inspection. For example, thread structures of threads across the forum, on an overall level of trends and
can be encoded with indented rectangles [3], where each in- on more detailed levels of individual thread groups, without
dentation corresponds to one level of nested reply to a post. restricting browsing to boundaries of specified sections. This
Such thread visualization is orthogonal to PIHVI novelties and answers the question: “Which discussion trends are individual
can be easily integrated into PIHVI. threads of interest related to?” 3. The system should compactly
show relationships of threads in terms of semantic content and
Dimensionality reduction in visual analytics. Dimension- position in the hierarchy. This answers the question: “What
ality reduction has been brought into the pipeline of visual other similar threads exist for a thread of interest?”
analytics by providing a 2D representation of threads, to help The above design principles should hold even when the num-
users to gain an overview. It is related to semantic feature con- ber of threads, the number of sections in the hierarchy, and the
struction discussed above, if obtained features are given to di- depth of the hierarchy, all become large. The resulting system
mensionality reduction algorithms. For example, t-distributed for visual analytics should support key tasks in Shneiderman’s
Stochastic Neighbor Embedding (t-SNE) [7] has been consid- [15] taxonomy of tasks for information visualizations, includ-
ered [16]. In PIHVI, we build our hierarchical exploration ing the visual information seeking mantra overview first, zoom
functionality on top of a t-SNE based embedding. See Section and filter, details on demand, and relate.
4 for details. Another branch of studies in this line is to let a
user personalize the representation by interaction. See, e.g., System Components. Following the design principles in Sec-
our work in [11, 10, 12] and references in [14]. We point out tion 3, we devise the following components for large-scale
the novel goal of PIHVI is to study relationships of large-scale interactive analysis on hierarchical forum posts, and to support
content to a large-scale hierarchy, hence the novelty is in in- our user tasks in PIHVI. Screen shots of the system are shown
corporation of the hierarchy into the content visualization and in Figure 1. The system has five linked views, where views
in hierarchical exploration of content through multiple views. 1-3 below are always shown and 4-5 are on demand:
1. Content-based map: An interactive scatter plot of the en-
Forum analysis system combined with different visual an- tire collection of threads from the online forum, which can
alytics. There have been integrated systems with views of be zoomed to several detail levels. The map shows a spatial
tion distributions: higher opacity reflects a higher density of
threads and vice versa; color mixing suggests mixtures of dom-
inating sections sharing similar content. Both patterns may
interest a forum analyst/administrator. For interactivity, each
dot, from different zoom levels, can be inspected by clicking
to show a pop-up: the pop-up may contain either the thread
title and the name of its section, if the dot represents a single
thread, or the name of the dominating common parent section,
if the dot represent a group of threads. To avoid occlusions of
multiple pop-ups, the analyst can bring a pop-up to front or
back by clicking or dragging it; also, when a pop-up is mouse-
overed, the remaining pop-ups without the cursor hovered will
turn half-transparent. Buttons on this view allow the analyst
to drag a rectangle over an area of the map: all threads in the
area will be described in the Threads and Sections panel.
2. Thumbnail view of the map: This view shows the entire
collection and a rectangle denoting which portion of the map
an analyst is currently viewing; this lets the analyst maintain
focus and context while browsing the collection.
3. Thumbnail view of the thread hierarchy: This view shows
a radial plot of the tree hierarchy of discussion sections, and
shows how optimized colors of the sections vary across the
hierarchy. Due to space constraint, showing all 2434 section
names at once is not feasible: we show section names on de-
mand when the analyst mouse-overs the tree nodes of sections.
The window size can be enlarged by standard zoom controls
to help mouse-over the nodes. This view allows the analyst
to relate where each thread or group of threads seen in the
content view arises from in the hierarchy. Buttons on this view
allow the analyst to drag a rectangle over a subset of sections:
those sections will be highlighted in the content-based map.
4. Threads and Sections panel: When the analyst has selected
an area of the content map, all threads in that area will be listed
here as a scrollable list. The panel has two tabs: the Thread
tab lists the threads by title and section, and the Sections tab
lists all unique sections from which the threads arise.
Figure 1: The PIHVI interface. Top: the initial state, where 5. Thread Information panel: When the analyst clicks any
the content-view zoomed out to show an overview of the entire individual thread in the content-based map (individual dot that
thread collection. Middle: section-filtering state. An analyst represents a single thread in the scatter plot) or clicks the title
selects a subset of sections in the section panel, circles of of any individual thread in the Thread tab of the Threads and
corresponding sections are then highlighted. Bottom: zoomed- Sections panel, the full text of that thread is shown here.
in state for an analyst exploring a cluster of hearing-related
threads from different sections across the forum; the analyst Walkthrough example. Here we depict a typical usage of
has selected a group of threads with a bounding box and is PIHVI for thread discovery. Suppose an analyst is interested
viewing their titles and content of an example thread. in sport and fitness related discussion but does not yet know
how such discussion is related to other discussion within the
Suomi24 forum. This can be a possible sequence of findings
organization of threads by content similarity: similar threads and interactions from the analyst when discovering the threads
are shown nearby. The map is created by machine learning of interest: 1. Initially the analyst views the content view
based dimensionality reduction. The map is overlaid with zoomed out to show all Suomi24, and notices several clusters
colors representing sections and their similarity, again created of color, indicating discussion similar in content and arising
by dimensionality reduction. At detailed zoom levels, each from tight branches of the section hierarchy.
thread is shown as a dot, and colors of the dots indicate the 2. The analyst clicks some of these clusters to find a starting
section of the threads. At less detailed zoom levels, groups of point for analysis. After a time the analyst find a green cluster
threads are shown as dots with diameters scaling with the num- (at the bottom in Figure 1 top) whose pop-up label shows the
bers of threads in the groups, and colors of the dots indicate a dominating section in the cluster is Sport and Fitness (“Urheilu
dominating common parent section of threads in the group, as ja kuntoilu”). Alternatively, the analyst could have browsed
detailed later. We draw the different-sized dots with moderate the section tree to find a Sport and Fitness section.
transparency, yielding color blending when there are overlaps 3. The analyst then zooms in to view individual threads within
between the dots, which approximates the content and sec- the cluster, and draws a rectangle around them to quickly view
Properties of the System. As desired in Design Principles,
the proposed system directly supports several key tasks in
Shneiderman’s [15] taxonomy, in particular the visual infor-
mation seeking mantra is supported as follows:
Overview: The two linked views (content-based map and the
tree of the section hierarchy) together show an at-a-glance
overview of threads in the entire online forum. The radial tree
view shows an at-a-glance overview of the section organiza-
tion: sections with numerous child sections are shown as wide
branches, and sections with multiple levels of subsections un-
der them are shown as long branches. On the other hand the
Figure 2: Possible interactions when an analyst is discovering content-based map shows an at-a-glance overview of seman-
threads of interest. Left: the analyst has used section filtering tic content across the whole forum, organized so that similar
to discover several locations where Sport and Fitness (“Urheilu content is shown nearby in the map, and groups of similar
ja kuntoilu”) is discussed (green highlighted clusters). Right: threads appear as clusters; moreover, section colors overlaid
the analyst zoomed in towards the middle-right green cluster, on the map provide an easy overview of which sections have
and inspected nearby other clusters: near the green Health and similar content, which sections have varied content appearing
Fitness cluster (“Urheilu ja kuntoilu”) are two clusters, Health in multiple places on the map, and what the lowest common
(“Terveys”; violet color) and Dogs (“Koirat”; red color). hierarchy section is for some cluster of content.
Zoom and filter: the analyst can zoom into the levels of detail
in the content-based map with the mouse wheel, and can filter
threads from both linked views with selection boxes, e.g. to
select threads in a particular location in the content map and
highlight threads from a particular branch of the section tree.
their titles. The analyst discovers many of them are about Details on demand: clicking on threads in the content based
ice hockey, a popular sport in Finland, such as threads about map shows their content, clicking on thread-clusters gives the
Finnish ice hockey teams and leagues and about related sports name of their lowest common section, and creating a selection
like “Jääpallo” (Bandy in Finnish). box over an area of the content map opens a linked view listing
4. The analyst draws a rectangle around the Sport and Fit- all threads in the area and all sections of those threads; and
ness section and its nearby sections, to detect where else such clicking any thread in that list shows the thread content.
threads occur in the content map: Figure 2 (Left) shows that Relate: the content-based map relates threads by their con-
besides the ice hockey related cluster at the bottom, there are tent similarity by placing similar threads nearby in the map;
small isolated clusters (light green dots) at edges of the map, the section tree relates sections by showing their parent-child
and a large cluster at center right in the map. Thus the analyst relationships; and the overlay of section colors onto the map
discovered a new cluster of Sport and Fitness discussion. relates similarity of thread content to similarity of their sec-
5. The analyst removes the section filtering since he/she is in- tions. Inspecting these two linked views lets analysts quickly
terested in all sections in the area. The analyst zooms towards study how threads and sections at different branches in the hi-
the discovered cluster, and notices it is located at the border erarchy relate to each other in terms of content, and study how
of two larger clusters (Figure 2 Right): clicking the clusters a cluster of semantically similar content is related to multiple
reveals their dominating sections as Health (“Terveys”; violet sections where discussion of the content happens in the forum.
colored cluster) and Dogs (“Koirat”; red colored cluster).
6. Since the three clusters are close-by, this indicates they may As noted above, PIHVI includes filtering/highlighting of the
have common topics. The analyst is then interested what com- content map by selected sections. Highlighting could also be
monalities there are in threads arising from Sport and Fitness done by results of text lookup searches; we do not focus on
and in threads arising from Health, so the analyst zooms in simple text search for several reasons: 1. Our focus is not on
more on the plot, and draw a rectangle around the boundary searching individual documents but on overall comprehension
between “Terveys” and “Urheilu ja kuntoilu”. of the forum where information needs of the user evolve as
7. The topics of the selected threads are listed in the detail he/she learns about the data; such comprehension would not
panel; the analyst finds several threads on weight-loss in the be well served by individual text lookups. 2. Naive text search
information panel. does not take into account complexity of natural language and
8. The analyst could continue investigating Sport and Fitness could miss related threads if they did not share search terms
(e.g. moving to view some of the other small clusters of Sport (due to synonyms etc.), whereas the content map is organized
and Fitness threads), or could e.g. investigate commonalities by overall similarity of all words in the documents, and serves
between Health and Dogs in the same window–this would turn exploration of discussion content better than simple search
out to be discussion of allergies when living with dogs. would. Nevertheless, where text lookup is needed it is trivial
The above described interaction can go on as long as desired to highlight lookup results like we do to selected sections.
by the analyst. For an analyst, e.g., in the forum administration
We will compare PIHVI to a baseline system we implemented
team, such exploration may help understand the distribution
to represent a more traditional interactive section-based brows-
of the threads, which can inspire, for example, reorganization
ing interface, described in the User Experiment section.
of forum sections, or a digest on hot discussion topics.
DIMENSIONALITY REDUCTION FOR THE PIHVI SYSTEM other in the section tree) get close-by colors. This color en-
We describe the dimensionality reduction solution for creating coding is a visual cue helping users distinguish sections based
the two main views – the content-based scatter plot and its on their locations in the hierarchy. We do not require users to
color-based link to the section tree view, and then describe how perceive section distances exactly from colors; it is enough to
zooming is implemented efficiently and using the hierarchy notice similar colors (close-by sections) when browsing the
for summarization of thread groups. content. We take path lengths between sections in the section
tree as input distances, and give weights to atomic paths –
Content-based Representation of Textual Data. PIHVI is paths connecting a section to its direct parent or child section
initiated by a content-based two-dimensional scatter plot of with wi↔ j = (Ci +C j )/2, where Ci is the number of child sec-
threads on the online forum (here Suomi24), followed by tions of section i. Intuitively, if one end of the path has a large
other interface components facilitating exploratory data anal- number of child sections, the parent is not connected to any
ysis on the forum. In PIHVI, dots are discussion threads particular one of its child sections, implying a looser connec-
in the “content-based” scatter plot. By “content-based”, we tion with the other end. The distance between two sections is
mean to pursue a two-dimensional representation such that then the sum of weights of atomic paths connecting them. The
1. if two threads are similar in content, they should be placed distances are reduced to coordinates in a 3D output space by
nearby; and 2. if two threads are dissimilar in content, they multi-dimensional scaling (MDS): the 3D output coordinates
should be placed faraway. We use BoW models to capture are then normalized to the unit cube where they can be used as
(dis)similarities between content of threads: the more words RGB color coordinates (more sophisticated color encodings
two threads share, the more similar they are in content. We ex- from perception theory are possible [8]). As a result, sections
tract thread features based on BoW as follows, and perform a close to the root get mild colors, with deepening saturation
nonlinear dimensionality reduction with neighbor embedding. for more specialized lower-level sections. We empirically find
BoW-based features in high-dimensional space. We MDS can yield a more continuous coloring than non-distance-
start with lemmatized tokens of Suomi24, provided by the preserving dimensionality reduction algorithms like t-SNE.
Finnish Language Bank service (“Kielipankki”, http://www. Zooming in the Content-based Plot. At the most detailed
kielipankki.fi). The resulting data set contains the lemma-
level in the content plot, threads are individual dots and the
tized version of each token in each thread, along with other user can easily see their content similarities from their spatial
meta-data of the token. We first calculate the TF-IDFs for each relationships and their section similarities from their colors.
token, constituting raw features of each thread. We further When the user zooms out, he/she should similarly be able to
process the raw features as follows: 1. Remove stop-words see content and section similarities of larger groups of threads.
based on a Finnish stop-word list; 2. Only keep tokens that However, naive zooming of a large scatter plot would not ac-
appear in at least k threads, to discard incidental tokens that complish this: firstly, in large discussion forums it would not
do not carry shared semantic content, such as rare URLs or be feasible to redraw huge numbers of threads into a scatter
non-word character strings. 3. For each thread, keep only l plot quickly; secondly, in areas where threads arise from mul-
tokens with highest TF-IDF values. With stop-words already tiple sections the zoomed-out view would become a clump of
removed, we assume these kept tokens are more informative different-colored dots, making it hard to notice overall simi-
than others in each thread. 4. Normalize each obtained feature larities in sections of such threads. We solve both problems
vector by dividing vector entries with their sum. We feed by an intelligent zoom. The plot area is divided into a grid
the final features into t-SNE to obtain the two-dimensional (the higher the zoom-out factor, the rougher the grid). Instead
content-based representation for PIHVI. Considering the size of plotting individual threads we plot one circle per grid cell,
of our corpus, t-SNE is a suitable choice since 1. t-SNE has summarizing all threads in it: the circle radius indicates num-
been shown to work well with high dimensional data; 2. scal- ber of threads in the cell, and circles are drawn with slight
able t-SNE variants exist. We use a tree-based implementation translucency to allow smooth overlap instead of occlusion.
[18] to achieve a O(N log N) complexity for N threads. Grid cells are indexed by Hilbert curves in each zoom level
Section Distance based Coloring for Linking the Views. In for better performance in querying points in cells. For each
the thumbnail view of the section hierarchy, we show a radial grid cell the color of the circle must be chosen intelligently
tree plot of the hierarchy and draw section nodes in differ- as the grid cell contains threads from multiple sections: we
ent colors so the analyst can distinguish the sections and link compute the color by starting from each thread and travel-
them with the content-based map. There are a huge number ing up the section hierarchy until we find the lowest section
of sections, and giving each a completely different color is covering at least 40% of threads in the grid cell, denoted the
not possible, thus assigning colors to successfully link the two lowest dominating section. Thus the color of a circle indicates
main views is nontrivial. We assume the analyst wishes to diversity of content in it: the less saturated (i.e., the closer to
notice similar content from far-apart sections of the hierarchy, the color of the root section), the more diverse the content.
and highly different content from close-by in the hierarchy, Such adaptive coloring lets the user quickly browse forum
hence color differences should correspond to distances in the content at multiple detail levels. At each level, circles, their
section hierarchy. We use dimensionality reduction to opti- colors, and their overlap yield an at-a-glance view of content
mize colors so that the most important differences are encoded and section variation. As the user zooms out, areas of mixed
with the most different colors, and close-by sections in the sections become large diffuse circles colored ever closer to
hierarchy (having small path length from one section to the the root color; as the user zooms in, such circles break into
subclusters with more vibrant colors, indicating thread groups but lacks the dimensionality reduction based content view and
arising from tight branches of the section hierarchy. focuses on the section hierarchy only, allowing the user to
browse it in a large resolution. This baseline represents a
USER EXPERIMENT AND RESULTS system that would be readily implementable without having
We perform a user experiment to show the advantage of PIHVI access to a dimensionality reduction based visualization of
compared to a baseline, in controlled information seeking sce- thread content and thread relationships. In the baseline system
narios representing typical analytics subtasks. We describe the there are four linked views: 1. Scrollable large-scale radial
experiment design (the forum and thread corpus, task design, plot of the section tree hierarchy: each node in the tree corre-
baseline system we designed&implemented, participants, and sponds to one section of the forum.
evaluation criteria) and then the results. 2. A thumbnail of the whole tree, with a square control for
navigating inside the tree; moving the square over the thumb-
Details of the Online Forum and Corpus. We run our user
nail changes the corresponding position in the large-scale plot
experiment on Suomi24, one of Finland’s largest online com-
of the tree. This thumbnail view is similar to the one in PIHVI,
munities: it is mostly in Finnish with several Swedish sections;
but focuses on navigation instead of filtering a content-based
in 2015 it had 0.8 million unique visitors per week1 . We con-
plot which is not available in the baseline system.
sider the realistic scenario where analysts are analyzing data
3. A panel that appears when the user clicks on any section in
of a specific time frame: here we filter data to the subset of
the tree. It shows the thread titles from the selected section.
thread from 2005; our methods can naturally be used for data
This panel is similar to the threads tab in the PIHVI system.
of any other years or for all of Suomi24. We set k = 20 and
4. A window for full-text content of threads; clicking any
l = 200 in the preprocessing told in BoW-based features in
thread in the title panel shows the full-text content, again simi-
high-dimensional space. We then have 314871 threads and
lar to the corresponding panel in PIHVI.
80123 tokens in the corpus, arising from 2434 sections.
The thumbnail window and the window containing the thread-
Task Design. In order to measure the success of our inter- list panel and full-text panel are movable by dragging. Figure
face for visual analytics of hierarchical online discussion, we 3 shows a screen shot of the baseline system. Essentially, the
evaluate performance in a typical subtask that would often be baseline corresponds to PIHVI without a content map, thread
present as part of an analytics session: given a thread of in- content of sections is then naturally accessed through the sec-
terest, find semantically related threads across the online tion hierarchy tree. This makes the two systems comparable.
forum. We point out that given a thread of interest, it is trivial
to browse other threads in the same section, and therefore we
focus on the nontrivial task of finding related content from
other sections across the online forum. The ability to find
related content comprehensively across an online forum is
crucial for comprehensive analytics of online discussion and
how topics of interest are spread across the forum.
We choose four cases where discussion on a semantic topic
spreads across several Suomi24 sections: for each task, we
indicate to the user one thread as a starting point, and ask to
find threads on the same topic from other sections. Users carry
out a training task to learn the interfaces: Kuulo (“hearing” in
Finnish; the task is to find threads about hearing outside of
the discussion section dedicated to hearing). We pick three Figure 3: A state of the baseline system. Components are
tasks for the actual experiments: 1. Matti Nykänen (a Finnish marked with labels: the tree of the section hierarchy, a thumb-
ski jumper; the task is to find threads about him outside of the nail of the full tree with a draggable control for navigation, and
discussion section dedicated to him), 2. Svenskt (“Swedish” a thread information panel showing thread titles and content.
in Swedish; the task is to find discussions in Swedish out- In the figure, the threads in the information panel are from the
side of discussion sections dedicated to Swedish-language selected section “Häävalmistelut” (“Wedding preparation” in
discussion), and 3. Hiukset (“hair” in Finnish; the task is to Finnish, marked in black in the section tree).
find discussion about hair/hair loss/hair care/hair fashion out-
side of the section dedicated to hair). Tasks involve several
words of interest due to informal names, loose language, etc., Participants. We perform a user study with 12 participants,
and are not fully solvable by naive text search. All the tasks including 1 female and 11 males; all participants are computer
correspond to the research questions we ask in Section 3. science PhD students or post-doctoral researchers who know
both Finnish and Swedish. Each participant first performs a
The Baseline System. We compare the proposed system in training stage on both systems by using the systems to perform
terms of task performance and usability against a baseline the training task. Next, the participant performs all three
system. The baseline is an interactive system for browsing experiment tasks on both systems (PIHVI and baseline). The
threads by interacting with a radial visualization of the section order of the task and system combinations (which order of the
hierarchy. It is comparable to the proposed PIHVI system, three tasks; which system first in each task) was randomized
1 As reported in https://goo.gl/Ukt49K. and counterbalanced over the participants to avoid bias.
Evaluation Measures. We evaluate the experiment results in Baseline PIHVI
four ways. First, we inspect the results qualitatively: is the Mean (Std) 1.045 (1.222) 4.137 (1.359)
PIHVI system generally better in the opinions of the partici-
pants? Secondly, we quantitatively measure task performance: Table 1: Means (over tasks and users) and standard deviations
how many relevant threads was the user able to find in each of quality measures R calculated from retrieved threads. A
task, compared to the amount of time spent? We measure this larger mean suggests better relevance.
as a proportion of total relevant threads to time spent, averaged
over users, for each task. Thirdly, we measure user experi- System Participant System×Participant
ence for both systems by a questionnaire asking users to rate p-value 6.82 × 10−13 0.938 0.440
their agreement to statements about the system on a 5-point
Likert scale from strong disagreement to strong agreement; Table 2: Two-way ANOVA for relevance of retrieved threads.
the questions are based on the standard ResQue questionnaire We have p < 0.05 in the system effect; the 95% confidence
with some modification for the online forum domain, and are interval of the PIHVI system coefficient is [0.379, 4.819], fully
listed in Table 3. Fourthly, after the tasks we also ask each above zero. Thus PIHVI performs significantly better than the
user which system they preferred overall. baseline in terms of relevance of retrieved threads.
Results
Qualitative analysis. We collect qualitative descriptions on results are in Table 2. We observe the system effect is statis-
the systems by allowing participants to give free-form written tically significant in this ANOVA, suggesting that PIHVI is
or oral feedbacks. Overall, the participants agree the tasks significantly better than the baseline system when measuring
are clear. Besides good quantitative performance in tasks, the relevance of the retrieved threads over time spent.
participants made discoveries, hence PIHVI helped them com-
prehend the discussion venue. One participant stated he dis- User responses to post-task questionnaire. Table 3 shows
covered Swedish is sometimes used for trolling in Suomi24 the distribution of answers to the post-task questionnaire. The
when he was doing the task “Svenskt” on PIHVI. In task questions can be divided into two groups, 13 positive questions
“Matti Nykänen”, one participant noted PIHVI can put threads (higher user agreement is better) and 5 negative questions
on Matti Nykänen together, though people in the threads call (lower user agreement is better). Roughly, positive questions
him different names. Another participant said discussions on cover overall user experience and ability to perform tasks with
Matti Nykänen in marriage related sections reminded him of the system, negative questions focus on ease of use. Half of the
18 questions yielded statistically significant differences. All
the ski jumper’s series of short-lived marriages. Participants
statistically significant differences are in favor of the PIHVI
commented PIHVI can give them threads from diverse sec-
system. Table 3 lists in detail the p-values showing statistical
tions, compared to the baseline. Participants remarked that the
significance (Q1, Q8, Q9, Q13–Q18) at the level of 0.05.
baseline system is reasonable, and one pointed out it is similar
to how people usually browse the forum; the only difference Users wanted to use PIHVI frequently more than the baseline
is that in the baseline we organize sections radially, not lin- (Q1), found PIHVI less cumbersome than the baseline (Q8),
early. As for the section filter, one participant stated it helps felt more confident using PIHVI than the baseline (Q9), found
understand section distributions when the map is zoomed-out. it more easy with PIHVI to find what they were interested in
Lastly, though some of them agreed that PIHVI is slightly (Q13), found it more easy to find a thread to read with PIHVI
more complicated than the baseline, the learning curve of (Q14), would use PIHVI more than the baseline to find threads
PIHVI is not steep, so they can learn to use it fairly quickly. to read (Q15), would use PIHVI again more than the baseline
(Q16), would use PIHVI frequently more than the baseline
Task performance. We evaluate task performance of partici- (Q17), and would prefer PIHVI over other analytics systems
pants in terms of relevance of retrieved threads and time spent. more than they would prefer the baseline over other systems
To evaluate relevance of threads retrieved by the participants, (Q18). Thus overall, participants report they can find what
a native Finnish speaker assessed all retrieved threads without they are interested in easily with PIHVI, and would like to use
knowing which system they were from, rating them on a three- it to do exploratory data analysis frequently (Q1, Q13-Q18).
point scale from 0 to 2 for non-relevance, partial relevance, Also, PIHVI is easier to use and gives the participant more
and full relevance. For each task on each system performed by confidence during the exploration compared with the baseline
each participant, we use the quantity R = ∑i ri /T as the qual- (Q8-Q9). The confidence may arise because PIHVI’s layout
ity measure for this (task, system, participant) combination, reflects thread similarity better so that in PIHVI the relevance
where T is the number of minutes taken to finish the task, ri is between the found threads and the tasks is clearer than in
the rated relevance of the i-th thread, with i going from 1 to the the baseline. For brevity we simply list the questions that
number of retrieved threads in the combination. The measure did not yield significant differences: Q2 I found the system
R can be interpreted as the amount of “retrieved relevance in unnecessarily complex; Q3 I thought the system is easy to use;
unit time”. A larger R indicates that a larger number of more Q4 I think that I would need the support of an analytic person
relevant threads are retrieved in a shorter time. We collect the to be able to use this system; Q5 I found the various functions
R values from all results of the participants on the two sys- in this system are well integrated; Q6 I thought there was
tems, and perform a two-way analysis of variance (two-way too much inconsistency in this system; Q7 I would imagine
ANOVA; “participant”× “system”). Means and standard devi- that most people would learn to use this system very quickly;
ations of the R values are shown in Table 1. And the ANOVA Q10 I needed to learn a lot of things before I could get going
Baseline PIHVI p (Welch’s
P/N ID Question mean (std) mean (std) t-test)
+ Q1 I think that I would like to use this system frequently 1.917 (0.862) 3.750 (0.433) 9.866 × 10−6
- Q8 I found the system very cumbersome to use 3.167 (1.280) 2.000 (0.816) 0.0198
+ Q9 I felt very confident using the system 3.083 (1.187) 4.000 (0.816) 0.0480
+ Q13 Using this system to find what I am interested in is easy 1.750 (0.829) 4.083 (1.115) 1.780 × 10−5
+ Q14 Finding a thread to read with the help of this system is easy 2.750 (1.090) 4.083 (1.037) 7.604 × 10−3
+ Q15 I will use this system to find threads to read 2.083 0.954) 3.333 (0.850) 3.757 × 10−3
+ Q16 I will use this system again 2.083 (0.862) 3.167 (1.143) 0.0206
+ Q17 I will use this system frequently 1.833 (0.799) 2.917 (1.115) 0.0165
I will use this system rather than other systems of exploratory
+ Q18 2.167 (0.898) 3.583 (0.640) 3.852 × 10−4
data analysis for online forums
Table 3: The questions yielding statistical significance in the post-questionnaire, with mean sand standard deviations of user
agreement with the questions. For “positive” questions (Q1, Q3, Q5, Q7, Q9, Q11, Q12, Q13, Q14, Q15, Q16, Q17, Q18) higher
agreement is better, for “negative” questions (Q2, Q4, Q6, Q8, Q10) lower agreement is better.
with this system; Q11 The layout of this system interface is 8. T. Munzner and E. Maguire. Visualization analysis and
attractive; Q12 The layout of this system is adequate. design. CRC Press, 2015.
User preference. Users were also asked which system they 9. A. I. Obasa, N. Salim, and A. Khan. Hybridization of
overall prefer: all users preferred the proposed PIHVI system. bag-of-words and forum metadata for web forum
question post detection. Indjst, 8(32), 2016.
CONCLUSIONS
We presented PIHVI – Posting Analysis with Interactive Hi- 10. J. Peltonen, K. Belorustceva, and T. Ruotsalo.
erarchical Visualization, a novel visual analytics system for Topic-relevance map: Visualization for improving search
hierarchical online forums based on multiple linked views. result comprehension. In ACM IUI 2017, 2017.
The main view (interactive thread content scatterplot) and 11. J. Peltonen, M. Sandholm, and S. Kaski. Information
the coloring that links it to another prominent view (radial retrieval perspective to interactive data visualization. In
tree of sections) are created by nonlinear dimensionality re- Eurovis 2013 short papers, 2013.
duction from high-dimensional similarities between threads
and between sections, respectively. A user study on a large 12. J. Peltonen, J. Strahl, and P. Floreen. Negative relevance
Finnish online forum shows significant advantages compared feedback for exploratory search with visual interactive
to a baseline without dimensionality reduction based views: intent modeling. In ACM IUI 2017, 2017.
users found more relevant results with PIHVI, and preferred 13. Z. Qu and Y. Liu. Finding problem solving threads in
it over the baseline, with statistical significance. The system online forum. In IJCNLP 2011.
helps analysts find and relate discussion content across large
hierarchies, an important need as forums keep growing. 14. D. Sacha, L. Zhang, M. Sedlmair, J. A. Lee, J. Peltonen,
D. Weiskopf, S. North, and D. A. Keim. Visual
REFERENCES interaction with dimensionality reduction: a structured
1. I. Borg and P. J. F. Groenen. Modern Multidimensional literature analysis. IEEE TVCG, 23(1):241–250, 2016.
Scaling: Theory and Applications. Springer, 2005. 15. B. Shneiderman. The eyes have it: A task by data type
2. S. Chen, L. Lin, and X. Yuan. Social media visual taxonomy for information visualizations. In IEEE VL
analytics. Computer Graphics Forum, 36(3), 2017. 1996.
3. K. Dave. Flash forums and forumreader: Navigating a 16. A. J. Soto, R. Kiros, V. Kešelj, and E. Milios. Exploratory
new kind of large-scale online discussion. In CSCW 2004. visual analysis and interactive pattern extraction from
semi-structured data. ACM TiiS, 5(3):16:1–16:36, 2015.
4. J. Duan, J. Zeng, and S. Zhang. Hierarchical semantic
model for objectionable web text content detection. In 17. J. Teevan, C. Alvarado, M. S. Ackerman, and D. R.
IEEE ASID 2012. Karger. The perfect search engine is not enough: A study
5. S. Fu, J. Zhao, W. Cui, and H. Qu. Visual analysis of of orienteering behavior in directed search. In CHI 2004.
mooc forums with iforum. IEEE TVCG, 23:201–210, 18. Z. Yang, J. Peltonen, and S. Kaski. Scalable optimization
2017. of neighbor embedding for visualization. In ICML 2013.
6. I.-H. Hsiao and P. Awasthi. Topic facet modeling: 19. J. Zhao, N. Cao, Z. Wen, Y. Song, Y.-R. Lin, and
Semantic visual analytics for online discussion forums. In C. Collins. # fluxflow: Visual analysis of anomalous
LAK 2015. ACM. information spreading on social media. IEEE TVCG,
7. L. v. d. Maaten and G. Hinton. Visualizing data using 20(12):1773–1782, 2014.
t-sne. JMLR, 9(Nov):2579–2605, 2008.