PIHVI: Online Forum Posting Analysis with Interactive Hierarchical Visualization Jaakko Peltonen∗ , Ziyuan Lin∗ , Kalervo Järvelin∗ , and Jyrki Nummenmaa∗ jaakko.peltonen@uta.fi, ziyuan.lin@uta.fi, kalervo.jarvelin@uta.fi, jyrki.nummenmaa@uta.fi ABSTRACT or administrators of the venue. Online forums are big data: We introduce PIHVI: a novel interactive system for visualizing as a case study we use a popular Finnish discussion forum and exploring a large hierarchical text corpus of online forum “Suomi24” (www.suomi24.fi) spanning 16 years and 6.5 mil- postings. The main view of the visual interface shows a large- lion threads. Hierarchically organized discussion occurs also scale scatter plot, created by flexible nonlinear dimensionality for example in user commentaries on news sites whose news reduction based on text contents of the postings, and we couple have a hierarchy, in online reviews of products that have a hi- it with a coloring optimized by a second dimensionality reduc- erarchy, and so on; our work is applicable to such hierarchies tion to represent the forum hierarchy. We exploit the hierarchy as well but we focus on the discussion forum case. to provide data-driven summaries of plot areas at multiple lev- In discussion forums, sections and their hierarchy are created els of detail, allowing the user to quickly see and compare both by administrators as a simplified division of discussions in- the content-based similarity of groups of posts and how near tended to represent a subset of prototypical user interests, and they arise in the forum hierarchy. A user can move between such a division does not suffice to describe the true variety of hierarchy levels, mark posts or spots of interest, filter posts by semantic content in online discussion. Our target users are ad- content similarity and by location within the hierarchy, and ministrators and analysts of the forums: if the hierarchy does inspect post contents. Experiments show the interface can not support the content, forum owners can add subsections or reveal hidden semantic relationships between postings that shortcuts to promote content, aiming to increase user engage- would be hard to find based on the known hierarchy alone. ment with the venue. An important task in data analytics of ACM Classification Keywords online forums is to perform visual analytics of how discus- D.2.2 Software Engineering: Design Tools and Techniques sions and their underlying topics spread across the hierarchy. User interfaces; H.5.2 Information Interfaces and Presenta- When the data and the hierarchy are large this becomes a hard tion: User Interfaces User-centered design; I.2.7 Artificial task, as exhaustive inspection of all data or comparisons across Intelligence: Natural Language Processing Text analysis all sections become unfeasible: computational support is then needed. However, most visual analytics tools for social media Author Keywords discussion focus on other aspects such as temporal evolution user interaction; visual analytics; text analysis. or networks of conversation participants. We propose a new system for analysis of online conversation in INTRODUCTION hierarchical forums in a large scale, by an interactive interface Online discussion often takes place in venues that have a hier- that scales to large data (Figure 1; details in later sections). We archical organization, such as large message boards organized focus on analysis of content variation and its relationship to the in sections, subsection, sub-sub-sections and so on. While hierarchical sections; we do not focus on e.g. temporal filter- some venues such as specialized message boards may use a ing or other such aspects which are extensively studied in other relatively flat organization having a small number of sections systems and can easily be integrated as additional views for directly under a root, other venues such as general-interest our system. Main novelties: first work to use dimensionality message boards have an extensive multilevel hierarchy form- reduction based linking of views through optimized coloring; ing a deep tree of sections. Users post new threads, and post among the first to use large-scale dimensionality reduction in comments and replies under threads, but the hierarchy of sec- visual analytics of discussion venues; novel multi-level inspec- tions is typically designed by owners of the discussion venue tion of a large-scale scatterplot; novel use case - analysis of * All the authors are affiliated with University of Tampere, Finland. suitability of existing hierarchy in a venue to describe content The work was supported by Academy of Finland decisions 295694 of the venue. We next discuss related work, introduce the and 313748 and by Aalto Science-IT. system, denoted PIHVI - Posting Analysis with Interactive Hi- erarchical Visualization, then describe our machine learning solution, describe a user study and discuss conclusions. RELATED WORK Visual analytics for social media has been an active research topic, as social media data are rich and heterogeneous in struc- ©2018. Copyright for the individual papers remains with the authors. ture and thus difficult to model. A recent survey [2] taxono- Copying permitted for private and academic purposes. mized methodologies on the field by Data sources – Twitter, ESIDA ’18, March 11, 2018, Tokyo, Japan Facebook, blogs, online forums, etc.; Entities of data – net- various visual analytics for inspecting online forums. Sim- work, geographic information, or text content; Goal of visual- ple text search does not suffice when users aim to broadly ization – pattern extraction, visual monitor, anomaly detection, comprehend the information space with multiple exploratory etc.; Visualization techniques – scatter plots, trees, node-link searches and information needs [17]. For example, a set of diagrams, flows, etc.; Target users – general public or analysts; visualization designs showing different aspects from MOOC Applications – journalism, politics, finance, sports, advertise- forums, including posts, users, and threads at different scales ments, etc. Our PIHVI system can be categorized as using is considered, with an aim to support online education with the interactive hierarchical scatter plots to analyze text content of forums [5]. A recent work [19] focuses on revealing anoma- general online forums for analysts, with the goals of extracting lous information spreading occurring in social media along the interesting patterns of threads, detecting outliers of threads time-line with coordinated views, which particularly contains like spam or novel topics, and advertising, to name a few. Be- a scatter plot of threads created from Multidimensional scaling low we summarize related work on using other visualization (MDS) [1] without the hierarchical functionality. We provide techniques to analyze aspects of online discussions. views that help gain comprehensive understanding of ongoing discussion and how it relates to a multi-level hierarchy of the Semantic features of threads in online forums. Bag-of- whole discussion venue, rather than on relationships within word (BoW) vectors like TF-IDF are typical to represent text individual threads; moreover our focus is on mostly anony- content for analysis [9, 13]. Topic facets to extract sentence mous discussion where most users are unregistered so user or level sub-topics are considered [6], The facets are shown to user-group views are less useful. reflect programming concepts for questions in programming discussion forums, visualized in a prototypical system. An alternative hierarchical semantic model for sentences incor- OVERVIEW OF THE SYSTEM: PIHVI - POSTING ANALYSIS porating discrete Fourier transform with a mixture of topic WITH INTERACTIVE HIERARCHICAL VISUALIZATION models was proposed [4]. In this first publication on PIHVI, We now describe the proposed system PIHVI - Posting Analy- we focus on the main goal of studying content relationships sis with Interactive Hierarchical Visualization, on an overview to the known hierarchy within the venue, and in order not to level, describing the design principles and how the system muddle that focus we use simple BoW-based features which operates. In the next section we describe how the system is already work well in capturing thread similarities in our experi- implemented through dimensionality reduction methods. ments. Other derived features like topical features are possible, Design Principles. The design principles of the PIHVI sys- the rest of our method works for any choice of features. tem are: 1. The system should illustrate, in a compact way, Showing posts from individual threads. In PIHVI, we em- the overall variety of discussion over the online forum, both phasize exploration based on the content-based hierarchical in terms of semantic content, and the hierarchical variety of scatter plot and the related tasks, thus we use a simple detail discussion sections where threads can be posted. This answers view showing plain texts from selected threads. However, we the question: “Which kinds of topics are users discussing, are aware of works that visually display thread content. This and in which sections is the discussion happening?” 2. The line of research usually focuses on visualizing the structure system should let analysts efficiently browse semantic content of the thread under inspection. For example, thread structures of threads across the forum, on an overall level of trends and can be encoded with indented rectangles [3], where each in- on more detailed levels of individual thread groups, without dentation corresponds to one level of nested reply to a post. restricting browsing to boundaries of specified sections. This Such thread visualization is orthogonal to PIHVI novelties and answers the question: “Which discussion trends are individual can be easily integrated into PIHVI. threads of interest related to?” 3. The system should compactly show relationships of threads in terms of semantic content and Dimensionality reduction in visual analytics. Dimension- position in the hierarchy. This answers the question: “What ality reduction has been brought into the pipeline of visual other similar threads exist for a thread of interest?” analytics by providing a 2D representation of threads, to help The above design principles should hold even when the num- users to gain an overview. It is related to semantic feature con- ber of threads, the number of sections in the hierarchy, and the struction discussed above, if obtained features are given to di- depth of the hierarchy, all become large. The resulting system mensionality reduction algorithms. For example, t-distributed for visual analytics should support key tasks in Shneiderman’s Stochastic Neighbor Embedding (t-SNE) [7] has been consid- [15] taxonomy of tasks for information visualizations, includ- ered [16]. In PIHVI, we build our hierarchical exploration ing the visual information seeking mantra overview first, zoom functionality on top of a t-SNE based embedding. See Section and filter, details on demand, and relate. 4 for details. Another branch of studies in this line is to let a user personalize the representation by interaction. See, e.g., System Components. Following the design principles in Sec- our work in [11, 10, 12] and references in [14]. We point out tion 3, we devise the following components for large-scale the novel goal of PIHVI is to study relationships of large-scale interactive analysis on hierarchical forum posts, and to support content to a large-scale hierarchy, hence the novelty is in in- our user tasks in PIHVI. Screen shots of the system are shown corporation of the hierarchy into the content visualization and in Figure 1. The system has five linked views, where views in hierarchical exploration of content through multiple views. 1-3 below are always shown and 4-5 are on demand: 1. Content-based map: An interactive scatter plot of the en- Forum analysis system combined with different visual an- tire collection of threads from the online forum, which can alytics. There have been integrated systems with views of be zoomed to several detail levels. The map shows a spatial tion distributions: higher opacity reflects a higher density of threads and vice versa; color mixing suggests mixtures of dom- inating sections sharing similar content. Both patterns may interest a forum analyst/administrator. For interactivity, each dot, from different zoom levels, can be inspected by clicking to show a pop-up: the pop-up may contain either the thread title and the name of its section, if the dot represents a single thread, or the name of the dominating common parent section, if the dot represent a group of threads. To avoid occlusions of multiple pop-ups, the analyst can bring a pop-up to front or back by clicking or dragging it; also, when a pop-up is mouse- overed, the remaining pop-ups without the cursor hovered will turn half-transparent. Buttons on this view allow the analyst to drag a rectangle over an area of the map: all threads in the area will be described in the Threads and Sections panel. 2. Thumbnail view of the map: This view shows the entire collection and a rectangle denoting which portion of the map an analyst is currently viewing; this lets the analyst maintain focus and context while browsing the collection. 3. Thumbnail view of the thread hierarchy: This view shows a radial plot of the tree hierarchy of discussion sections, and shows how optimized colors of the sections vary across the hierarchy. Due to space constraint, showing all 2434 section names at once is not feasible: we show section names on de- mand when the analyst mouse-overs the tree nodes of sections. The window size can be enlarged by standard zoom controls to help mouse-over the nodes. This view allows the analyst to relate where each thread or group of threads seen in the content view arises from in the hierarchy. Buttons on this view allow the analyst to drag a rectangle over a subset of sections: those sections will be highlighted in the content-based map. 4. Threads and Sections panel: When the analyst has selected an area of the content map, all threads in that area will be listed here as a scrollable list. The panel has two tabs: the Thread tab lists the threads by title and section, and the Sections tab lists all unique sections from which the threads arise. Figure 1: The PIHVI interface. Top: the initial state, where 5. Thread Information panel: When the analyst clicks any the content-view zoomed out to show an overview of the entire individual thread in the content-based map (individual dot that thread collection. Middle: section-filtering state. An analyst represents a single thread in the scatter plot) or clicks the title selects a subset of sections in the section panel, circles of of any individual thread in the Thread tab of the Threads and corresponding sections are then highlighted. Bottom: zoomed- Sections panel, the full text of that thread is shown here. in state for an analyst exploring a cluster of hearing-related threads from different sections across the forum; the analyst Walkthrough example. Here we depict a typical usage of has selected a group of threads with a bounding box and is PIHVI for thread discovery. Suppose an analyst is interested viewing their titles and content of an example thread. in sport and fitness related discussion but does not yet know how such discussion is related to other discussion within the Suomi24 forum. This can be a possible sequence of findings organization of threads by content similarity: similar threads and interactions from the analyst when discovering the threads are shown nearby. The map is created by machine learning of interest: 1. Initially the analyst views the content view based dimensionality reduction. The map is overlaid with zoomed out to show all Suomi24, and notices several clusters colors representing sections and their similarity, again created of color, indicating discussion similar in content and arising by dimensionality reduction. At detailed zoom levels, each from tight branches of the section hierarchy. thread is shown as a dot, and colors of the dots indicate the 2. The analyst clicks some of these clusters to find a starting section of the threads. At less detailed zoom levels, groups of point for analysis. After a time the analyst find a green cluster threads are shown as dots with diameters scaling with the num- (at the bottom in Figure 1 top) whose pop-up label shows the bers of threads in the groups, and colors of the dots indicate a dominating section in the cluster is Sport and Fitness (“Urheilu dominating common parent section of threads in the group, as ja kuntoilu”). Alternatively, the analyst could have browsed detailed later. We draw the different-sized dots with moderate the section tree to find a Sport and Fitness section. transparency, yielding color blending when there are overlaps 3. The analyst then zooms in to view individual threads within between the dots, which approximates the content and sec- the cluster, and draws a rectangle around them to quickly view Properties of the System. As desired in Design Principles, the proposed system directly supports several key tasks in Shneiderman’s [15] taxonomy, in particular the visual infor- mation seeking mantra is supported as follows: Overview: The two linked views (content-based map and the tree of the section hierarchy) together show an at-a-glance overview of threads in the entire online forum. The radial tree view shows an at-a-glance overview of the section organiza- tion: sections with numerous child sections are shown as wide branches, and sections with multiple levels of subsections un- der them are shown as long branches. On the other hand the Figure 2: Possible interactions when an analyst is discovering content-based map shows an at-a-glance overview of seman- threads of interest. Left: the analyst has used section filtering tic content across the whole forum, organized so that similar to discover several locations where Sport and Fitness (“Urheilu content is shown nearby in the map, and groups of similar ja kuntoilu”) is discussed (green highlighted clusters). Right: threads appear as clusters; moreover, section colors overlaid the analyst zoomed in towards the middle-right green cluster, on the map provide an easy overview of which sections have and inspected nearby other clusters: near the green Health and similar content, which sections have varied content appearing Fitness cluster (“Urheilu ja kuntoilu”) are two clusters, Health in multiple places on the map, and what the lowest common (“Terveys”; violet color) and Dogs (“Koirat”; red color). hierarchy section is for some cluster of content. Zoom and filter: the analyst can zoom into the levels of detail in the content-based map with the mouse wheel, and can filter threads from both linked views with selection boxes, e.g. to select threads in a particular location in the content map and highlight threads from a particular branch of the section tree. their titles. The analyst discovers many of them are about Details on demand: clicking on threads in the content based ice hockey, a popular sport in Finland, such as threads about map shows their content, clicking on thread-clusters gives the Finnish ice hockey teams and leagues and about related sports name of their lowest common section, and creating a selection like “Jääpallo” (Bandy in Finnish). box over an area of the content map opens a linked view listing 4. The analyst draws a rectangle around the Sport and Fit- all threads in the area and all sections of those threads; and ness section and its nearby sections, to detect where else such clicking any thread in that list shows the thread content. threads occur in the content map: Figure 2 (Left) shows that Relate: the content-based map relates threads by their con- besides the ice hockey related cluster at the bottom, there are tent similarity by placing similar threads nearby in the map; small isolated clusters (light green dots) at edges of the map, the section tree relates sections by showing their parent-child and a large cluster at center right in the map. Thus the analyst relationships; and the overlay of section colors onto the map discovered a new cluster of Sport and Fitness discussion. relates similarity of thread content to similarity of their sec- 5. The analyst removes the section filtering since he/she is in- tions. Inspecting these two linked views lets analysts quickly terested in all sections in the area. The analyst zooms towards study how threads and sections at different branches in the hi- the discovered cluster, and notices it is located at the border erarchy relate to each other in terms of content, and study how of two larger clusters (Figure 2 Right): clicking the clusters a cluster of semantically similar content is related to multiple reveals their dominating sections as Health (“Terveys”; violet sections where discussion of the content happens in the forum. colored cluster) and Dogs (“Koirat”; red colored cluster). 6. Since the three clusters are close-by, this indicates they may As noted above, PIHVI includes filtering/highlighting of the have common topics. The analyst is then interested what com- content map by selected sections. Highlighting could also be monalities there are in threads arising from Sport and Fitness done by results of text lookup searches; we do not focus on and in threads arising from Health, so the analyst zooms in simple text search for several reasons: 1. Our focus is not on more on the plot, and draw a rectangle around the boundary searching individual documents but on overall comprehension between “Terveys” and “Urheilu ja kuntoilu”. of the forum where information needs of the user evolve as 7. The topics of the selected threads are listed in the detail he/she learns about the data; such comprehension would not panel; the analyst finds several threads on weight-loss in the be well served by individual text lookups. 2. Naive text search information panel. does not take into account complexity of natural language and 8. The analyst could continue investigating Sport and Fitness could miss related threads if they did not share search terms (e.g. moving to view some of the other small clusters of Sport (due to synonyms etc.), whereas the content map is organized and Fitness threads), or could e.g. investigate commonalities by overall similarity of all words in the documents, and serves between Health and Dogs in the same window–this would turn exploration of discussion content better than simple search out to be discussion of allergies when living with dogs. would. Nevertheless, where text lookup is needed it is trivial The above described interaction can go on as long as desired to highlight lookup results like we do to selected sections. by the analyst. For an analyst, e.g., in the forum administration We will compare PIHVI to a baseline system we implemented team, such exploration may help understand the distribution to represent a more traditional interactive section-based brows- of the threads, which can inspire, for example, reorganization ing interface, described in the User Experiment section. of forum sections, or a digest on hot discussion topics. DIMENSIONALITY REDUCTION FOR THE PIHVI SYSTEM other in the section tree) get close-by colors. This color en- We describe the dimensionality reduction solution for creating coding is a visual cue helping users distinguish sections based the two main views – the content-based scatter plot and its on their locations in the hierarchy. We do not require users to color-based link to the section tree view, and then describe how perceive section distances exactly from colors; it is enough to zooming is implemented efficiently and using the hierarchy notice similar colors (close-by sections) when browsing the for summarization of thread groups. content. We take path lengths between sections in the section tree as input distances, and give weights to atomic paths – Content-based Representation of Textual Data. PIHVI is paths connecting a section to its direct parent or child section initiated by a content-based two-dimensional scatter plot of with wi↔ j = (Ci +C j )/2, where Ci is the number of child sec- threads on the online forum (here Suomi24), followed by tions of section i. Intuitively, if one end of the path has a large other interface components facilitating exploratory data anal- number of child sections, the parent is not connected to any ysis on the forum. In PIHVI, dots are discussion threads particular one of its child sections, implying a looser connec- in the “content-based” scatter plot. By “content-based”, we tion with the other end. The distance between two sections is mean to pursue a two-dimensional representation such that then the sum of weights of atomic paths connecting them. The 1. if two threads are similar in content, they should be placed distances are reduced to coordinates in a 3D output space by nearby; and 2. if two threads are dissimilar in content, they multi-dimensional scaling (MDS): the 3D output coordinates should be placed faraway. We use BoW models to capture are then normalized to the unit cube where they can be used as (dis)similarities between content of threads: the more words RGB color coordinates (more sophisticated color encodings two threads share, the more similar they are in content. We ex- from perception theory are possible [8]). As a result, sections tract thread features based on BoW as follows, and perform a close to the root get mild colors, with deepening saturation nonlinear dimensionality reduction with neighbor embedding. for more specialized lower-level sections. We empirically find BoW-based features in high-dimensional space. We MDS can yield a more continuous coloring than non-distance- start with lemmatized tokens of Suomi24, provided by the preserving dimensionality reduction algorithms like t-SNE. Finnish Language Bank service (“Kielipankki”, http://www. Zooming in the Content-based Plot. At the most detailed kielipankki.fi). The resulting data set contains the lemma- level in the content plot, threads are individual dots and the tized version of each token in each thread, along with other user can easily see their content similarities from their spatial meta-data of the token. We first calculate the TF-IDFs for each relationships and their section similarities from their colors. token, constituting raw features of each thread. We further When the user zooms out, he/she should similarly be able to process the raw features as follows: 1. Remove stop-words see content and section similarities of larger groups of threads. based on a Finnish stop-word list; 2. Only keep tokens that However, naive zooming of a large scatter plot would not ac- appear in at least k threads, to discard incidental tokens that complish this: firstly, in large discussion forums it would not do not carry shared semantic content, such as rare URLs or be feasible to redraw huge numbers of threads into a scatter non-word character strings. 3. For each thread, keep only l plot quickly; secondly, in areas where threads arise from mul- tokens with highest TF-IDF values. With stop-words already tiple sections the zoomed-out view would become a clump of removed, we assume these kept tokens are more informative different-colored dots, making it hard to notice overall simi- than others in each thread. 4. Normalize each obtained feature larities in sections of such threads. We solve both problems vector by dividing vector entries with their sum. We feed by an intelligent zoom. The plot area is divided into a grid the final features into t-SNE to obtain the two-dimensional (the higher the zoom-out factor, the rougher the grid). Instead content-based representation for PIHVI. Considering the size of plotting individual threads we plot one circle per grid cell, of our corpus, t-SNE is a suitable choice since 1. t-SNE has summarizing all threads in it: the circle radius indicates num- been shown to work well with high dimensional data; 2. scal- ber of threads in the cell, and circles are drawn with slight able t-SNE variants exist. We use a tree-based implementation translucency to allow smooth overlap instead of occlusion. [18] to achieve a O(N log N) complexity for N threads. Grid cells are indexed by Hilbert curves in each zoom level Section Distance based Coloring for Linking the Views. In for better performance in querying points in cells. For each the thumbnail view of the section hierarchy, we show a radial grid cell the color of the circle must be chosen intelligently tree plot of the hierarchy and draw section nodes in differ- as the grid cell contains threads from multiple sections: we ent colors so the analyst can distinguish the sections and link compute the color by starting from each thread and travel- them with the content-based map. There are a huge number ing up the section hierarchy until we find the lowest section of sections, and giving each a completely different color is covering at least 40% of threads in the grid cell, denoted the not possible, thus assigning colors to successfully link the two lowest dominating section. Thus the color of a circle indicates main views is nontrivial. We assume the analyst wishes to diversity of content in it: the less saturated (i.e., the closer to notice similar content from far-apart sections of the hierarchy, the color of the root section), the more diverse the content. and highly different content from close-by in the hierarchy, Such adaptive coloring lets the user quickly browse forum hence color differences should correspond to distances in the content at multiple detail levels. At each level, circles, their section hierarchy. We use dimensionality reduction to opti- colors, and their overlap yield an at-a-glance view of content mize colors so that the most important differences are encoded and section variation. As the user zooms out, areas of mixed with the most different colors, and close-by sections in the sections become large diffuse circles colored ever closer to hierarchy (having small path length from one section to the the root color; as the user zooms in, such circles break into subclusters with more vibrant colors, indicating thread groups but lacks the dimensionality reduction based content view and arising from tight branches of the section hierarchy. focuses on the section hierarchy only, allowing the user to browse it in a large resolution. This baseline represents a USER EXPERIMENT AND RESULTS system that would be readily implementable without having We perform a user experiment to show the advantage of PIHVI access to a dimensionality reduction based visualization of compared to a baseline, in controlled information seeking sce- thread content and thread relationships. In the baseline system narios representing typical analytics subtasks. We describe the there are four linked views: 1. Scrollable large-scale radial experiment design (the forum and thread corpus, task design, plot of the section tree hierarchy: each node in the tree corre- baseline system we designed&implemented, participants, and sponds to one section of the forum. evaluation criteria) and then the results. 2. A thumbnail of the whole tree, with a square control for navigating inside the tree; moving the square over the thumb- Details of the Online Forum and Corpus. We run our user nail changes the corresponding position in the large-scale plot experiment on Suomi24, one of Finland’s largest online com- of the tree. This thumbnail view is similar to the one in PIHVI, munities: it is mostly in Finnish with several Swedish sections; but focuses on navigation instead of filtering a content-based in 2015 it had 0.8 million unique visitors per week1 . We con- plot which is not available in the baseline system. sider the realistic scenario where analysts are analyzing data 3. A panel that appears when the user clicks on any section in of a specific time frame: here we filter data to the subset of the tree. It shows the thread titles from the selected section. thread from 2005; our methods can naturally be used for data This panel is similar to the threads tab in the PIHVI system. of any other years or for all of Suomi24. We set k = 20 and 4. A window for full-text content of threads; clicking any l = 200 in the preprocessing told in BoW-based features in thread in the title panel shows the full-text content, again simi- high-dimensional space. We then have 314871 threads and lar to the corresponding panel in PIHVI. 80123 tokens in the corpus, arising from 2434 sections. The thumbnail window and the window containing the thread- Task Design. In order to measure the success of our inter- list panel and full-text panel are movable by dragging. Figure face for visual analytics of hierarchical online discussion, we 3 shows a screen shot of the baseline system. Essentially, the evaluate performance in a typical subtask that would often be baseline corresponds to PIHVI without a content map, thread present as part of an analytics session: given a thread of in- content of sections is then naturally accessed through the sec- terest, find semantically related threads across the online tion hierarchy tree. This makes the two systems comparable. forum. We point out that given a thread of interest, it is trivial to browse other threads in the same section, and therefore we focus on the nontrivial task of finding related content from other sections across the online forum. The ability to find related content comprehensively across an online forum is crucial for comprehensive analytics of online discussion and how topics of interest are spread across the forum. We choose four cases where discussion on a semantic topic spreads across several Suomi24 sections: for each task, we indicate to the user one thread as a starting point, and ask to find threads on the same topic from other sections. Users carry out a training task to learn the interfaces: Kuulo (“hearing” in Finnish; the task is to find threads about hearing outside of the discussion section dedicated to hearing). We pick three Figure 3: A state of the baseline system. Components are tasks for the actual experiments: 1. Matti Nykänen (a Finnish marked with labels: the tree of the section hierarchy, a thumb- ski jumper; the task is to find threads about him outside of the nail of the full tree with a draggable control for navigation, and discussion section dedicated to him), 2. Svenskt (“Swedish” a thread information panel showing thread titles and content. in Swedish; the task is to find discussions in Swedish out- In the figure, the threads in the information panel are from the side of discussion sections dedicated to Swedish-language selected section “Häävalmistelut” (“Wedding preparation” in discussion), and 3. Hiukset (“hair” in Finnish; the task is to Finnish, marked in black in the section tree). find discussion about hair/hair loss/hair care/hair fashion out- side of the section dedicated to hair). Tasks involve several words of interest due to informal names, loose language, etc., Participants. We perform a user study with 12 participants, and are not fully solvable by naive text search. All the tasks including 1 female and 11 males; all participants are computer correspond to the research questions we ask in Section 3. science PhD students or post-doctoral researchers who know both Finnish and Swedish. Each participant first performs a The Baseline System. We compare the proposed system in training stage on both systems by using the systems to perform terms of task performance and usability against a baseline the training task. Next, the participant performs all three system. The baseline is an interactive system for browsing experiment tasks on both systems (PIHVI and baseline). The threads by interacting with a radial visualization of the section order of the task and system combinations (which order of the hierarchy. It is comparable to the proposed PIHVI system, three tasks; which system first in each task) was randomized 1 As reported in https://goo.gl/Ukt49K. and counterbalanced over the participants to avoid bias. Evaluation Measures. We evaluate the experiment results in Baseline PIHVI four ways. First, we inspect the results qualitatively: is the Mean (Std) 1.045 (1.222) 4.137 (1.359) PIHVI system generally better in the opinions of the partici- pants? Secondly, we quantitatively measure task performance: Table 1: Means (over tasks and users) and standard deviations how many relevant threads was the user able to find in each of quality measures R calculated from retrieved threads. A task, compared to the amount of time spent? We measure this larger mean suggests better relevance. as a proportion of total relevant threads to time spent, averaged over users, for each task. Thirdly, we measure user experi- System Participant System×Participant ence for both systems by a questionnaire asking users to rate p-value 6.82 × 10−13 0.938 0.440 their agreement to statements about the system on a 5-point Likert scale from strong disagreement to strong agreement; Table 2: Two-way ANOVA for relevance of retrieved threads. the questions are based on the standard ResQue questionnaire We have p < 0.05 in the system effect; the 95% confidence with some modification for the online forum domain, and are interval of the PIHVI system coefficient is [0.379, 4.819], fully listed in Table 3. Fourthly, after the tasks we also ask each above zero. Thus PIHVI performs significantly better than the user which system they preferred overall. baseline in terms of relevance of retrieved threads. Results Qualitative analysis. We collect qualitative descriptions on results are in Table 2. We observe the system effect is statis- the systems by allowing participants to give free-form written tically significant in this ANOVA, suggesting that PIHVI is or oral feedbacks. Overall, the participants agree the tasks significantly better than the baseline system when measuring are clear. Besides good quantitative performance in tasks, the relevance of the retrieved threads over time spent. participants made discoveries, hence PIHVI helped them com- prehend the discussion venue. One participant stated he dis- User responses to post-task questionnaire. Table 3 shows covered Swedish is sometimes used for trolling in Suomi24 the distribution of answers to the post-task questionnaire. The when he was doing the task “Svenskt” on PIHVI. In task questions can be divided into two groups, 13 positive questions “Matti Nykänen”, one participant noted PIHVI can put threads (higher user agreement is better) and 5 negative questions on Matti Nykänen together, though people in the threads call (lower user agreement is better). Roughly, positive questions him different names. Another participant said discussions on cover overall user experience and ability to perform tasks with Matti Nykänen in marriage related sections reminded him of the system, negative questions focus on ease of use. Half of the 18 questions yielded statistically significant differences. All the ski jumper’s series of short-lived marriages. Participants statistically significant differences are in favor of the PIHVI commented PIHVI can give them threads from diverse sec- system. Table 3 lists in detail the p-values showing statistical tions, compared to the baseline. Participants remarked that the significance (Q1, Q8, Q9, Q13–Q18) at the level of 0.05. baseline system is reasonable, and one pointed out it is similar to how people usually browse the forum; the only difference Users wanted to use PIHVI frequently more than the baseline is that in the baseline we organize sections radially, not lin- (Q1), found PIHVI less cumbersome than the baseline (Q8), early. As for the section filter, one participant stated it helps felt more confident using PIHVI than the baseline (Q9), found understand section distributions when the map is zoomed-out. it more easy with PIHVI to find what they were interested in Lastly, though some of them agreed that PIHVI is slightly (Q13), found it more easy to find a thread to read with PIHVI more complicated than the baseline, the learning curve of (Q14), would use PIHVI more than the baseline to find threads PIHVI is not steep, so they can learn to use it fairly quickly. to read (Q15), would use PIHVI again more than the baseline (Q16), would use PIHVI frequently more than the baseline Task performance. We evaluate task performance of partici- (Q17), and would prefer PIHVI over other analytics systems pants in terms of relevance of retrieved threads and time spent. more than they would prefer the baseline over other systems To evaluate relevance of threads retrieved by the participants, (Q18). Thus overall, participants report they can find what a native Finnish speaker assessed all retrieved threads without they are interested in easily with PIHVI, and would like to use knowing which system they were from, rating them on a three- it to do exploratory data analysis frequently (Q1, Q13-Q18). point scale from 0 to 2 for non-relevance, partial relevance, Also, PIHVI is easier to use and gives the participant more and full relevance. For each task on each system performed by confidence during the exploration compared with the baseline each participant, we use the quantity R = ∑i ri /T as the qual- (Q8-Q9). The confidence may arise because PIHVI’s layout ity measure for this (task, system, participant) combination, reflects thread similarity better so that in PIHVI the relevance where T is the number of minutes taken to finish the task, ri is between the found threads and the tasks is clearer than in the rated relevance of the i-th thread, with i going from 1 to the the baseline. For brevity we simply list the questions that number of retrieved threads in the combination. The measure did not yield significant differences: Q2 I found the system R can be interpreted as the amount of “retrieved relevance in unnecessarily complex; Q3 I thought the system is easy to use; unit time”. A larger R indicates that a larger number of more Q4 I think that I would need the support of an analytic person relevant threads are retrieved in a shorter time. We collect the to be able to use this system; Q5 I found the various functions R values from all results of the participants on the two sys- in this system are well integrated; Q6 I thought there was tems, and perform a two-way analysis of variance (two-way too much inconsistency in this system; Q7 I would imagine ANOVA; “participant”× “system”). Means and standard devi- that most people would learn to use this system very quickly; ations of the R values are shown in Table 1. And the ANOVA Q10 I needed to learn a lot of things before I could get going Baseline PIHVI p (Welch’s P/N ID Question mean (std) mean (std) t-test) + Q1 I think that I would like to use this system frequently 1.917 (0.862) 3.750 (0.433) 9.866 × 10−6 - Q8 I found the system very cumbersome to use 3.167 (1.280) 2.000 (0.816) 0.0198 + Q9 I felt very confident using the system 3.083 (1.187) 4.000 (0.816) 0.0480 + Q13 Using this system to find what I am interested in is easy 1.750 (0.829) 4.083 (1.115) 1.780 × 10−5 + Q14 Finding a thread to read with the help of this system is easy 2.750 (1.090) 4.083 (1.037) 7.604 × 10−3 + Q15 I will use this system to find threads to read 2.083 0.954) 3.333 (0.850) 3.757 × 10−3 + Q16 I will use this system again 2.083 (0.862) 3.167 (1.143) 0.0206 + Q17 I will use this system frequently 1.833 (0.799) 2.917 (1.115) 0.0165 I will use this system rather than other systems of exploratory + Q18 2.167 (0.898) 3.583 (0.640) 3.852 × 10−4 data analysis for online forums Table 3: The questions yielding statistical significance in the post-questionnaire, with mean sand standard deviations of user agreement with the questions. For “positive” questions (Q1, Q3, Q5, Q7, Q9, Q11, Q12, Q13, Q14, Q15, Q16, Q17, Q18) higher agreement is better, for “negative” questions (Q2, Q4, Q6, Q8, Q10) lower agreement is better. with this system; Q11 The layout of this system interface is 8. T. Munzner and E. Maguire. Visualization analysis and attractive; Q12 The layout of this system is adequate. design. CRC Press, 2015. User preference. Users were also asked which system they 9. A. I. Obasa, N. Salim, and A. Khan. Hybridization of overall prefer: all users preferred the proposed PIHVI system. bag-of-words and forum metadata for web forum question post detection. Indjst, 8(32), 2016. CONCLUSIONS We presented PIHVI – Posting Analysis with Interactive Hi- 10. J. Peltonen, K. Belorustceva, and T. Ruotsalo. erarchical Visualization, a novel visual analytics system for Topic-relevance map: Visualization for improving search hierarchical online forums based on multiple linked views. result comprehension. In ACM IUI 2017, 2017. The main view (interactive thread content scatterplot) and 11. J. Peltonen, M. Sandholm, and S. Kaski. Information the coloring that links it to another prominent view (radial retrieval perspective to interactive data visualization. In tree of sections) are created by nonlinear dimensionality re- Eurovis 2013 short papers, 2013. duction from high-dimensional similarities between threads and between sections, respectively. A user study on a large 12. J. Peltonen, J. Strahl, and P. Floreen. Negative relevance Finnish online forum shows significant advantages compared feedback for exploratory search with visual interactive to a baseline without dimensionality reduction based views: intent modeling. In ACM IUI 2017, 2017. users found more relevant results with PIHVI, and preferred 13. Z. Qu and Y. Liu. Finding problem solving threads in it over the baseline, with statistical significance. The system online forum. In IJCNLP 2011. helps analysts find and relate discussion content across large hierarchies, an important need as forums keep growing. 14. D. Sacha, L. Zhang, M. Sedlmair, J. A. Lee, J. Peltonen, D. Weiskopf, S. North, and D. A. Keim. Visual REFERENCES interaction with dimensionality reduction: a structured 1. I. Borg and P. J. F. Groenen. Modern Multidimensional literature analysis. IEEE TVCG, 23(1):241–250, 2016. Scaling: Theory and Applications. Springer, 2005. 15. B. Shneiderman. The eyes have it: A task by data type 2. S. Chen, L. Lin, and X. Yuan. Social media visual taxonomy for information visualizations. In IEEE VL analytics. Computer Graphics Forum, 36(3), 2017. 1996. 3. K. Dave. Flash forums and forumreader: Navigating a 16. A. J. Soto, R. Kiros, V. Kešelj, and E. Milios. Exploratory new kind of large-scale online discussion. In CSCW 2004. visual analysis and interactive pattern extraction from semi-structured data. ACM TiiS, 5(3):16:1–16:36, 2015. 4. J. Duan, J. Zeng, and S. Zhang. Hierarchical semantic model for objectionable web text content detection. In 17. J. Teevan, C. Alvarado, M. S. Ackerman, and D. R. IEEE ASID 2012. Karger. The perfect search engine is not enough: A study 5. S. Fu, J. Zhao, W. Cui, and H. Qu. Visual analysis of of orienteering behavior in directed search. In CHI 2004. mooc forums with iforum. IEEE TVCG, 23:201–210, 18. Z. Yang, J. Peltonen, and S. Kaski. Scalable optimization 2017. of neighbor embedding for visualization. In ICML 2013. 6. I.-H. Hsiao and P. Awasthi. Topic facet modeling: 19. J. Zhao, N. Cao, Z. Wen, Y. Song, Y.-R. Lin, and Semantic visual analytics for online discussion forums. In C. Collins. # fluxflow: Visual analysis of anomalous LAK 2015. ACM. information spreading on social media. IEEE TVCG, 7. L. v. d. Maaten and G. Hinton. Visualizing data using 20(12):1773–1782, 2014. t-sne. JMLR, 9(Nov):2579–2605, 2008.