Visual Analytics has achieved a lot of attention for its abilities to support exploratory knowledge discovery in large data sets. In this work-in-progress paper, we develop a Visual Analytics framework for user comments in the domain of online journalism. First, we examine how journalists' needs can be mapped to a visual interactive interface to make sense of user comments. We further investigate how diferent classes of Machine Learning algorithms like supervised and unsupervised learning can be integrated into Visual Analytics to enable a more user centric analysis. Due to the variety of Machine Learning approaches, we expect that diferent forms of integration will be needed. Our goal is to place the domain experts (e.g. journalists) in the loop to improve analytical reasoning.
As data accessibility increases analytical methods and techniques
to handle the data become more important. Visual Analytics (VA)
is a novel approach to gain insights from heterogeneous and
unstructured data [
In order to create such tools for big data scenarios, a precise understanding of the coupling of Machine Learning (ML) and IV is required. As not much work is done on how to modify and steer ML methods through interactive interfaces, we focus on user centric possibilities to adapt ML algorithms in the progress of the analysis.
In this paper, we investigate how diferent types of ML like supervised and unsupervised learning as well as specific methods like clustering, classification, regression and dimension reduction can be integrated in VA. Diferences are expected, because of the diversity of those approaches. We use our findings to create a VA framework for user comments in the domain of online journalism.
First, we shortly introduce the VA approach, the concept of “Human in the Loop” and the domain of making sense of news discussions. Secondly, we present our VA Framework for user comments. Hereafter, we argue in section 3.2 for a "Human in the Loop" approach in VA in order to improve our framework. Then, we outline questions for our upcoming research. Finally, we describe related work and end up with our conclusion. This section shortly introduces the central concepts of VA.
VA is defined as "the science of analytical reasoning facilitated
by interactive visual interfaces" [
In the early stages of ML the overall question was, "how to construct
computer programs that automatically improve with experience"
[
A ML approach which can utilise domain knowledge is described
by the phrase “Human in the Loop” (HitL). HitL is a special case
of interactive ML (iML) and can be defined as algorithms that can
optimize their learning behaviour through the interaction with
humans [
The primary advantage of HitL is the ability to reach inside the models black box. The approach enables the advanced use of the human knowledge and expertise inside a continuous feedback loop. User interaction empowers model steering and is not limited to model selection and parameterization. However, the HitL approach rises new challenges in getting the most out of the participation with humans. Novel approaches are needed to make this interplay more intuitive and beneficial.
A typical use case or HitL is supervised learning. In this setting,
the goal is to learn a mapping between X and Y given a set of
training pairs (xi , yi ), whereas xi ∈ X are called examples and yi ∈
Y are labels for i ∈ {1, ..., n}. A typical supervised HitL approach is
to increase n as the number of example pairs to gain more accuracy
through enhancing the grounded truth. However, HitL can also
be applied to unsupervised learning settings. Here only a set of n
examples X = {x1, ..., xn } is given. The goal is to find interesting
structures in the examples X . By applying HitL the problem can
be transformed into a semi-supervised learning problem if a user
provides l labels for some examples in X . The examples get divided
in Xl := {x1, ..., xl } for which the labels Yl := {y1, ..., yl } are given
and Xu := {xl +1, ..., xn } as the set with no labels. In this case, the
training pairs can be used to dynamically guide the computation
by constraints or additional information. See Capelle et al. [
This section introduces and motivates the domain of analysing user feedback in news discussions. User feedbacks refers to various forms of user participation regarding journalistic contents. The focus lies on textual comments from multiple channels like news webpages, emails and social media.
There is a high demand of gaining insights from user comments,
as they are a valuable source of information. They can contain
useful aspects like feedback, critics, new perspectives and expertise.
Furthermore, comments mirror personal opinions which are
normally hard to capture [
Studies show that user comments support the daily work of
journalists and editors [
Consequently, newsrooms are faced with an increasing demand
for computer supported ways to analyse, aggregate and visualize
user comments. Unfortunately, there is a lack of analytical tools to
provide high quality comments that can be leveraged for journalistic
purposes [
From a data science point of view, user comments are documents with heterogeneous information contexts and several connections between each other. They consist of multiple attributes like a commenter identification, the related article, a timestamp, a title, a ranking from other readers and several annotations from manually or automated classifications like sentiment analysis and swearword detection. 3
As the manual analysis of user comments is resource consuming and
unpractical with a rising volume, velocity and variety of user
comments, various researchers focus on approaches to detect patterns
automatically [
• Article Selection
The occurrence of comments in time/progress of a discussion. The user can select samples of articles from which the comments are analysed. • Topics and Addressees
What is discussed and who is mentioned and directly addressed in a sample of user comments. • Discussion and Argumentation
The direction of a discussion and risen arguments over the time. The user can analyse the development of pro- and contra-arguments towards a certain question or topic over time. • Quality
Metrics to quantify the quality of the user comments to ofer a condensed overview. • Selected User Comments
A close read function for selected user comments.
Each dimension is implemented as a separate view within a web application. Views supply a set of visualisations and interactions to enable and support an analytical discourse with the user. The views are coordinated and implement diferent interaction strategies like selections, filtering , focus+context and linking+brushing. Every view is an optional part of the analysis. The user decides dynamically, which views he or she wants to use. The layout of the views are based on a “filter flow ” metaphor. The views possess a specified ordering in which they are placed on the screen. Changes inside a view are only forwarded to subsequent views. The user is able to iflter in arbitrary order and there are no top-down restrictions.
Figure 1 shows the Article Selection view. The upper part of the ifgure depicts the distribution of comments over time, whereas the lower part ofers several selection options. Comments can be selected individually or grouped by sections, topics and authors. The Discussion and Argumentation view is depicted in figure 2. After the user selected a stance or topic the distribution of the for and against comments are shown in a line-graph. Related arguments are listed in the bottom part. Furthermore, selected comments are plotted regarding to their sentiment and user ratings. The grey boxes relate to filtering operation and only comments within are considered. Figure 3 illustrates the Quality view. User comments are represented as a set of diferent indicators. In the upper part user comments are categorised along the dimensions article reference, compliance and originality. For further analysis, each of the above selected comments are than depicted as polylines inside parallel coordinates. As indicators e.g. the length, sentiment, or number of references are used.
Another central part in our investigations is the evaluation of our prototype. We conduct a quantitative usability study after the end of the first implementation cycle to assess the overall system and to counteract any weak points. Seven participants are observed while solving real world problems with our prototype. In addition, they answer a questionnaire about the usability. The findings are used for improvements and further requirements. The evaluation also reveals that the participants unconditionally trust the results. This is not surprising as the tool does not show the accuracy of the analytical results.
In our first prototype the processing capabilities of VA is not exhausted. One drawback of our prototype is the lack of interactive model adaptation. The prototype lacks in integrating users’ expertise and experience in the analysis. The current interaction takes place as a sequence of selections, overviews and filter operations. The integration of ML is limited to pre-processing.
To enable HitL and interactive model steering and thus exploit the possibilities of VA, we extend our prototype with interactive ML components. We have to develop specific use cases that include HitL approach, which can also be used in a more general form across domains.
The question arises why it could be beneficial to place the journalist in the loop. At first, our framework is based on several classification models that provide annotations for interactive visualisation. Since these algorithms are based on simplified models of reality, misclassifications are to be expected. The misclassification likelihood can be quantified by relative error frequencies. By using HitL, the user can interactively correct spotted misclassifications and initiate new training cycles to improve the model’s accuracy, which afect the visual correctness.
Secondly, HitL opens up new possibilities of individualisation. It is dificult to satisfy the desire for customizable queries through a predefined set of filter operations. HitL makes it possible to dynamically select data that are of special interest. The user can be ofered opportunities to create and apply generic classification models at runtime.
Furthermore, many ML algorithms work on restricted information basis. There is only a limited amount of data available, only a sample is used for computation or the algorithms are incorrectly configured. This leads to diferent understandings of similarities or weights between human expectations and computational results. HitL makes it possible through experience of the users to correct these deviations and thus to improve the results.
following questions arises: For our next steps to integrate diferent ML methods in VA, the RQ1: To what extent can diferent ML methods be adapted within an iterative process? A taxonomy for user centric adaptations of ML methods will be developed to guide the development of VA applications. The integration of HitL aspects is carried out with regard to the diference of supervised and unsupervised approaches. In addition, specific methods like classification, regression, clustering and dimension reductions will be considered.
RQ2: How can model adaptations and responses be translated and
mapped into a visual metaphor?
It is a dificult task to steer computational models to match
expectations. There is a gap between computational processing on the
machine side and cognition on the human side that can lead to hard
usability problems [
RQ3: How can the uncertainties of ML method be visually
communicated and utilised in a constructive manner?
Uncertainty is created and communicated over the complete VA
process [
In order to answer these questions, we examine the integration of ML and IV from a user centric point of view. To support the suitability of our findings, we develop and evaluate several prototypes in the domain of news discussions. A goal is to cover diferent ML approaches. We will carry out quantitative usability tests with representatives to spot and document strengths and weaknesses. The observation will take place inside our usability lab which consists of eye-tracers, cameras, screen captures and key loggers. 5
In [
We have developed a VA tool for user comments in online journalism and have outlined next steps for a user centric integration of ML in our prototype. The next steps cover how diferent ML methods like supervised and unsupervised learning can be adapted within an iterative process, how these adaptations can be visual translated and mapped to a visual metaphor as well as how occurred uncertainties can be communicated and constructively used. A broader understanding of the coupling of ML and IV is necessary to fully exploit the strengths of VA.
The paper was supported by BWFG Hamburg within the “Forum 4.0” project as part of the ahoi.digital funding line.