=Paper=
{{Paper
|id=Vol-2903/IUI21WS-ESIDA-2
|storemode=property
|title=Exploring the "Double-Edged Sword" Effect of Auto-Insight Recommendation in Exploratory Data Analysis
|pdfUrl=https://ceur-ws.org/Vol-2903/IUI21WS-ESIDA-2.pdf
|volume=Vol-2903
|authors=Quan Li,Huanbin Lin,Chunfeng Tang,Xiguang Wei,Zhenhui Peng,Xiaojuan Ma,Tianjian Chen
|dblpUrl=https://dblp.org/rec/conf/iui/LiLTWPMC21
}}
==Exploring the "Double-Edged Sword" Effect of Auto-Insight Recommendation in Exploratory Data Analysis==
https://ceur-ws.org/Vol-2903/IUI21WS-ESIDA-2.pdf
Exploring the “Double-Edged Sword” Effect of Auto-Insight
Recommendation in Exploratory Data Analysis
Quan Lia , Huanbin Linb , Chunfeng Tangc , Xiguang Weid , Zhenhui Penge , Xiaojuan Maf and
Tianjian Cheng
a
School of Information Science and Technology, ShanghaiTech University
b
AI Group, WeBank
c
AI Group, WeBank
d
AI Department, Shenzhen Semacare Medical Technology Co.
e
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology
f
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology
g
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology
Abstract
Modern data analytics tools often provide visualizations as an accessible data window to users in exploratory data analysis
(EDA). Still, many analysts feel lost in this process due to issues such as the high complexity of data. Auto-insight recom-
mendations offer a promising alternative by suggesting possible interpretations of the data to users during EDA but might
impose undesirable effects on users. In this study, we systematically explore the “double-edged sword” effect of auto-insight
recommendations on EDA in terms of exploration assistance, message reliability, and interference. Particularly, we design and
develop two versions of a Tableau-like visualization system termed TurboVis: one supports auto-insight recommendations
while the other does not. We first demonstrate how typical visualization specification tools can be augmented by incorporat-
ing auto-insight recommendations and then conduct a within-subjects user study with 18 participants during which they
experience both versions in EDA tasks. We find that although auto-insight recommendations encourage more visualization
inspections, they also introduce biases to data exploration. The perceived level of message reliability and interference of
auto-insight recommendations depend on data familiarity and task structures. Our work elicits design implications for
embedding auto-insight recommendations into the EDA process.
Keywords
Visualization recommendations, exploratory data analysis, auto-insight
1. Introduction inquire into the data and scaffold a graphical represen-
tation for insight interpretation and communication. In
Exploratory Data Analysis (EDA) refers to the critical an ideal setting, analysts are supposed to immerse them-
process of performing initial investigations on data to selves into such a creative process without breaking the
discover patterns, spot anomalies, test hypotheses, and flow. However, in reality, analysts are often stuck with
check assumptions with the help of summary statistics questions such as “where should I start? ” and “what else
and graphical representations [1]. For example, financial can I find? ” [5]. These issues could slow down the process
analysts conduct EDA to identify the main trend of en- and yield fewer meaningful EDA outcomes.
terprise business metrics, discover data outliers to locate To support analysts to gain insights from the data, pre-
potential problems that need attention or even action, and vious data analytics tools have proposed auto-insight rec-
form further hypotheses for more in-depth data explo- ommendation services that suggest potential interesting
rations. The outcomes of EDA processes are data insights visual patterns [6] or simulate exploration hunches [7].
that are often integrated into a visual dashboard [2, 3]. In other words, as users visualize data, these tools could si-
In a sense, EDA is kind of a creative process [4], during multaneously conduct analysis, and recommend trends/pat-
which users leverage their knowledge and intuitions to terns termed auto-insight recommendations [8]. For ex-
ample, Mackinlay et al. proposed Show Me [9] which sup-
Joint Proceedings of the ACM IUI 2021 Workshops, April 13-17, 2021, ports users to search for graphical presentations when
College Station, USA
analyzing data. However, current auto-insight recom-
Envelope-Open qliba@connect.ust.hk (Q. Li); bindylin@webank.com (H. Lin);
jsontang@webank.com (C. Tang); marcus@semacare.com (X. Wei); mendation techniques such as Voyager 2 [10] and Fore-
zpengab@connect.ust.hk (Z. Peng); mxj@cse.ust.hk (X. Ma); sight [11] primarily focus on the design of insight dis-
tobychen@webank.com (T. Chen) covery algorithm or perceptually effective insight pre-
Orcid 0000-0003-2249-0728 (Q. Li); 0000-0002-5700-3136 (Z. Peng); sentations without deep considerations on the repre-
0000-0002-9847-7784 (X. Ma)
© 2021 Copyright for this paper by its authors. Use permitted under Creative sentation of the intended goal of the recommendations,
CEUR
Workshop
Proceedings
Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
http://ceur-ws.org
ISSN 1613-0073
ease of understanding and contexts, and user prefer-
ences. Consequently, the resulting auto-insight recom- simpler version of TurboVis that only has the later three
menders may introduce the following side effects to the features but no suggestions from the system. Our within-
EDA process. ⃝ 1 Bias. Prior studies in the field of rec- subjects (TurboVis with vs. without auto-insight recom-
ommender systems indicate that without a clear repre- mendation) study with 18 industrial business analysts
sentation of the intended goal of the recommendations, shows that the auto-insight design makes them inspect
elaborately designed recommendation algorithms have more visualizations but introduces bias to the direction
the potential to limit exploration breadth as users may of exploration. Auto-insight recommendations offer new
unconsciously confine their explorations to the items perspectives when analysts are not familiar with the data
recommended [12, 13, 14]. Systems such as Foresight or have a vague idea about how to proceed, and may
and Voyager 2 uncover visual insights independent of the distract analysts when they are facing a familiar dataset
EDA pipeline and represent visual insights in an implicit or usage scenario. Meanwhile, auto-insight recommen-
way [10, 11]. An anecdotal evidence shows that, if such dations would interrupt analysts more when they have
auxiliary findings generated by these systems and the specific target for exploration in mind than when they
intended goal of these insights are not explicitly repre- are completely open-minded. Based on these results, we
sented and mentioned, users still have little clue as to further elicit design implications for embedding auto-
how the augmented information can be pieced into the insight recommendations into the EDA process.
final story and where it leads them in EDA [10]. Con-
sequently, it brings bias to analysts in interpreting and
exploiting the recommended visual insights for EDA. ⃝ 2 2. Related Work
Reliability. Prior research in recommender systems sug-
Literature that overlaps with this work can be classified
gested that recommendation service is context-specific
into three categories: exploratory data analysis, visual-
and should improve its readability in order to make the re-
ization recommendations, and recommender system.
sults more reliable to users [15]. For one thing, previous
Exploratory Data Analysis. EDA is a term coined
studies illustrate that the perceived utility of recommen-
by John W. Tukey for describing the act of “looking at
dations is context-specific, i.e., with a limited knowledge
data to see what it seems to say” [21]. In EDA, attempts
of users, intelligent systems will be less competent in of-
are made to identify the major features of a dataset of
fering recommendation services [12, 16, 17]. For another,
interest and to generate ideas for further investigation.
as we may not know the system users at all, “some auto-
Particularly, Tukey drew an analogy between EDA and a
matically recommending insights are still like a baffling
series of detective work, during which analysts form a
mystery to laymen” [18]. If the auto-insight recommen-
set of hypotheses by asking questions, and integrate their
dation service fails to present its suggestions in an easy-
domain knowledge to obtain rich data insights [22, 21].
to-understand manner, the insights’ reliability drops. ⃝ 3
Data visualization is perhaps the most widely used tool
Interruption. Given that the structure of an EDA pro-
in the EDA process. With the rise of interest in data
cess could be anywhere from fully open exploration to
science and the need to derive value from data, analysts
target-oriented inspection, automatically recommending
increasingly leverage visualization tools to conduct ex-
insights of the data may bring interruption to the EDA
ploratory data analysis, spot data anomalies, and correla-
process when users prefer open exploration or exami-
tions, and identify patterns and trends [23, 24]. The state
nation on the non-suggested data aspects [14, 19, 20].
of the art in data visualization involves a lot of manual
Auto-insight recommendation service could thus be a
generation of visualizations through tools such as Excel,
“double-edged sword” in EDA, and it would impair the
Tableau [25] and Qlik [26] to facilitate the EDA process
analysis experience and results if it is designed inap-
for non-expert analysts. However, it is still challenging
propriately. Hence, to design an effective auto-insight
for data visualization novices to rapidly construct visu-
recommendation service to support EDA, we need to first
alizations during the EDA process. Grammel et al. [27]
identify what are required and concerned in a smooth
conducted an exploratory laboratory study in which data
EDA pipeline, and then we need to systematically explore
visualization novices explored fictitious sales data by
the potential “double-edged sword” effect of auto-insight
communicating visualizations to a human mediator, who
recommendations on the EDA process and outcome.
rapidly constructed the visualizations using commercial
To this end, we design TurboVis, a Tableau-like visual-
visualization software. Apart from identifying activities
ization system that supports analysts in EDA activities
that are central to the iterative visualization construction
with features including auto-insight recommendation
process, they also found that the major barriers faced
based on an extensible repository of statistic metrics,
by the participants are translating questions into data
graphics matching, manual visualization specification,
attributes, designing visual mappings, and interpreting
and dashboard editing and interaction. To evaluate how
the visualizations. In this study, we explore the role of
auto-insight recommendation could positively or nega-
auto-insight recommendation in the EDA process.
tively affect the EDA process and outcome, we create a
Figure 1: Visualizations are generated with a spectrum of tools automatically or manually.
Visualization Recommendations. As the demand Inspired by the previous findings and to obtain a system-
for rapid analysis for visualization grows, there is an atic understanding of how the auto-insight recommenda-
increasing requirement to design visualization tools al- tion systems might pose a hindrance to the exploratory
lowing users to efficiently generate visualizations. Prior data analysis process, we design and develop two ver-
studies show that the relevant authoring tools are in- sions of a Tableau-like visualization tool and attempt to
creasingly towards automatic [28], which can be classi- explore the “double-edged sword” effect of auto-insight
fied into four categories. Initially, users have to manually recommendations on EDA in terms of exploration assis-
write codes for visualizing data by using imperative lan- tance, message reliability, and interference.
guages and libraries such as D3 [29], Vega-Lite [30], and Recommender System. Recommender systems col-
ECharts [31], which are designed for users who are famil- lect their target users’ preferences for a set of items
iar with coding and visualizations. Later, researchers con- such as movies, songs, books, and travel destinations.
tributed visual building frameworks for easy visualiza- They leverage different sources of information for pro-
tions including template editing [32, 33], shelf configura- viding users with predictions and recommendations of
tion [34], and visual building [35, 36]. These tools are de- items [46]. With the ever-growing volume of online in-
signed for users who can write codes but not familiar with formation, recommender systems have been an effective
visualizations. Particularly, users need to “pre-conceive strategy to overcome such information overload [47],
blueprints, then interact with the system” [28] to obtain particularly useful when users do not have sufficient
more expressive, appropriate and aesthetic visualizations. experience to make a choice from a large number of
Then, semi-automatic methods involved with few inter- alternatives [48]. Existing research in the field of recom-
actions were proposed for efficiently obtaining visualiza- mender systems mainly focus on the recommendation
tions like SAGE [37] and Tableau [25]. Fully automatic accuracy and the explainability of recommendation algo-
methods are designed for no-human-involved tools for rithms [46, 47, 49, 50, 51], which inevitably result in that
efficiently obtaining visualization recommendations such people are increasingly relying on recommender systems
as Text-to-Viz [38], Click2Annotate [39], Data2Vis [40], that employ algorithmic content curation to organize, se-
and DeepEye [41]. These tools resolve the issues when lect and present information [12, 52, 53]. Despite its wide
users are not familiar with either visualizations or cod- utility, researchers have indicated that its potential im-
ing. On the other hand, to resolve the issues that analysts pact to improve problems related to over-choice should
often have no idea what they are looking for especially be concerned [14]. Therefore, inspired by the studies
at the initial stage of data exploration, researchers have on the potential harm of such recommendations, we be-
developed various algorithms and systems to recommend lieve that the potential “double-edged sword” effect of
insightful visualizations that can depict data trends and auto-insight recommendations warrant a separate study.
patterns [41, 42, 43]. In this study, we combine a semi-
automatic method that involves user interactions along
with algorithms that can recommend interesting insights 3. Research Questions
on the basis of extensible metric repository. Therefore,
The literature suggested that visualization recommenda-
analysts can benefit from a quick launch of data explo-
tions encourage users to explore more visualizations [10],
ration from automated recommendations of potentially
while prior studies in the field of recommender systems
interesting data patterns.
indicated that recommendations would introduce explo-
The existing studies mainly focus on employing either
ration bias, i.e., users might confine their exploration to
machine learning algorithms or user-defined rules and
the recommended items [13, 54]. In other words, recom-
visual embellishments into the creation of infographics
mendations can encourage exploration breath but may
to lower the barrier for data exploration by automatically
introduce a lack of breadth diversity. We posit that with
generating visualizations. However, indicators from pre-
the auto-insight recommendations, target users may ex-
vious studies also point out that recommendations may
plore more visualizations but will be biased to the visu-
potentially hamper users during data exploration [44, 45].
Figure 2: (A) Data processing module of TurboVis: ⃝ 1 Data menu; ⃝ 2 Entry to auto-insight recommendation and inter-
active visual analysis modules; ⃝
3 Data table that provides necessary processing functions. (B) Interactive visual analysis
module of TurboVis: ⃝ 1 Data; ⃝
2 Dimensions; ⃝ 3 Metrics; ⃝
4 𝑥 and 𝑦 axis; ⃝
5 Display area of visualization; ⃝
6 Visualization
recommendations; ⃝ 7 Chart configuration area. ⃝
8 (C) Dashboard editing and export module of TurboVis.
alizations that are only supported by the recommended eratively refine the system through a series of informal
items. Therefore, we have our first research question: usability testings. To be specific, TurboVis consists of
RQ⃝ 1 How do analysts utilize auto-insight recom- four main modules, namely, the data processing, auto-
mendations and manual specification of visualiza- insight recommendation, interactive visual analysis, and
tions collectively as a latent impetus in their EDA dashboard editing and export modules. Particularly, the
process? data processing module (Figure 2(A)) handles common
Previous studies in the field of recommendations indi- data formats, obtains and analyzes data fields, and pro-
cate that the perceived reliability of recommendations is vides necessary processing functions such as sorting,
context-specific [12, 15, 16, 17, 54]. For example, when filtering, and editing. The auto-insight recommendation
users are exploring an unfamiliar dataset, recommen- module serves as assistance to inspect potentially in-
dation services should improve its readability to make teresting data patterns including trending, correlation,
the results more reliable to users. Therefore, we posit distribution, clustering, and outlier detection based on an
a similar effect of auto-insight recommendations need extensible metric repository (covered later). Particularly,
to verify it in the second research question: RQ⃝ 2 How data auto-insight recommendation is embedded in a typi-
does the perceived reliability of auto-insight recom- cal EDA process, recommending interesting visualization
mendations depend on the context of the EDA pro- and enables automatically modifying users’ visualization
cess such as dataset familiarity? specifications to achieve the desired visualizations. The
Analysts may hold different purposes when conduct- interactive visual analysis module supports analysts to
ing EDA. Previous studies indicate that the degree of perform a drag-and-drop interaction on-demand to man-
applying auto-insight recommendation may vary in dif- ually specify visualizations (Figure 2(B)) . The dashboard
ferent EDA scenarios [55]. Given that the structure of an editing and export module (Figure 2(C)) allows analysts
EDA process could be anywhere between a fully open to interactively edit each data insight visualization by
exploration and a specific target-oriented inspection, we “linking + view” technique. The finalized dashboard can
want to know RQ⃝ 3 Would auto-insight recommen- be exported on demand.
dations act differently due to different exploration TurboVis without Auto-Insight Recommendation.
purposes in EDA? TurboVis without auto-insight recommendation version
only supports manual visualization specification on the
basis of graph matching. To be specific, as shown in
4. TurboVis Figure 2(B), after loading the data (⃝),
1 TurboVis automat-
ically splits data fields into dimensions (⃝)2 and metrics
To understand how auto-insight recommendations could
(⃝).
3 Analysts can perform a drag-and-drop interaction to
be leveraged to assist analysts’ EDA processes and ex-
drag any attribute(s) onto the x- or y-axis (⃝)
4 and the dis-
plore its potential “double-edged sword” effect, we design
play area (⃝)5 would simultaneously present the default
and develop TurboVis, an auto-insight recommendation-
chart ranking the first in the recommendation list (⃝). 6
powered exploratory data analysis system. To enhance
We do not explicitly provide other visual encoding chan-
the generalizability of our findings to common data ana-
nels such as size and color due to the observation that
lytics tools, we design TurboVis by reference to existing
participants often get lost to determining where each se-
commercial exploratory data analytics software and tools
lected attribute goes. Therefore, TurboVis automatically
such as Tableau. TurboVis serves as instruments “for de-
determines an appropriate visual encoding channel, e.g.,
sign and understand” and are not intended to suggest
color and size after analysts’ specification of the 𝑥- and
new interaction techniques [56]. Through a prolonged
𝑦-axis. Quantitative attributes can be aggregated in seven
collaborative design process with data analysts, we it-
ways, i.e., sum, mean, maximum, minimum, median, vari-
Figure 3: Auto-insight recommendation of TurboVis. ⃝ 1 Insight options; ⃝
2 Auto-insights and the corresponding natural
language descriptions; ⃝
3 Clusters and outlier detection.
ance, and standard deviation. For instance, analysts can ommendation serves as assistance to help achieve visual-
drag a quantitative attribute to the 𝑥-axis shelf and an- izations with interesting patterns. Asides from manual
other quantitative attribute to the 𝑦-axis shelf to generate visualization specification, this version supports proac-
a scatterplot; to create a bar chart, analysts can drag a tive unsolicited recommendations and reactive query-
nominal attribute to the 𝑥-axis shelf and a quantitative based recommendations. Particularly, proactive unso-
attribute by mean, and the chart can be replaced to a line licited recommendations list all potential auto-insights,
chart if the 𝑥-axis is filled with a temporal attribute. and reactive query-based recommendations list all the
With respect to the recommendation list, in the version recommended charts on the right side of the interface.
without auto-insight recommendation, the list only has This design considers the selected attributes a query and
the results from graph matching. Specifically, we classify generates recommendations relevant to the selected at-
basic visualizations based on input data format, which tributes.
comes in the form of a decision tree [57] that leads to a Regarding the proactive unsolicited recommendations,
set of potentially appropriate visualizations to represent we put the entry to the auto-insight recommendation
the current data configuration. For example, considering above the data table (Figure 2(A)⃝), 2 maximizing the util-
quantitative attributes, if with only one numeric variable, ity of auto-insight recommendation service. Particularly,
graphs that are appropriate in this case are histogram as shown in Figure 3, different types of auto-insight rec-
and density plot. We adopt the graph matching on the ommendations are classified into pull-down list options
basis of two underlying philosophies [57]. First, most (⃝), 1 i.e., patterns measured by statistical metrics and
data analysis can be summarized in about twenty differ- clustering and outlier detection algorithms. Currently,
ent dataset formats. Second, both data and context can auto-insight recommendation supports four insight types:
determine the appropriate chart. Therefore, our graph trend detection shows line charts with obvious increasing
matching scheme consists of identifying and trying all or decreasing temporal pattern between temporal and
feasible chart types to find out which one(s) suit(s) the quantitative dimensions; the correlation between two
data and idea best. In ⃝,7 analysts can rename the current highly correlated attributes between two quantitative di-
visualization or adjust the color and size encoding if no mensions; pairwise distribution comparison concerns two
additional attributes are encoded by color or size. groups where the distributions are significantly differ-
TurboVis with Auto-Insight Recommendation. We ent, and clustering and outlier detection shows potential
design the second version based on prior studies in proac- clusters and outliers. To facilitate quick browsing and in-
tive, e.g., Voder [58] and reactive, e.g., DIVE [59] insight- spection and easy-to-understand, each recommendation
based recommendations. TurboVis with auto-insight rec- contains a concise natural language description which
is generated by templates (⃝),2 such as “Miles per Gallon insight configuration are automatically filled. For ex-
and Weight have a strong correlation”. With respect to ample, given a match of both “type_x” and “type_y” is
clustering and outlier detection, we select t-SNE as the di- a quantitative attribute, we generate a candidate auto-
mensionality reduction technique because it shows supe- insight template with the following fields: ‘mask’ : {‘type’
riority in generating 2D projection that “can reveal mean- : supported_graphs[0], ‘tooltip’ : True}, ‘encoding’ : {‘x’
ingful insights about data, e.g., clusters and outliers” [60]. : {‘field’ : name_x, ‘type’ : “quantitative”},‘y’ : {‘field’ :
In addition, advanced parameter settings are also pro- name_y, ‘type’ : “quantitative”}}, ‘priority’ : priority. Fur-
vided such as quantitative attributes for projection and thermore, according to different auto-insight measures,
t-SNE parameters in terms of perplexity, learning rate, we compute different metrics for each candidate visual-
maximum iterations, and distance metric (⃝). 3 By pre- ization specification. For example, 𝑥 with a quantitative
viewing all the auto-insight recommendations, analysts attribute and 𝑦 with a quantitative attribute are evalu-
can select any of them by clicking on + to submit to the ated by a Spearman correlation coefficient with associated
target dashboard. We employ exhaustion, match, gener- p-value while 𝑥 with a temporal attribute and 𝑦 with a
ate for mining interesting visualizations. We show how quantitative attribute are evaluated by the trend detec-
to generate data auto-insights through the following four tion measure. Candidates with a metric value higher
steps. than a predefined threshold parameter are recommended
Step ⃝ 1 Determining attribute types. After loading to analysts. Based on the results, we fill the ‘𝑟𝑒𝑠𝑢𝑙𝑡’ in
a dataset, TurboVis first gathers metadata of data types the template with {‘correlation’ : correlation, ‘p-value’ :
by iterating on all data records. Particularly, we main- p-value}; ‘message’ : “name_x and name_y has a strong
tain several metadata to determine whether the value correlation depending the value of p-value”.
on a certain attribute is e.g., numeric, date, or coordi- Regarding reactive query-based recommendations, Tur-
nate. We also maintain the number of unique values and boVis with auto-insight recommendations merges auto-
the maximum of replication corresponding to a certain insight recommendations into the recommendation list
data attribute. Data types can be thus determined for that appear in the right panel (Figure 2(B)⃝), 6 which
subsequent processing. tailors the auto-insight recommendations into the EDA
Step ⃝ 2 Maintaining recommendation configura- pipeline. In Figure 4, the left subfigure shows the graph
tion. Each data auto-insight corresponds to a recommen- matching based on a particular data attribute configura-
dation configuration, which consists of six dimensions in tion and the right subfigure displays the auto-insights in
terms of “type_x”, “type_y”, “position_exchange”, “mea- the version with auto-insight recommendations. In other
sure”, “supported_graphs” and “priority”. For example, words, we only display graph matching in the version
trend corresponds to a bar recommendation configura- without auto-insight recommendations and display both
tion: {type_x : temporal, type_y : quantitative, position_ex- in the version with auto-insight recommendations.
change : 0, measure : trend, supported_graphs : [‘bar’],
priority : 0 }, which means that when a temporal attribute
meets a quantitative attribute, we can use bar to visualize 5. Experiment
the relationship with exchangeable axes. “Priority” indi-
To investigate the “double-edged sword” effect of auto-
cates the recommendation priority when demonstrating
insight recommendation design on EDA, we conduct a
all the data auto-insight patterns to audiences. Similarly,
within-subjects study with 18 data analysts in EDA tasks
correlation corresponds to a scatterplot recommenda-
on two datasets.
tion configuration with both “type_x” and “type_y” is a
Participants. We recruit 18 industrial data analysts
quantitative attribute and the “supported_graphs” can be
(9 females, 9 males, age: 28 ± 3.03) from a local Inter-
[‘scatter’].
net bank, most of whom have 2 to 5 years of working
Step ⃝ 3 Preparing and matching all feasible com-
experiences. We invite participants who need to con-
binations to recommendation configuration. We ini-
duct EDA almost every day according to their self report.
tialize feasible combinations to mine potential patterns
Particularly, participants had used tools for EDA, includ-
hidden in the combination of any two different attributes.
ing Tableau (10/18), Excel (18/18), Python (8/10), and
To optimize the exhaustion process, we allow users to
R (12/18). They are representatives of our target users
specify mask attributes thus the calculation will not con-
and could provide us more comprehensive insights. We
sider those masked attributes. We then match each feasi-
compensate participants with a $20 gift card.
ble combination against the targets of the auto-insight
Datasets and Data Processing. We choose two datasets
measures on the basis of “𝑡𝑦𝑝𝑒_𝑥” and “𝑡𝑦𝑝𝑒_𝑦”.
to evaluate the effects of the auto-insight recommenda-
Step ⃝ 4 Generating candidate auto-Insight recom-
tions. The first one is the happiness ranking dataset that
mendation. Upon a match between a feasible combi-
our participants analyze less in their daily work, and it
nation and an auto-insight recommendation configura-
consists of 1093 records with attributes of date, country,
tion, parameters in the corresponding candidate auto-
Figure 4: Graph matching and auto-insight recommendation results.
region, happiness ranking, happiness, GDP per capita, GDP TurboVis to ensure that they have no problems conduct-
per family, healthy, freedom, trustness, generosity, and res- ing the subsequent tasks on their own. In the main study,
idence 1 . The second dataset is the Chinese bank’s annual for participants who start with the one TurboVis version,
report, which is closer to the type of data that our partici- we ask them to conduct a 15-minute data exploration of
pants use everyday, and it comprises 1590 records with 18 the happiness ranking sub dataset (session 1). Then, they
financial attributes. As a demo to introduce our system proceed to another 15-minute data exploration of another
to the participants, we also include a car dataset which happiness ranking sub dataset using another version of
consists of 403 records with attributes of name, miles TurboVis (session 2). Participants are also asked to think
per gallon, cylinders, displacement, horsepower, weight, aloud their ideas when performing all the tasks. Their
acceleration, year, and origin. exploration processes are automatically recorded as sys-
To prepare datasets for the two versions, we split the tem logs for the subsequent quantitative analysis. Then,
happiness ranking dataset and bank dataset into two we repeat the above process by using another dataset,
parts. During the study, the first dataset is used in the i.e., the bank annual report dataset for session 3 and 4.
first data exploration session while the second one is used After finishing all the tasks, participants are required
in the second session. This mitigates potential learning to complete a questionnaire with 7-point Likert scale
effects across the two sessions while ensuring that the questions, followed by a semi-structured interview with
data collected from the two sessions could be compared. each participant to make sense of their ratings and col-
Procedure. After obtaining the participants’ consent, lect their opinions about auto-insight recommendations.
we conduct the experiment in four sessions, each with a The whole experiment lasts around 90 minutes for each
subset of a dataset and one version of TurboVis. In other participant.
words, every participant gets to explore both datasets Measures. In the above-mentioned experiment, we
in different tasks using both versions of our tool. We collect 72 log files (18 participants × 4 sessions). The
counterbalance the order of TurboVis’s version in each log data details every action and the associated entities
dataset to minimize the learning effects. We arrange the conducted by the participant. The actions include but are
experimental procedure in the following steps. First, we not limited to click, select, delete, and drag, and objects
give a tutorial on how to use both versions of TurboVis are like the name of auto-insight recommendation and
(each for 10 minutes; the system is running on ThinkPad attributes. We then derive our dependent measures from
X270 notebook with a 12.5-inch display) to explore data these data in relation to the previously mentioned three
with the car dataset and then allow the participants to research questions.
freely explore the tool for another 10 minutes. During M⃝ 1 Number of inspected visualizations. We count
this process, we encourage the participants to raise any the number of visualization if a specific visualization is
question about the usability, functions, and features of loaded, selected, edited, or added in a dashboard. This
1
https://www.kaggle.com/mathurinache/world-happiness-
measure is a conservative estimate of the inspected visu-
report alizations for the exploration with recommendations.
M⃝ 2 Proportion of supported visualizations. To 6. Results and Analysis
understand whether the potential exploration bias, i.e.,
confining to the scope of auto-insight recommendations, We report the quantitative analysis of participants’ op-
we compare the proportion of visualizations in terms of eration logs and quantitative ratings and feedback on
correlation, trend, pairwise distribution comparison, and the three research questions, as shown in Figure 5. Par-
clusters and outliers among those visualizations logged as ticularly, we analyze the first five measures (M⃝ 1 - ⃝)
5
manual or auto-insight recommendations between with using Wilcoxon signed-rank tests (very appropriate for
and without auto-insight recommendation versions. a repeated measure design where the same subjects are
M⃝ 3 Number of manually specified visualizations. evaluated under two different conditions) with a signifi-
We count the number of manually created visualizations cance level of 0.05 and we report the median values for
for each session. This measure indicates the adoption or the subjective measures collected from the questionnaires
potential reliance on the auto-insight recommendations on each item (M⃝ 6 - ⃝).
9
for EDA, since participants might utilize the auto-insights RQ 1
⃝ Utilizing auto-insights and manual visual-
and thus create fewer visualizations on their own. ization specification collectively. As shown in Fig-
M⃝ 4 Time duration between opening and closing ure 5(a), we find a significant difference in the M⃝ 1 num-
auto-insight recommendations. To investigate the ef- ber of inspected visualizations at a significance level of
fect of how auto-insights may advance their EDA process, 0.05 (both 𝑍 = −3.726, 𝑝 = 0.00019) by using happiness
we record and calculate the time intervals between partic- ranking and bank datasets, respectively. On average, par-
ipant opening and closing (actions recorded in the logs) ticipants inspect more visualizations when using Turbo-
the auto-insight recommendation panel when given the Vis with auto-insight recommendations (𝑀 = 45.33 with
version of our tool with this function, to explore different happiness ranking and 𝑀 = 44.11 with bank dataset)
datasets. than in the without auto-insight recommendation condi-
M⃝ 5 Number of modifications to the recommended tion (𝑀 = 12.61 with happiness ranking and 𝑀 = 15.44
visualizations. When participants are exploring the with bank dataset), indicating a wider coverage of vi-
data using the auto-insight recommendation version, sualizations with auto-insight recommendations. M⃝ 2
they can select any visualization from both the chan- Proportion of visualizations that are supported by recom-
nels of graph matching and the tool’s recommendations. mendation is higher with auto-insight recommendations
When participants drag “asset size” to one axis and the (𝑀 = .465 with happiness ranking and 𝑀 = .36 with
display area would immediately present the auto-insight bank dataset) than without this feature (Figure 5(b)). The
recommendation relevant to this attribute and fill the difference is significant for both the happiness rank-
other axis with information e.g., asset size that has a ing (𝑍 = −3.725, 𝑝 = .000196) and the bank dataset
high correlation relationship. However, one issue we fre- (𝑍 = −3.483, 𝑝 = .000499), indicating that the auto-
quently observe is that if this “automatic completion” is insight recommendations bias participants towards cer-
inconsistent with participants’ intent, they would modify tain types of visualizations during the EDA process.
the recommendation result. Therefore, to investigate to Generally, participants specify more visualization in
what extend participants would directly accept the auto- the absence of auto-insight recommendations (𝑀 = 6.61
insights when they are exploring dataset with different with happiness ranking and 𝑀 = 9.94 with bank dataset)
familiarity, we calculate the number of modification ac- than in the presence of this service (𝑀 = 4.5 with hap-
tions immediately occur after a recommendation result piness ranking and 𝑀 = 8.67 with bank dataset), as
is populated in the display area. shown in Figure 5(c) (M⃝).3 However, we observe differ-
To complement the quantitative data and provide in- ent results of the manual specification of visualization
depth understanding of users’ perceptions towards the on our two datasets. Specifically, the difference is sig-
auto-insight recommendations, we also collect the par- nificant when participants explore the happiness rank-
ticipants’ responses to an end-of-study questionnaires, ing dataset (𝑍 = −2.166, 𝑝 = .03), but not significant
in which we ask them about their preference of tool (𝑍 = −1.742, 𝑝 = .081) when using a bank dataset. Con-
versions when conducing EDA with a clear exploration sidering that our participants frequently analyze bank
task, and whether the auto-insights offer new knowl- data and rarely inspect happiness data in their daily work,
edge. Particularly, we have: M⃝ 6 usefulness of auto- this implies that our participants have a tendency to
insights in an open exploration, M⃝ 7 usefulness of resort to auto-insight recommendations for inspecting
auto-insights with a target-oriented inspection (1 - visual patterns rather than constructing visualizations
Extremely unuseful, 7 - Extremely useful), M⃝ 8 version manually when this service is available for exploring
preference when exploring in an open exploration, an unfamiliar dataset, but not as much when facing a
and M⃝ 9 version preference when exploring with a familiar dataset. “I think I can do more inspection with-
target-oriented inspection (1 - Prefer non-auto-insight out auto-insight recommendations” (P12, male, age: 31).
version a lot, 7 - Prefer auto-insight version a lot). “With my intuition and knowledge, I just want to see the
Figure 5: Results of log data in happiness ranking task and bank task, either with or without auto-insight recommendations.
ns: 𝑝 ≥ .05, ∗ ∶ 𝑝 < .05, ∗∗ ∶ 𝑝 < .01, ∗ ∗ ∗ ∶ 𝑝 < .001.
data this way” (P10, female, age: 29). “I feel like I can ploration or with a target-oriented inspection) in terms of
reason more on my own than the recommendations” (P6, perceived usefulness and version preference. We ask par-
male, age: 26). ticipants in the questionnaire about the perceived useful-
RQ⃝ 2 Auto-insight reliability in EDA. To evaluate ness and preference of auto-insights when they conduct
the acceptance of participants towards the auto-insights an open exploration or a target-oriented inspection in our
conveyed by the recommendations, we measure the M⃝ 4 EDA tasks. During the experiment, we often find that par-
time duration between opening and closing the auto-insight ticipants modify the recommended results by deleting the
recommendation panel given the version of TurboVis with attribute that has been automatically populated on one
this feature. As shown in Figure 5(d), we find a signif- axis. Therefore, we obtain the M⃝ 5 number of modifica-
icant difference in this measure (𝑍 = −3.301, 𝑝 = .001). tions to the recommended visualizations and compare the
An average duration of 2.88 minutes is spent on brows- counts between the two dataset. As shown in Figure 5(e),
ing the results of auto-insight recommendations of the we find that although the mean value of the number of
happiness ranking dataset, compared with an average modification differs, i.e., 𝑀 = 3.39 with the happiness
duration of 1.97 minutes on the bank dataset’s recom- ranking dataset and 𝑀 = 4.56 with the bank dataset, the
mended auto-insights. “I am not familiar with happiness difference is not significant (𝑍 = −1.579, 𝑝 = .114). The
ranking dataset so I have a lower expectation with auto- questionnaire item of M⃝ 6⃝7 usefulness of auto-insight
insight recommendations” (P2, male, age: 28). “I would recommendations in an open exploration or with a target-
try to find why these auto-insights are recommended when oriented inspection also shows that participants appre-
I am exploring the happiness ranking dataset” (P14, fe- ciate the usefulness of auto-insight recommendations
male, age: 25). “I probably would not have figured out the regardless of the tasks (open: 𝑀 = 6.11, 𝑆𝐷 = .96 and tar-
outliers by intuition and I am happy that it has been recom- get: 𝑀 = 6.06, 𝑆𝐷 = .87), suggesting that the acceptance
mended” (P6, male, age: 26). With respect to a relatively of auto-insights in different EDA tasks does not change
more familiar dataset (e.g., bank dataset in our case), the significantly.
auto-insight recommendation serves as assistance for However, in participants’ response to the question
quick verification, “I can quickly identify the interesting M⃝ 8⃝9 version preference in an open exploration or with
auto-insights since I am familiar with them” (P8, female, a target-oriented inspection, the median rating was 5.89
age: 30). with an SD of 0.83 for open exploration on a scale from
RQ⃝ 3 Auto-insights in open exploration and tar- preferring without auto-insight recommendation much
get oriented inspection. We investigate the acceptance more (1) to preferring without auto-insight recommenda-
of auto-insights in different EDA tasks (i.e., an open ex- tions much more (7), suggesting that participants prefer
having the service much more when they only have a Message Reliability. When analysts are quite famil-
vague idea about what they are looking for. “When you iar with the dataset and exploration scenarios, they have
introduce a new dataset that I haven’t see before, I don’t a higher expectation of the auto-insight recommenda-
know where to start” (P2, male, age: 28). “It is hard for tions. They would try to draw conclusions by observing
me to figure out where to go first and auto-insights help the auto-insight recommendation results, e.g., determin-
me with the first step” (P14, female, age: 25). However, ing whether these insights make sense or not, i.e., they
when they have specific questions to investigate, they may question the message reliability. Otherwise, when
prefer TurboVis without auto-insight recommendations they have a vague idea about what they are looking for,
(𝑀 = 3.8, 𝑆𝐷 = 1.1). “I have a very clear target in my they have a lower expectation on the auto-insight recom-
mind so I directly turn to the manually specifying visual- mendations; they appreciate the interestingness of the
izations interface to see what I can get”, “since I am quite recommended patterns, instead of identifying whether
familiar with the data and I know how to select attributes these insights are right or wrong. A design implication
that have relationships” (P8, female, age: 30). “When I is that it is necessary to note next to the auto-insight
was exploring on my own, I feel like I am creating what I recommendations what methods are adopted to generate
want” (P12, male, age: 31). these recommendations and what the system has done
in order to make analysts clear about the underlying
recommendation mechanisms.
7. Discussion Exploration Interruption. When we inferring user
intention from interaction log data, we observed that
In this section, we first discuss the identified “double-
auto-insight recommendations sometimes interrupt ana-
edged sword” effect of auto-insight recommendations on
lysts. When they were exploring a familiar dataset and
the EDA process. Then, we elicit the design implications
trying to construct a desired visualization for inspec-
regarding the observed findings. In the end, we reflect
tion, they commented that they prefer the without auto-
on the limitations of this study.
insight recommendation version in the drag-and-drop
Exploration Bias. We further our awareness of the
process. For example, when analysts drag a data attribute
side effects of auto-insight recommendations on EDA by
that has been involved in an auto-insight recommenda-
first highlighting potential exploration bias and excessive
tion to one axis, TurboVis automatically refreshes the
reliance. Analysts tend to adjust their degree of reliance
recommendation view and lists all the auto-insight rec-
on auto-insight recommendations or manual visualiza-
ommendations related to the specific data attribute and
tion specifications on the basis of data familiarity and
even populates the other fields, “I was intending to put
task structure. For one thing, the domain experts with
‘acceleration’ and ‘cylinders’ together to see what kind of
a high degree of familiarity with data and analytic tasks
visualization results could appear, but the recommended
are more likely to explore more visualizations on their
auto-insights grabbed my attention.” A plausible hypothe-
own. For another, they believe that their domain knowl-
sis is that the low cognitive cost of gleaning insights from
edge and intuitions can help them achieve a smooth EDA
the recommendations makes them too tempting to con-
process when facing familiar data and scenarios. How-
sume, thereby inducing undesirable interruption effects.
ever, if they encounter a new dataset, they might heavily
To mitigate the interruption effects of auto-insight rec-
rely on the auto-insight recommendations by immers-
ommendations, one alternative is to hide the auto-insight
ing themselves in browsing the recommendation results.
recommendation services in a toolbar by following Show
When auto-insight recommendations were present, par-
Me or split the recommended results from the existing
ticipants demonstrated less desire to explore data on their
panel and display them in a separate panel.
own, e.g., the number of manually-specified visualiza-
Limitations. First, although we conducted a pro-
tions drops significantly. Auto-insight recommendation
longed collaborative design with a limited number of
service can produce a large number of recommendations
industrial domain experts, TurboVis is still limited in its
to implicitly impel users to explore more visualizations,
current form with respect to the raised requirements
thus, leading to biased data exploration. A potential de-
collected from their feedback. Second, we derived de-
sign implication is that auto-insight recommendations
sign alternatives by surveying prior systems and con-
should be designed differently based on how people can
ducting iterative design with our collaboration experts.
tolerant auto-insights. In scenarios that welcome diverse
Admittedly, we only tapped into a limited design space of
exploration results, recommendations should be hidden
recommendations, i.e., by providing a button to see the
or at least receive less concerns. Also, tooltips can be pro-
auto-insight recommendation on-demand and linking
vided to explicitly inform analysts that how many auto-
auto-insight recommendations to manual user interac-
insight recommendations or the ratio of auto-insights to
tion. With a different design of auto-insight recommen-
the overall visualizations have been added to the dash-
dation service, users may perceive differently. Third, we
board.
design the auto-insight recommendation service only
based on a limited number of an extensible repository [6] M. Vartak, S. Rahman, S. Madden, A. Parameswaran,
of statistic metrics, which quantify interesting visual- N. Polyzotis, Seedb: Efficient data-driven visualiza-
ization patterns in basic charts. Meanwhile, only ex- tion recommendations to support visual analytics,
perienced data analysts working on a particular set of in: Proceedings of the VLDB Endowment Interna-
problems were included in the user study. The results tional Conference on Very Large Data Bases, vol-
therefore might generalize only to this kind of users. ume 8, NIH Public Access, 2015, p. 2182.
Furthermore, auto-insight recommendations fail to rec- [7] K. Wongsuphasawat, D. Moritz, A. Anand,
ommend any patterns if involving multiple attributes. J. Mackinlay, B. Howe, J. Heer, Voyager: Ex-
Our collaboration experts also commented that there ploratory analysis via faceted browsing of visualiza-
should be more types of recommendations. Future work tion recommendations, IEEE transactions on visu-
will systematically conduct more investigation into more alization and computer graphics 22 (2015) 649–658.
real-world business scenarios to identify more preferable [8] Z. Cui, S. K. Badam, M. A. Yalçin, N. Elmqvist, Dat-
auto-insight types. asite: Proactive visual data exploration with com-
putation of insight-based recommendations, Infor-
mation Visualization 18 (2019) 251–267.
8. Conclusion [9] J. Mackinlay, P. Hanrahan, C. Stolte, Show me:
Automatic presentation for visual analysis, IEEE
In this study, we explore the potential “double-edged
Transactions on Visualization & Computer Graph-
sword” effects of auto-insight recommendations on the
ics 13 (2007) 1137–1144.
EDA process. We demonstrate how auto-insight recom-
[10] K. Wongsuphasawat, Z. Qu, D. Moritz, R. Chang,
mendations could be incorporated into a self-developed
J. Heer, Voyager 2: Augmenting visual analysis
Tableau-like visualization tool termed TurboVis. By com-
with partial view specifications, in: the 2017 CHI
paring two versions of TurboVis, we find that auto-insight
Conference, 2017.
recommendations not only encourage more visualization
[11] Ç. Demiralp, P. J. Haas, S. Parthasarathy, T. Pedapati,
inspections but also introduce biases to data exploration.
Foresight: Recommending visual insights, arXiv
Meanwhile, the perceived level of message reliability and
preprint arXiv:1707.03877 (2017).
interruption of auto-insight recommendation service de-
[12] Q. Li, Z. Peng, H. Zeng, Q. Chen, L. Yi, Z. Wu, X. Ma,
pend on data familiarity and task structures. Our work
T. Chen, Friend network as gatekeeper: A study of
offers initial implications for embedding auto-insight rec-
wechat users’ consumption of friend-curated con-
ommendations into the EDA process.
tents, in: The eighth International Workshop of
Chinese CHI, 2020, pp. 21–31.
Acknowledgments [13] A. Jameson, Adaptive interfaces and agents, in:
The human-computer interaction handbook, CRC
We are grateful for the valuable feedback and comments Press, 2007, pp. 459–484.
provided by the anonymous reviewers. [14] A. D. Jameson, Understanding and dealing with
usability side effects of intelligent processing, AI
Magazine 30 (2009) 23–23.
References [15] F. Du, C. Plaisant, N. Spring, K. Crowley, B. Shnei-
derman, Eventaction: A visual analytics ap-
[1] R. Goyal, A. Tamizharasan, A. Singhal, Exploratory
proach to explainable recommendation for event
data analysis (1999).
sequences, ACM Transactions on Interactive Intel-
[2] M. Diamond, A. Mattia, Data visualization: An
ligent Systems (TiiS) 9 (2019) 1–31.
exploratory study into the software tools used by
[16] D. Kelly, J. Teevan, Implicit feedback for inferring
businesses., Journal of Instructional Pedagogies 18
user preference: a bibliography, in: Acm Sigir Fo-
(2017).
rum, volume 37, ACM New York, NY, USA, 2003,
[3] J. W. Gustafson, C. H. Jones, L. Pape-Haugaard, De-
pp. 18–28.
signing a dashboard to visualize patient informa-
[17] J. Xiao, J. Stasko, R. Catrambone, et al., An empirical
tion, in: Proceedings from The 16th Scandinavian
study of the effect of agent competence on user
Conference on Health Informatics 2018, Aalborg,
performance and perception, in: AAMAS, volume 4,
Denmark August 28–29, 2018, 151, Linköping Uni-
Citeseer, 2004, pp. 178–185.
versity Electronic Press, 2018, pp. 23–28.
[18] G. Dove, K. Halskov, J. Forlizzi, J. Zimmerman, Ux
[4] R. K. Sawyer, Explaining creativity: The science of
design innovation: Challenges for working with
human innovation, Oxford university press, 2011.
machine learning as a design material, in: Chi Con-
[5] J. W. Tukey, We need both exploratory and confir-
ference on Human Factors in Computing Systems,
matory, The American Statistician 34 (1980) 23–25.
2017.
[19] D. Booth, The human-computer interaction hand- knowledge management and sensemaking tools for
book: fundamentals, evolving technologies and intelligence analysts, in: Proceedings of the 15th
emerging applications, L Erlbaum Associates Inc ACM international conference on Information and
15 (2012) 85–87. knowledge management, 2006, pp. 513–521.
[20] M. Vartak, S. Huang, T. Siddiqui, S. Madden, [35] Z. Liu, J. Thompson, A. Wilson, M. Dontcheva, J. De-
A. Parameswaran, Towards visualization recom- lorey, S. Grigg, B. Kerr, J. Stasko, Data illustrator:
mendation systems, Acm Sigmod Record 45 (2017) Augmenting vector design tools with lazy data bind-
34–39. ing for expressive visualization authoring, in: Pro-
[21] J. Tukey, Exploratory Data Analysis, Addison- ceedings of the 2018 CHI Conference on Human
Wesley Pub. Co.„ ???? Factors in Computing Systems, 2018, pp. 1–13.
[22] J. T. Behrens, Principles and procedures of ex- [36] D. Ren, T. Höllerer, X. Yuan, ivisdesigner: Expres-
ploratory data analysis., Psychological Methods sive interactive design of information visualizations,
2 (1997) 131. IEEE transactions on visualization and computer
[23] P. Hanrahan, Analytic database technologies for a graphics 20 (2014) 2092–2101.
new kind of user: the data enthusiast, in: Proceed- [37] S. F. Roth, J. Kolojejchick, J. Mattis, M. C. Chuah,
ings of the 2012 ACM SIGMOD International Con- Sagetools: An intelligent environment for sketch-
ference on Management of Data, 2012, pp. 577–578. ing, browsing, and customizing data-graphics, in:
[24] K. Morton, M. Balazinska, D. Grossman, J. Mackin- Conference companion on Human factors in com-
lay, Support the data enthusiast: Challenges for puting systems, 1995, pp. 409–410.
next-generation data-analysis systems, Proceed- [38] W. Cui, X. Zhang, Y. Wang, H. Huang, B. Chen,
ings of the VLDB Endowment 7 (2014) 453–456. L. Fang, H. Zhang, J.-G. Lou, D. Zhang, Text-to-
[25] M. D’Agostino, D. M. Gabbay, R. Hähnle, J. Posegga, viz: Automatic generation of infographics from
Handbook of tableau methods, Springer Science & proportion-related natural language statements,
Business Media, 2013. IEEE transactions on visualization and computer
[26] O. Troyansky, T. Gibson, C. Leichtweis, QlikView graphics 26 (2019) 906–916.
Your Business: An Expert Guide to Business Dis- [39] Y. Chen, S. Barlowe, J. Yang, Click2annotate: Auto-
covery with QlikView and Qlik Sense, John Wiley mated insight externalization with rich semantics,
& Sons, 2015. in: 2010 IEEE Symposium on Visual Analytics Sci-
[27] L. Grammel, M. Tory, M.-A. Storey, How informa- ence and Technology, IEEE, 2010, pp. 155–162.
tion visualization novices construct visualizations, [40] V. Dibia, Ç. Demiralp, Data2vis: Automatic gen-
IEEE transactions on visualization and computer eration of data visualizations using sequence-to-
graphics 16 (2010) 943–952. sequence recurrent neural networks, IEEE com-
[28] S. Zhu, G. Sun, Q. Jiang, M. Zha, R. Liang, A sur- puter graphics and applications 39 (2019) 33–46.
vey on automatic infographics and visualization [41] Y. Luo, X. Qin, N. Tang, G. Li, Deepeye: Towards au-
recommendations, Visual Informatics (2020). tomatic data visualization, in: 2018 IEEE 34th Inter-
[29] M. Bostock, V. Ogievetsky, J. Heer, D3 data-driven national Conference on Data Engineering (ICDE),
documents, IEEE transactions on visualization and IEEE, 2018, pp. 101–112.
computer graphics 17 (2011) 2301–2309. [42] R. Ding, S. Han, Y. Xu, H. Zhang, D. Zhang, Quick-
[30] A. Satyanarayan, D. Moritz, K. Wongsuphasawat, insights: Quick and automatic discovery of insights
J. Heer, Vega-lite: A grammar of interactive graph- from multi-dimensional data, in: Proceedings of
ics, IEEE transactions on visualization and com- the 2019 International Conference on Management
puter graphics 23 (2016) 341–350. of Data, 2019, pp. 317–332.
[31] D. Li, H. Mei, Y. Shen, S. Su, W. Zhang, J. Wang, [43] B. Tang, S. Han, M. L. Yiu, R. Ding, D. Zhang,
M. Zu, W. Chen, Echarts: A declarative framework Extracting top-k insights from multi-dimensional
for rapid construction of web-based visualization, data, in: Proceedings of the 2017 ACM Interna-
Visual Informatics 2 (2018) 136–146. tional Conference on Management of Data, 2017,
[32] R. López-Cortijo, J. G. Guzmán, A. A. Seco, icharts: pp. 1509–1524.
charts for software process improvement value [44] J. Heer, Agency plus automation: Designing artifi-
management, in: European Conference on Software cial intelligence into interactive systems, Proceed-
Process Improvement, Springer, 2007, pp. 124–135. ings of the National Academy of Sciences 116 (2019)
[33] M. Mauri, T. Elli, G. Caviglia, G. Uboldi, M. Azzi, 1844–1850.
Rawgraphs: a visualisation platform to create open [45] K. Wongsuphasawat, Z. Qu, D. Moritz, R. Chang,
outputs, in: Proceedings of the 12th biannual con- J. Heer, Voyager 2: Augmenting visual analysis
ference on Italian SIGCHI chapter, 2017, pp. 1–5. with partial view specifications, in: the 2017 CHI
[34] N. J. Pioch, J. O. Everett, Polestar: collaborative Conference, 2017.
[46] J. Bobadilla, F. Ortega, A. Hernando, A. Gutiérrez, algorithmic curation in the facebook news feed, in:
Recommender systems survey, Knowledge-Based Proceedings of the 33rd annual ACM conference
Systems 46 (2013) 109–132. on human factors in computing systems, 2015, pp.
[47] S. Zhang, L. Yao, A. Sun, Y. Tay, Deep learning 173–182.
based recommender system: A survey and new [54] A. Jameson, Understanding and dealing with us-
perspectives, ACM Computing Surveys (CSUR) 52 ability side effects of intelligent processing, Ai Mag-
(2019) 1–38. azine 30 (2009) 23–40.
[48] D. Parra, S. Sahebi, Recommender systems: Sources [55] P. D. Adamczyk, B. P. Bailey, If not now, when? the
of knowledge and evaluation metrics, in: Advanced effects of interruption at different moments within
techniques in web intelligence-2, Springer, 2013, pp. task execution, in: Proceedings of the SIGCHI con-
149–175. ference on Human factors in computing systems,
[49] J. L. Herlocker, J. A. Konstan, L. G. Terveen, J. T. 2004, pp. 271–278.
Riedl, Evaluating collaborative filtering recom- [56] J. Wallace, J. Mccarthy, P. C. Wright, P. Olivier, Mak-
mender systems, ACM Transactions on Information ing design probes work, in: Sigchi Conference on
Systems (TOIS) 22 (2004) 5–53. Human Factors in Computing Systems, 2013.
[50] P. Pu, L. Chen, R. Hu, A user-centric evaluation [57] Y. Holtz, C. Healy, From data to viz, 2018, URL
framework for recommender systems, in: Proceed- https://www. data-to-viz. com (2019).
ings of the fifth ACM conference on Recommender [58] A. Srinivasan, S. M. Drucker, A. Endert, J. Stasko,
systems, 2011, pp. 157–164. Augmenting visualizations with interactive data
[51] R. Sinha, K. Swearingen, The role of transparency facts to facilitate interpretation and communication,
in recommender systems, in: CHI’02 extended IEEE transactions on visualization and computer
abstracts on Human factors in computing systems, graphics 25 (2018) 672–681.
2002, pp. 830–831. [59] K. Hu, D. Orghian, C. Hidalgo, Dive: A mixed-
[52] M. Eslami, A. Rickman, K. Vaccaro, A. Aleyasen, initiative system supporting integrated data explo-
A. Vuong, K. Karahalios, K. Hamilton, C. Sandvig, ration workflows, in: Proceedings of the Workshop
” i always assumed that i wasn’t really that close on Human-In-the-Loop Data Analytics, 2018, pp.
to [her]” reasoning about invisible algorithms in 1–7.
news feeds, in: Proceedings of the 33rd annual [60] Q. Li, K. S. Njotoprawiro, H. Haleem, Q. Chen, C. Yi,
ACM conference on human factors in computing X. Ma, Embeddingvis: A visual analytics approach
systems, 2015, pp. 153–162. to comparative network embedding inspection, in:
[53] E. Rader, R. Gray, Understanding user beliefs about 2018 IEEE Conference on Visual Analytics Science
and Technology (VAST), IEEE, 2018, pp. 48–59.