=Paper=
{{Paper
|id=Vol-2903/IUI21WS-ESIDA-1
|storemode=property
|title=
|pdfUrl=https://ceur-ws.org/Vol-2903/IUI21WS-ESIDA-1.pdf
|volume=Vol-2903
|authors=Bharat Kale,Maoyuan Sun,Michael E. Papka
|dblpUrl=https://dblp.org/rec/conf/iui/KaleSP21
}}
====
https://ceur-ws.org/Vol-2903/IUI21WS-ESIDA-1.pdf
Direct Ordering: A Direct Manipulation Based Ordering
Technique
BHARAT KALEa , MAOYUAN SUNa and MICHAEL E. PAPKAa,b
a
Northern Illinois University, DeKalb, IL, USA
b
Argonne National Laboratory, Lemont, IL, USA
Abstract
The ordering of data can reveal patterns in various data visualizations. It significantly affects the expressiveness of visual-
izations by making them clear or cluttered. Clutter makes it hard for users to perceive patterns, even for a small dataset.
Almost all visualization tools available today support ordering through indirect manipulation techniques where users rely on
a widget (e.g., a button or a checkbox) to perform ordering in some predefined way. For example, using widgets for sorting
a selected subset of data involves creating widgets to filter the data, highlight the filtered data, and then apply the order-
ing. This work presents a new interaction technique based on direct manipulation, which supports flexible visual ordering.
This direct manipulation-based interaction helps to visually filter the data and then order them. We demonstrate how the
proposed interaction technique works with bar charts and matrices, two heavily used visualization techniques that involve
ordering.
Keywords
Data Visualization, Direct Manipulation, Visual Ordering, Interaction
1. Introduction context and finally widgets to specify ordering. Instead,
direct manipulation based techniques can offer a better
Ordering is a critical function offered by various visual- solution by providing simple ways of selecting visual ele-
izations (e.g., axes ordering in parallel coordinates [1], ments (e.g., using an interactive lasso selection [5]). Once
entity ordering in multiple lists [2, 3], and matrix ordering a selection is finished, it is possible to apply ordering with
[4]). It helps to organize visual elements in a meaningful demonstrative interactions [6].
way, revealing patterns that are hard to see otherwise. A In this work, we present a direct manipulation based
flexible ordering capability potentially supports users ex- interaction technique to flexibly order visual elements by
ploring data from multiple perspectives. Thus, providing creating direct ordering. Direct ordering enables users
an interactive solution to achieve this goal is vital for the to directly interact with the visual glyphs and manipu-
sensemaking of data. late the position channel to organize visual elements in a
Two types of interaction styles can be used in inter- preferred order. Also, it allows applying the interaction
active visualizations: indirect manipulation, and direct on selections in the same way as that on the entire visu-
manipulation. The former is enabled by using UI wid- alization. Applying the interactive ordering to selections
gets. The latter requires interacting with visual elements (when present) has applications in places such as multiple
directly. The Windows, Icons, Menus, Pointer (WIMP) coordinated views [7], in which selections applied in one
interfaces created with indirect manipulation techniques view are reflected in other views. It is useful to have an
require adjusting parameters, which draws user atten- interaction technique that can target interactions only on
tion away from the visualization. Sometimes creating the selected data parts. In summary, this work highlights
selections may be a problem for the WIMP interface. the following two contributions:
For example, users want to select a set of points in a
scatter plot, but the points cannot be defined using a sim- • We present a direct manipulation based interac-
ple range filter. To achieve this behavior with indirect tion technique for flexibly ordering visual ele-
manipulation based techniques is tedious, as it involves ments.
developing widgets to define filters which are difficult to • We demonstrate the proposed technique for a bar
express, and then developing widgets to provide visual chart and a matrix visualization.
Joint Proceedings of the ACM IUI 2021 Workshops, April 13–17, 2021,
College Station, USA 2. Related Work
" bkale@niu.edu (B. KALE); smaoyuan@niu.edu (M. SUN);
papka@niu.edu (M. E. PAPKA)
Ordering (also known as seriation) involves assigning an
0000-0003-3653-228X (B. KALE); 0000-0002-0990-2620
(M. SUN); 0000-0002-6418-5767 (M. E. PAPKA) order to visually displayed data items. It has been studied
© 2021 Copyright for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
in a close correlation with specific type of visualization
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org) (e.g., matrix visualizations [8, 4] and parallel coordinates
[1]). They focus on improving the ordering algorithms following requirements that our proposed interaction
but not on interactions to achieve some desired orders. should provide:
Direct manipulation highlights manipulating displayed
R1 Allow interaction with the visual representations
visual elements directly. Shneiderman [9] states that
to foster exploration. Using visual representa-
visibility of the object of interest; rapid, reversible, and
tions to enable user interaction solves the prob-
incremental actions are central ideas behind direct ma-
lem of shifting user focus [20].
nipulation. It reduces user cognitive load by decreasing
R2 Ability to apply the interaction to a selected sub-
the distance between the source and target of the inter-
set of data in a visualization. Users may want
action [10]. Direct manipulation interfaces are getting
to see entire data in a visualization but empha-
popular nowadays as they remove the dependency on
size on only parts of the data and further apply
extra widgets (e.g., menus and buttons). Sarvghad et
interactions to them.
al. [11] created an embedded interactive technique to
R3 Provide visual cues to help users understand the
merge and split bars in bar charts and histograms for
available actions to achieve a desired interaction
manipulating data groupings. DimpVis [12] allows di-
[21].
rect interaction with various visual encoding channels
to navigate visualizations in the temporal dimension. In-
terver [13] offers dynamic ordering as a user selects a 4. Direct Ordering
numerical interval via brushing. Saket et al. [6] proposed
a new paradigm, visualization by demonstration, which 4.1. Design Considerations
allows directly manipulating graphical encoding and rec-
ommends view transformations accordingly. Vuillemot To meet these requirements, we particularly focus on the
and Perin [14] applied direct manipulation on ranking following design considerations.
tables to navigate rows of interest temporally. Saket et D1 Visual marks used in a visualization should be
al. [15] conducted a qualitative study on scatter-plot, bar directly moved as desired without disturbing the
chart, and histogram to understand how people convey relationship among data items (R1). For example,
the intended operations if they use only direct manipula- moving a glyph to a new position requires adjust-
tion based interactions. Another interesting application ing all other glyphs in visualization not to lose
of direct manipulation techniques is the semantic inter- the inherent relationship.
action introduced by Endert et al. [16]. It highlights that D2 Ordering should be supported via simple gestures
as users directly interact with visual elements, the mean- (R3).
ings of such interactions should be considered as “soft D3 Show visual cues to help user understand what
data" by a visual analysis system, and such data further to interact with, where and how to perform an
steers underlying computation models responsible for interaction (R3).
information foraging. D4 Apply the interaction to a selection. In case there
Support for ordering visual elements with direct ma- is no selection, use the entire chart as the selection
nipulation based techniques has also been studied. Siir- (R2).
tola [17] proposed reorderable matrix as an interactive ap-
proach to apply Bertin’s analysis [18] to physical matrices.
Later Perin et al. [19] created Bertifier as a comprehensive 4.2. Implementation
implementation of Bertin’s original ideas. Reorderable We implemented the technique in JavaScript and SVG us-
matrix [17] has limited functions but it is completely ing D3. The prototype we created for the demonstration
based on direct manipulation. In contrast, bertifier [19] uses drag functionality to allow users to interactively or-
sacrifices directness, to some extent, to support more fea- ganize visual elements. The drag event tracks the state of
tures and achieve the desired results on multiple objects visual elements to detect cases, such as sorting in ascend-
of interest at a time (e.g., replicating an operation on one ing or descending order in a bar chart and clustering in a
row or column to the entire matrix). All these works are matrix. Also, during the ordering process, visual cues are
done only in the context of matrix visualization. Our offered to handle ambiguity. For example, consider two
goal is to develop an approach that can support flexibly scenarios: 1) moving the tallest bar to the first position,
ordering visual elements for a broad set of visualizations. and 2) ordering the entire bar chart in descending order.
Both cases require a user to move the tallest bar to the left
side. To resolve such ambiguity, as a user starts moving
3. Requirement Analysis the tallest bar, left and right most boundaries appear to
Based on the literature on ordering in visualizations and indicate that moving the tallest bar beyond them only
the role of direct manipulation, we have derived the triggers the sorting operation (scenario 2) otherwise, a
simple position change (scenario 1).
4.2.1. Automated vs. Manual Ordering
While automated ordering has benefits such as speed and
reproducibility, it brings problems like interpretability
when the number of dimensions is high. It is difficult
to include user domain knowledge in an automated or-
dering process. On the other hand, manual ordering
is time-consuming. Hence, direct ordering employs a
combination of automated and manual ordering. Direct
Ordering first uses an automated approach to provide a
preliminary order. Then users can interactively adjust
the automated ordering results.
4.2.2. All vs. Selections Ordering
The proposed technique applies ordering to visual glyphs
corresponding to all data points if either the selection has
all the data or there is no selection at all. With selection,
the technique can restrict the ordering only to visual
elements in the selection. Figure 1: Direct ordering in a bar chart: ordering all items
In next two sections, we present the implementation (A), changing the position of a bar (B), ordering all items
of direct ordering in bar charts and matrices. Our im- in-between two selected items (C), ordering neighboring
plementation gives initial evidence that it is possible to selected items (D), and ordering non-neighboring selected
understand user intention based on user interactions items (E).
and is a first step toward exploring the visualization by
demonstration [6] interaction paradigm.
in a bar chart. The selections, in figure 1, are created
by clicking on individual bars. With a selection, as user
5. Use Case: Bar Charts starts moving a bar, boundaries are shown beside the
left and right most bars in the selection. This enables
A bar chart encodes data values with height and position
sorting a selection by dragging the tallest bar (in the
of bars. It is used to compare values, so ordering is nec-
selection) beyond the boundaries of this selection (D4).
essary. As ordering in a bar chart is in one dimension,
This avoids dragging the tallest bar to the boundaries of
dragging along one axis can achieve a desired result.
the entire chart even when the area of interest is very
small. Also, in a selection, the bars within the selection
5.1. Changing Bar Position can be either contiguous or dis-contiguous. In either
To change position of a bar, a user drags the bar to a case, only bars in a selection will be ordered (i.e., in a
desired position (D1). When the cursor is hovered on a case of dis-contiguous bars, the sorting does not make
bar, it gets highlighted indicating that a user can interact the selected bars contiguous as is shown in Figure 1 (E)).
with it (D3).
Sorting is a particular case of ordering, where all items 5.3. For Grouped Bar Charts
are ordered in either an ascending or descending order.
Direct ordering can be applied to a grouped bar chart,
While users can perform sorting by dragging each bar, it
as shown in Figure 2 (B1-B4). Groups can be sorted by
is time-consuming. We supported sorting with a simple
selecting a criteria like a common sub-type across groups
gesture of dragging the tallest bar to the leftmost (de-
(see Figure 2 (B1-B2)). In Figure 2 (B1), there is no specific
scending) or rightmost (ascending) end of a bar chart
ordering in groups. Once a bar is selected, 65 Years and
(D2). Figure 1 (A) shows the sorting operation in ascend-
Over in this case, bars that belong to the same type across
ing order. Moreover, animated transitions are applied to
other groups are highlighted. Now treating each group
help users understand the changes.
as a single bar in a simple bar chart, the groups can be
rearranged using the group containing the tallest bar
5.2. Applying on Selections among the selected bars. Figure 2 (B2) shows the result
Direct ordering allows applying it to a selection in the of sorted groups in a descending order of 65 Years and
same way as that to an entire bar chart. Figure 1 (C, D, Over. Moreover, we can propagate the ordering applied
and E) shows how direct ordering works on a selection to bars in one group to all other groups. In Figure 2 (B3),
bars in the group CA are sorted in descending order and
Figure 2: Direct ordering on a bar chart (A1-A4), a grouped bar chart (B1-B4), and a matrix (C1-C4): ordering all bars in a
descending order using the tallest bar and the bounds (A1-A2), ordering selected bars in an ascending order using the tallest
bar in the selection and the bounds (A3-A4), ordering all groups in a descending order of selected type using the group with
the tallest bar of the selected type and the bounds (B1-B2), propagating the order of bars in one group to all groups using the
tallest bar in the selected group and its bounds (B3-B4), ordering all columns in a matrix based on visual similarity (C1-C2),
ordering a matrix with an automatic ordering algorithm – first by visual similarity of columns and then rows to reveal clusters
(C3), manually fine-tuning automated ordering results (C4).
the same order is propagated across all other groups (see 6.2. Using Visual Similarity
Figure 2 (B4)). This follows the paradigm of visualization
A selected set of rows/columns are automatically orga-
by demonstration [6], as the system learns from user
nized by visual similarity [19]. We use differences in
interaction in one group and applies that to others.
visual encodings instead of raw data values to compute
similarity. The visual similarity computation algorithm
6. Use Case: Matrix takes two steps. First, it takes visual encoding values of
a selected set of rows or columns as input vectors. For
Matrix has been used to display network data and tabular example, as each column is encoded independently, we
data. The data values, in case of tabular data, and rela- used the circles’ radius as the input vectors for computing
tionships, in case of network data, are encoded in cells distance. Second, using the Euclidean distance metric, it
using colors or visual glyphs. Bertin’s matrices [18] is a computes an optimal order that minimizes the sum of
typical example. Ordering is critical in a matrix to reveal distances between consecutive vectors [22].
clusters, and it is performed in two dimensions. Hence To enable multi row/column ordering, we used a de-
the proposed approach needs two drag operations, one sign similar to Crossets [23]. Selecting a row/column
along each dimension, to achieve the desired result. header and dragging to consecutive row/column headers
highlights and adds them to a selected subset, which is
6.1. Rearranging rows/columns then sent to the similarity calculation algorithm. In Fig-
ure 3, (C) shows column based ordering and (D) presents
To re-position a row/column, a user needs to drag and row based ordering. As updates are incremental, apply-
drop it. Figure 3 (A, B) shows dragging a column and a ing a row based ordering followed by a column based or
row to new positions, respectively. It acts as a fundamen- vice-versa organizes a matrix.
tal interaction and allows manually adjusting the results The result shown in Figure 2 (C3) is obtained by apply-
of an automatic ordering. We used the initial dragging ing a 2D ordering to the matrix shown in Figure 2 (C1).
direction to restrict the drag operation to either row or First, the matrix is ordered by visual similarity across all
column. For example, if an initial dragging is close to the columns (Figure 2 (C2)), and the resulting matrix is
moving up or down, we use only row dragging, other- ordered again using all rows. Finally, if a user wants to
wise column. In dragging, animated transitions are used tweak the automated ordering results, the manual order-
to help users understand and follow changes. ing that we supported helps. Figure 2 (C4) shows some
of the rows from the Figure 2 (C3) are manually adjusted.
Along with the positive comments, suggested improve-
ments are listed as follows.
• Why does it only support using the highest bar, not
the shortest one? Finding the highest bar needs cog-
nitive effort. [demos 1, 2]
• It would be better to use gestures for ordering neigh-
boring bars. [demos 1, 2]
• can I brush on the y-axis, and ordering bars based
on my selection? [demos 1, 2]
• why does it only allow me to select neighboring
rows/cols for sorting? The ordering behavior for
non-neighboring rows/cols should be consistent with
that in bar chart. [demo 3]
8. Conclusion
We presented direct ordering, a direct manipulation based
interactive technique to order visual elements in a visu-
alization. We implemented the technique on bar charts
and matrices and evaluated the technique by gathering
an initial expert feedback. The feedback shows interest
Figure 3: Direct ordering in a matrix: manually changing the and potential usefulness of the proposed interaction tech-
order of a column (A), a row (B), ordering all columns (C), all nique in terms of its flexibility and ability to understand
rows (D), and all columns followed by all rows (E) by visual user intention. We believe our proposed technique can
similarity.
help design and build intelligent interactive systems, e.g.,
designing consistent ways of organizing visual elements
in well-aligned 1-D and 2-D layouts, enriching the de-
This function offers more flexibility. sign space of semantic interactions and visualization by
demonstration by contributing novel direct manipulation
based techniques to order visual elements, and help in
7. Initial Expert Feedback designing techniques to understand user intention by
capturing and analyzing user interactions and recom-
To evaluate direct ordering, we showed it to a domain
mending new possible transformations matching user
expert in information visualization and visual analytics.
interest. While this work demonstrates the use of inter-
The feedback indicated the usefulness of the technique
active ordering, there are few limitations that need to be
in terms of flexibly organizing visual elements. During
studied further.
the evaluation, similar to those displayed in Figure 2,
First, test possible generalizability of direct ordering
three demonstrations are explored: 1) ordering bars in
to a variety of visualizations. This work can be extended
a bar chart, 2) ordering groups and bars with in a group
to conduct a study on applying it to diverse visualiza-
in a group bar chart, and 3) ordering rows and columns
tions (e.g., parallel coordinates, pie charts, and radial bar
in a matrix. Key positive comments from the feedback
charts). Similarly, conducting a study to understand its
include:
application for existing visualization tools and analyzing
• It’s really good for me to flexibly order any bars. the differences in flexibility and ease of use helps further
This explicit manual ordering implies user inten- identify trade-offs of using it.
tion and capturing that will be helpful. [demos 1, Second, the initial evaluation of direct ordering came
2] from one domain expert only. To gain an in-depth un-
derstanding of this technique’s impact on the interactive
• This technique can help with matrix ordering, such
user interface, we need to conduct user studies in the
as after an automatic matrix sorting, user manu-
future, using diverse real-world datasets. One key area
ally adjust the order. If such user adjustment can
we plan to focus on moving forward is understanding
be analyzed and sent back to the automatic order-
how to capture user intention based on user interactions.
ing algorithms, then we can iteratively order the
We could then recommend further possible interactions
whole matrix. [demo 3]
and transformations that might be of interest to the user.
Thus, instead of merely updating a view after the user [10] E. L. Hutchins, J. D. Hollan, D. A. Norman, Di-
actions, we can empower the user interface with intelli- rect manipulation interfaces, Human–Computer
gence to suggest users with possibilities. Interaction 1 (1985) 311–338.
[11] A. Sarvghad, B. Saket, A. Endert, N. Weibel, Em-
bedded merge & split: Visual adjustment of data
9. Acknowledgments grouping, IEEE transactions on visualization and
computer graphics 25 (2018) 800–809.
This research was supported in part by the NSF Grant
[12] B. Kondo, C. Collins, Dimpvis: Exploring time-
IIS-2002082 and the Argonne Leadership Computing Fa-
varying information visualizations by direct ma-
cility, which is a U.S. Department of Energy Office of
nipulation, IEEE transactions on visualization and
Science User Facility operated under contract DE-AC02-
computer graphics 20 (2014) 2003–2012.
06CH11357.
[13] M. Sun, G. Convertino, Interver: Drilling into
categorical-numerical relationships, in: Proceed-
References ings of the International Working Conference on
Advanced Visual Interfaces, 2016, pp. 44–51.
[1] L. F. Lu, M. L. Huang, T.-H. Huang, A new axes [14] R. Vuillemot, C. Perin, Investigating the direct ma-
re-ordering method in parallel coordinates visual- nipulation of ranking tables for time navigation, in:
ization, in: 2012 11th International Conference Proceedings of the 33rd Annual ACM Conference
on Machine Learning and Applications, volume 2, on Human Factors in Computing Systems, 2015, pp.
IEEE, 2012, pp. 252–257. 2703–2706.
[2] J. Stasko, C. Görg, Z. Liu, Jigsaw: supporting inves- [15] B. Saket, S. Huron, C. Perin, A. Endert, Investigating
tigative analysis through interactive visualization, direct manipulation of graphical encodings as a
Information visualization 7 (2008) 118–132. method for user interaction, IEEE transactions on
[3] M. Sun, J. Zhao, H. Wu, K. Luther, C. North, N. Ra- visualization and computer graphics 26 (2019) 482–
makrishnan, The effect of edge bundling and se- 491.
riation on sensemaking of biclusters in bipartite [16] A. Endert, P. Fiaux, C. North, Semantic interaction
graphs, IEEE transactions on visualization and com- for visual text analytics, in: Proceedings of the
puter graphics 25 (2018) 2983–2998. SIGCHI conference on Human factors in computing
[4] M. Behrisch, B. Bach, N. Henry Riche, T. Schreck, systems, 2012, pp. 473–482.
J.-D. Fekete, Matrix reordering methods for table [17] H. Siirtola, Interaction with the reorderable matrix,
and network visualization, in: Computer Graphics in: IEEE International Conference on Information
Forum, volume 35, Wiley Online Library, 2016, pp. Visualization, IEEE, 1999, pp. 272–277.
693–716. [18] J. Bertin, Graphics and graphic information process-
[5] M. J. McGuffin, I. Jurisica, Interaction techniques for ing, Walter de Gruyter, 2011.
selecting and manipulating subgraphs in network [19] C. Perin, P. Dragicevic, J. D. Fekete, Revisiting
visualizations, IEEE Transactions on Visualization bertin matrices: New interactions for crafting tabu-
and Computer Graphics 15 (2009) 937–944. lar visualizations, IEEE Transactions on Visualiza-
[6] B. Saket, H. Kim, E. T. Brown, A. Endert, Visual- tion and Computer Graphics 20 (2014) 2082–2091.
ization by demonstration: An interaction paradigm [20] B. Saket, A. Srinivasan, E. D. Ragan, A. Endert, Eval-
for visual data exploration, IEEE transactions on uating interactive graphical encodings for data vi-
visualization and computer graphics 23 (2016) 331– sualization, IEEE transactions on visualization and
340. computer graphics 24 (2017) 1316–1330.
[7] J. C. Roberts, State of the art: Coordinated & mul- [21] D. Norman, Things that make us smart: Defend-
tiple views in exploratory visualization, in: Inter- ing human attributes in the age of the machine,
national Conference on Coordinated and Multiple Diversion Books, 2014.
Views in Exploratory Visualization, IEEE, 2007, pp. [22] Z. Bar-Joseph, D. K. Gifford, T. S. Jaakkola, Fast
61–71. optimal leaf ordering for hierarchical clustering,
[8] I. Liiv, Seriation and matrix reordering methods: An Bioinformatics 17 (2001) S22–S29.
historical overview, Statistical Analysis and Data [23] C. Perin, P. Dragicevic, J.-D. Fekete, Crossets: Ma-
Mining: The ASA Data Science Journal 3 (2010) nipulating multiple sliders by crossing, in: Pro-
70–91. ceedings of the 41st Graphics Interface Conference,
[9] B. Shneiderman, 1.1 direct manipulation: a step 2015, p. 233–240.
beyond programming languages, Sparks of innova-
tion in human-computer interaction 17 (1993) 1993.