The ordering of data can reveal patterns in various data visualizations. It significantly afects the expressiveness of visualizations by making them clear or cluttered. Clutter makes it hard for users to perceive patterns, even for a small dataset. Almost all visualization tools available today support ordering through indirect manipulation techniques where users rely on a widget (e.g., a button or a checkbox) to perform ordering in some predefined way. For example, using widgets for sorting a selected subset of data involves creating widgets to filter the data, highlight the filtered data, and then apply the ordering. This work presents a new interaction technique based on direct manipulation, which supports flexible visual ordering. This direct manipulation-based interaction helps to visually filter the data and then order them. We demonstrate how the proposed interaction technique works with bar charts and matrices, two heavily used visualization techniques that involve ordering.
context and finally widgets to specify ordering. Instead, direct manipulation based techniques can ofer a better Ordering is a critical function ofered by various visual- solution by providing simple ways of selecting visual eleizations (e.g., axes ordering in parallel coordinates [1], ments (e.g., using an interactive lasso selection [5]). Once entity ordering in multiple lists [2, 3], and matrix ordering a selection is finished, it is possible to apply ordering with [4]). It helps to organize visual elements in a meaningful demonstrative interactions [6]. way, revealing patterns that are hard to see otherwise. A In this work, we present a direct manipulation based lfexible ordering capability potentially supports users ex- interaction technique to flexibly order visual elements by ploring data from multiple perspectives. Thus, providing creating direct ordering. Direct ordering enables users an interactive solution to achieve this goal is vital for the to directly interact with the visual glyphs and manipusensemaking of data. late the position channel to organize visual elements in a
Two types of interaction styles can be used in inter- preferred order. Also, it allows applying the interaction active visualizations: indirect manipulation, and direct on selections in the same way as that on the entire visumanipulation. The former is enabled by using UI wid- alization. Applying the interactive ordering to selections gets. The latter requires interacting with visual elements (when present) has applications in places such as multiple directly. The Windows, Icons, Menus, Pointer (WIMP) coordinated views [7], in which selections applied in one interfaces created with indirect manipulation techniques view are reflected in other views. It is useful to have an require adjusting parameters, which draws user atten- interaction technique that can target interactions only on tion away from the visualization. Sometimes creating the selected data parts. In summary, this work highlights selections may be a problem for the WIMP interface. the following two contributions: For example, users want to select a set of points in a scatter plot, but the points cannot be defined using a sim- • We present a direct manipulation based interacple range filter. To achieve this behavior with indirect tion technique for flexibly ordering visual elemanipulation based techniques is tedious, as it involves ments. developing widgets to define filters which are dificult to • We demonstrate the proposed technique for a bar express, and then developing widgets to provide visual chart and a matrix visualization.
order to visually displayed data items. It has been studied in a close correlation with specific type of visualization (e.g., matrix visualizations [8, 4] and parallel coordinates [1]). They focus on improving the ordering algorithms following requirements that our proposed interaction but not on interactions to achieve some desired orders. should provide:
Direct manipulation highlights manipulating displayed visual elements directly. Shneiderman [9] states that R1 Allow interaction with the visual representations visibility of the object of interest; rapid, reversible, and to foster exploration. Using visual representaincremental actions are central ideas behind direct ma- tions to enable user interaction solves the probnipulation. It reduces user cognitive load by decreasing lem of shifting user focus [20]. the distance between the source and target of the inter- R2 Ability to apply the interaction to a selected subaction [10]. Direct manipulation interfaces are getting set of data in a visualization. Users may want popular nowadays as they remove the dependency on to see entire data in a visualization but emphaextra widgets (e.g., menus and buttons). Sarvghad et size on only parts of the data and further apply al. [11] created an embedded interactive technique to interactions to them. merge and split bars in bar charts and histograms for R3 Provide visual cues to help users understand the manipulating data groupings. DimpVis [12] allows di- available actions to achieve a desired interaction rect interaction with various visual encoding channels [21]. to navigate visualizations in the temporal dimension. Interver [13] ofers dynamic ordering as a user selects a 4. Direct Ordering numerical interval via brushing. Saket et al. [6] proposed a new paradigm, visualization by demonstration, which 4.1. Design Considerations allows directly manipulating graphical encoding and recommends view transformations accordingly. Vuillemot To meet these requirements, we particularly focus on the and Perin [14] applied direct manipulation on ranking following design considerations. tables to navigate rows of interest temporally. Saket et D1 Visual marks used in a visualization should be al. [15] conducted a qualitative study on scatter-plot, bar directly moved as desired without disturbing the chart, and histogram to understand how people convey relationship among data items (R1). For example, the intended operations if they use only direct manipula- moving a glyph to a new position requires adjusttion based interactions. Another interesting application ing all other glyphs in visualization not to lose of direct manipulation techniques is the semantic inter- the inherent relationship. action introduced by Endert et al. [16]. It highlights that D2 Ordering should be supported via simple gestures as users directly interact with visual elements, the mean- (R3). ings of such interactions should be considered as “soft D3 Show visual cues to help user understand what data" by a visual analysis system, and such data further to interact with, where and how to perform an steers underlying computation models responsible for interaction (R3). information foraging. D4 Apply the interaction to a selection. In case there
Support for ordering visual elements with direct ma- is no selection, use the entire chart as the selection nipulation based techniques has also been studied. Siir- (R2). tola [17] proposed reorderable matrix as an interactive approach to apply Bertin’s analysis [18] to physical matrices.
Later Perin et al. [19] created Bertifier as a comprehensive 4.2. Implementation implementation of Bertin’s original ideas. Reorderable We implemented the technique in JavaScript and SVG usmatrix [17] has limited functions but it is completely ing D3. The prototype we created for the demonstration based on direct manipulation. In contrast, bertifier [ 19] uses drag functionality to allow users to interactively orsacrifices directness, to some extent, to support more fea- ganize visual elements. The drag event tracks the state of tures and achieve the desired results on multiple objects visual elements to detect cases, such as sorting in ascendof interest at a time (e.g., replicating an operation on one ing or descending order in a bar chart and clustering in a row or column to the entire matrix). All these works are matrix. Also, during the ordering process, visual cues are done only in the context of matrix visualization. Our ofered to handle ambiguity. For example, consider two goal is to develop an approach that can support flexibly scenarios: 1) moving the tallest bar to the first position, ordering visual elements for a broad set of visualizations. and 2) ordering the entire bar chart in descending order. Both cases require a user to move the tallest bar to the left 3. Requirement Analysis side. To resolve such ambiguity, as a user starts moving the tallest bar, left and right most boundaries appear to Based on the literature on ordering in visualizations and indicate that moving the tallest bar beyond them only the role of direct manipulation, we have derived the triggers the sorting operation (scenario 2) otherwise, a simple position change (scenario 1).
While automated ordering has benefits such as speed and reproducibility, it brings problems like interpretability when the number of dimensions is high. It is dificult to include user domain knowledge in an automated ordering process. On the other hand, manual ordering is time-consuming. Hence, direct ordering employs a combination of automated and manual ordering. Direct Ordering first uses an automated approach to provide a preliminary order. Then users can interactively adjust the automated ordering results.
corresponding to all data points if either the selection has all the data or there is no selection at all. With selection, the technique can restrict the ordering only to visual elements in the selection. Figure 1: Direct ordering in a bar chart: ordering all items
In next two sections, we present the implementation (A), changing the position of a bar (B), ordering all items of direct ordering in bar charts and matrices. Our im- in-between two selected items (C), ordering neighboring plementation gives initial evidence that it is possible to selected items (D), and ordering non-neighboring selected understand user intention based on user interactions items (E). and is a first step toward exploring the visualization by demonstration [6] interaction paradigm.
5.1. Changing Bar Position
desired position (D1). When the cursor is hovered on a bar, it gets highlighted indicating that a user can interact with it (D3).
Sorting is a particular case of ordering, where all items are ordered in either an ascending or descending order. While users can perform sorting by dragging each bar, it is time-consuming. We supported sorting with a simple gesture of dragging the tallest bar to the leftmost (descending) or rightmost (ascending) end of a bar chart (D2). Figure 1 (A) shows the sorting operation in ascending order. Moreover, animated transitions are applied to help users understand the changes. 5.2. Applying on Selections Direct ordering allows applying it to a selection in the same way as that to an entire bar chart. Figure 1 (C, D, and E) shows how direct ordering works on a selection in a bar chart. The selections, in figure 1, are created by clicking on individual bars. With a selection, as user starts moving a bar, boundaries are shown beside the left and right most bars in the selection. This enables sorting a selection by dragging the tallest bar (in the selection) beyond the boundaries of this selection (D4). This avoids dragging the tallest bar to the boundaries of the entire chart even when the area of interest is very small. Also, in a selection, the bars within the selection can be either contiguous or dis-contiguous. In either case, only bars in a selection will be ordered (i.e., in a case of dis-contiguous bars, the sorting does not make the selected bars contiguous as is shown in Figure 1 (E)). 5.3. For Grouped Bar Charts
as shown in Figure 2 (B1-B4). Groups can be sorted by selecting a criteria like a common sub-type across groups (see Figure 2 (B1-B2)). In Figure 2 (B1), there is no specific ordering in groups. Once a bar is selected, 65 Years and Over in this case, bars that belong to the same type across other groups are highlighted. Now treating each group as a single bar in a simple bar chart, the groups can be rearranged using the group containing the tallest bar among the selected bars. Figure 2 (B2) shows the result of sorted groups in a descending order of 65 Years and Over. Moreover, we can propagate the ordering applied to bars in one group to all other groups. In Figure 2 (B3), bars in the group CA are sorted in descending order and the same order is propagated across all other groups (see Figure 2 (B4)). This follows the paradigm of visualization by demonstration [6], as the system learns from user interaction in one group and applies that to others.
Matrix has been used to display network data and tabular data. The data values, in case of tabular data, and relationships, in case of network data, are encoded in cells using colors or visual glyphs. Bertin’s matrices [18] is a typical example. Ordering is critical in a matrix to reveal clusters, and it is performed in two dimensions. Hence the proposed approach needs two drag operations, one along each dimension, to achieve the desired result. 6.1. Rearranging rows/columns To re-position a row/column, a user needs to drag and drop it. Figure 3 (A, B) shows dragging a column and a row to new positions, respectively. It acts as a fundamental interaction and allows manually adjusting the results of an automatic ordering. We used the initial dragging direction to restrict the drag operation to either row or column. For example, if an initial dragging is close to moving up or down, we use only row dragging, otherwise column. In dragging, animated transitions are used to help users understand and follow changes. 6.2. Using Visual Similarity A selected set of rows/columns are automatically organized by visual similarity [19]. We use diferences in visual encodings instead of raw data values to compute similarity. The visual similarity computation algorithm takes two steps. First, it takes visual encoding values of a selected set of rows or columns as input vectors. For example, as each column is encoded independently, we used the circles’ radius as the input vectors for computing distance. Second, using the Euclidean distance metric, it computes an optimal order that minimizes the sum of distances between consecutive vectors [22].
To enable multi row/column ordering, we used a design similar to Crossets [23]. Selecting a row/column header and dragging to consecutive row/column headers highlights and adds them to a selected subset, which is then sent to the similarity calculation algorithm. In Figure 3, (C) shows column based ordering and (D) presents row based ordering. As updates are incremental, applying a row based ordering followed by a column based or vice-versa organizes a matrix.
The result shown in Figure 2 (C3) is obtained by applying a 2D ordering to the matrix shown in Figure 2 (C1). First, the matrix is ordered by visual similarity across all the columns (Figure 2 (C2)), and the resulting matrix is ordered again using all rows. Finally, if a user wants to tweak the automated ordering results, the manual ordering that we supported helps. Figure 2 (C4) shows some of the rows from the Figure 2 (C3) are manually adjusted.
Along with the positive comments, suggested improvements are listed as follows.
• Why does it only support using the highest bar, not the shortest one? Finding the highest bar needs cognitive efort. [demos 1, 2] • It would be better to use gestures for ordering neighboring bars. [demos 1, 2] • can I brush on the y-axis, and ordering bars based on my selection? [demos 1, 2] • why does it only allow me to select neighboring rows/cols for sorting? The ordering behavior for non-neighboring rows/cols should be consistent with that in bar chart. [demo 3]
interactive technique to order visual elements in a visualization. We implemented the technique on bar charts and matrices and evaluated the technique by gathering an initial expert feedback. The feedback shows interest Figure 3: Direct ordering in a matrix: manually changing the and potential usefulness of the proposed interaction techorder of a column (A), a row (B), ordering all columns (C), all nique in terms of its flexibility and ability to understand rows (D), and all columns followed by all rows (E) by visual user intention. We believe our proposed technique can similarity. help design and build intelligent interactive systems, e.g., designing consistent ways of organizing visual elements in well-aligned 1-D and 2-D layouts, enriching the deThis function ofers more flexibility. sign space of semantic interactions and visualization by demonstration by contributing novel direct manipulation based techniques to order visual elements, and help in 7. Initial Expert Feedback designing techniques to understand user intention by capturing and analyzing user interactions and recomTo evaluate direct ordering, we showed it to a domain mending new possible transformations matching user expert in information visualization and visual analytics. interest. While this work demonstrates the use of interThe feedback indicated the usefulness of the technique active ordering, there are few limitations that need to be in terms of flexibly organizing visual elements. During studied further. the evaluation, similar to those displayed in Figure 2, First, test possible generalizability of direct ordering three demonstrations are explored: 1) ordering bars in to a variety of visualizations. This work can be extended a bar chart, 2) ordering groups and bars with in a group to conduct a study on applying it to diverse visualizain a group bar chart, and 3) ordering rows and columns tions (e.g., parallel coordinates, pie charts, and radial bar in a matrix. Key positive comments from the feedback charts). Similarly, conducting a study to understand its include: application for existing visualization tools and analyzing • It’s really good for me to flexibly order any bars. the diferences in flexibility and ease of use helps further This explicit manual ordering implies user inten- identify trade-ofs of using it. tion and capturing that will be helpful. [demos 1, Second, the initial evaluation of direct ordering came 2] from one domain expert only. To gain an in-depth understanding of this technique’s impact on the interactive • This technique can help with matrix ordering, such user interface, we need to conduct user studies in the as after an automatic matrix sorting, user manu- future, using diverse real-world datasets. One key area ally adjust the order. If such user adjustment can we plan to focus on moving forward is understanding be analyzed and sent back to the automatic order- how to capture user intention based on user interactions. ing algorithms, then we can iteratively order the We could then recommend further possible interactions whole matrix. [demo 3] and transformations that might be of interest to the user. Thus, instead of merely updating a view after the user [10] E. L. Hutchins, J. D. Hollan, D. A. Norman, Diactions, we can empower the user interface with intelli- rect manipulation interfaces, Human–Computer gence to suggest users with possibilities. Interaction 1 (1985) 311–338. [11] A. Sarvghad, B. Saket, A. Endert, N. Weibel, Embedded merge & split: Visual adjustment of data 9. Acknowledgments grouping, IEEE transactions on visualization and computer graphics 25 (2018) 800–809.
This research was supported in part by the NSF Grant [12] B. Kondo, C. Collins, Dimpvis: Exploring timeIIS-2002082 and the Argonne Leadership Computing Fa- varying information visualizations by direct macility, which is a U.S. Department of Energy Ofice of nipulation, IEEE transactions on visualization and Science User Facility operated under contract DE-AC02- computer graphics 20 (2014) 2003–2012. 06CH11357. [13] M. Sun, G. Convertino, Interver: Drilling into categorical-numerical relationships, in: ProceedReferences ings of the International Working Conference on Advanced Visual Interfaces, 2016, pp. 44–51. [1] L. F. Lu, M. L. Huang, T.-H. Huang, A new axes [14] R. Vuillemot, C. Perin, Investigating the direct mare-ordering method in parallel coordinates visual- nipulation of ranking tables for time navigation, in: ization, in: 2012 11th International Conference Proceedings of the 33rd Annual ACM Conference on Machine Learning and Applications, volume 2, on Human Factors in Computing Systems, 2015, pp.
IEEE, 2012, pp. 252–257. 2703–2706. [2] J. Stasko, C. Görg, Z. Liu, Jigsaw: supporting inves- [15] B. Saket, S. Huron, C. Perin, A. Endert, Investigating tigative analysis through interactive visualization, direct manipulation of graphical encodings as a Information visualization 7 (2008) 118–132. method for user interaction, IEEE transactions on [3] M. Sun, J. Zhao, H. Wu, K. Luther, C. North, N. Ra- visualization and computer graphics 26 (2019) 482– makrishnan, The efect of edge bundling and se- 491. riation on sensemaking of biclusters in bipartite [16] A. Endert, P. Fiaux, C. North, Semantic interaction graphs, IEEE transactions on visualization and com- for visual text analytics, in: Proceedings of the puter graphics 25 (2018) 2983–2998. SIGCHI conference on Human factors in computing [4] M. Behrisch, B. Bach, N. Henry Riche, T. Schreck, systems, 2012, pp. 473–482.
J.-D. Fekete, Matrix reordering methods for table [17] H. Siirtola, Interaction with the reorderable matrix, and network visualization, in: Computer Graphics in: IEEE International Conference on Information Forum, volume 35, Wiley Online Library, 2016, pp. Visualization, IEEE, 1999, pp. 272–277. 693–716. [18] J. Bertin, Graphics and graphic information process[5] M. J. McGufin, I. Jurisica, Interaction techniques for ing, Walter de Gruyter, 2011.
selecting and manipulating subgraphs in network [19] C. Perin, P. Dragicevic, J. D. Fekete, Revisiting visualizations, IEEE Transactions on Visualization bertin matrices: New interactions for crafting tabuand Computer Graphics 15 (2009) 937–944. lar visualizations, IEEE Transactions on Visualiza[6] B. Saket, H. Kim, E. T. Brown, A. Endert, Visual- tion and Computer Graphics 20 (2014) 2082–2091. ization by demonstration: An interaction paradigm [20] B. Saket, A. Srinivasan, E. D. Ragan, A. Endert, Evalfor visual data exploration, IEEE transactions on uating interactive graphical encodings for data vivisualization and computer graphics 23 (2016) 331– sualization, IEEE transactions on visualization and 340. computer graphics 24 (2017) 1316–1330. [7] J. C. Roberts, State of the art: Coordinated & mul- [21] D. Norman, Things that make us smart: Defendtiple views in exploratory visualization, in: Inter- ing human attributes in the age of the machine, national Conference on Coordinated and Multiple Diversion Books, 2014.
Views in Exploratory Visualization, IEEE, 2007, pp. [22] Z. Bar-Joseph, D. K. Giford, T. S. Jaakkola, Fast 61–71. optimal leaf ordering for hierarchical clustering, [8] I. Liiv, Seriation and matrix reordering methods: An Bioinformatics 17 (2001) S22–S29. historical overview, Statistical Analysis and Data [23] C. Perin, P. Dragicevic, J.-D. Fekete, Crossets: MaMining: The ASA Data Science Journal 3 (2010) nipulating multiple sliders by crossing, in: Pro70–91. ceedings of the 41st Graphics Interface Conference, [9] B. Shneiderman, 1.1 direct manipulation: a step 2015, p. 233–240.
beyond programming languages, Sparks of innovation in human-computer interaction 17 (1993) 1993.