Large high-resolution displays are becoming an affordable option for the visualisation of data [1]. Large displays have proved to be effective for tasks such as comparative genomics analysis [2], graph topology exploration [3], and sensemaking [4]. Large vertically-mounted (landscape-orientation) highresolution multi-touch displays are particularly effective for collaborative analysis by small teams. However, previous research has often focused on horizontally-mounted tabletop surfaces or vertically-mounted displays with more distant interaction [5]. In this paper, a set of user interactions to support scatterplot matrices analysis on vertically-mounted displays are introduced. These techniques help analysts to efficiently select a scatterplot from scatterplot matrices and explore it collaboratively.
Some physical and virtual interactions with large displays were described in the previous literature. Modalities range from natural interactions like speech, body tracking, gaze, and gestures to the use of secondary control devices like mobile phones, tablets, or Wii controllers [6]. Of these, multi-touch interactions provide a fluid and intuitive interface suitable for up-close interaction in front of the display by small groups.
Although there are studies about collaborative interaction with large displays (e.g. [7], [8]), they usually focus on singleuser interaction [9]. Since typical multi-touch interactions do not support collaboration, more research needs to be done on cooperative gestures, modalities and the dynamics of group work around these devices. Cooperative gestures are known to enhance the sense of teamwork and increase the participation of team members [10].
Screen size and resolution are particularly important for information visualisation of multivariate datasets. Having a large display allows multiple, linked views, such as scatterplot matrices and parallel coordinates [11] to be provided simultaneously. If the screen is not high-resolution, the user experience of near distance interaction decreases significantly. For instance, on screens with less than sixty pixels per inch, the user is not able to read from the screen up-close [12]. Furthermore, users can make more observations with less effort using physical navigation (e.g., walking) rather than virtual [1]. More screen space can be used to either provide a better overview of a dataset or to provide more details of a portion of it. For example, users can see both an entire scatterplot matrix, specific scatterplots, and parallel coordinates plots at the same time. As a result, users may have the opportunity to gain more insight into large datasets.
Previous studies [5] suggest that vertically-mounted displays are more suited to parallel tasks within a group, due to reduced visual distraction and the possibility to share information through physical navigation like turning the head or walking. On tabletop displays, if users are not on the same side of the table, the shared view often needs to be reoriented.
This paper addresses the design gap between standard interaction techniques for large, multi-touch displays and advanced interaction techniques and visual feedback for collaborative scatterplot and scatterplot matrix analysis. Design concepts for such interaction techniques have been implemented as a proof of concept and are presented. The techniques include scatterplot selection from scatterplot matrices, collaborative regression model analysis, and an extension of the Regression Lens [13] to include a floating toolbox. As a proof of concept, the techniques are developed on a large display.
The paper is structured as follows: Section II discusses related work. Several novel interaction designs for collaborative visual analysis of scatterplots on large displays are introduced in Section III. The use case and current implementation of the proposed interaction techniques are described in Section IV. Section VI introduces the concept of perception-based level of visual detail. The paper concludes with a discussion of open problems and future work in Section VII.
At a high level, information visualisation systems consist of two components: visual representation and interaction. Visual representation concerns the mapping from data to display [14]. The interaction starts with a user's intent to perform a task, followed by a user action. The system then reacts and feedback is given to the user [15]. It is essential to consider both visual representation and interaction when designing an application for information visualisation.
Researchers in various fields are increasingly confronted with the challenge of visualising and exploring highdimensional datasets [13], [16]. Keim argues that although many traditional techniques exist to represent data, they are often not scalable to high-dimensional datasets without suitable analytical or interaction design [16].
With the current size and resolution of typical computer displays, it is challenging to represent entire datasets on one screen using techniques like scatterplot matrices or parallel coordinates. The user is often forced to resort to panning and zooming, leading to frustration and longer task completion times. Ruddle et al. [17] conducted an experiment in which participants searched maps on three different displays for densely or sparsely distributed targets. They concluded that since the whole dataset fits on a larger display, sparse targets can be found faster.
Multiple linked views are often used to gain a better understanding of a high-dimensional dataset. Such views are usually connected by techniques such as brushing or combined navigation [18]. Every view occupies space on display. If more space is available, additional views can be shown simultaneously. Allowing the user to access multiple windows increases performance and satisfaction [19]. Isenberg et al. [20] present hybrid-image visualisation for data analysis, where two images are blended to achieve distance-dependent perception. This concept might be especially helpful for collaborative visual analysis tasks on vertically-mounted displays, where users observe data from various distances.
Previous researchers proposed various interaction techniques for large displays and multi-dimensional dataset interaction on multi-touch displays. Ardito et al. [18] proposed a classification of large display interaction having five dimensions: visualisation technology, display setup, interaction modality, application purpose, and location. Khan presented a survey of interaction techniques and devices for large, highresolution displays [6]. The survey categorises modalities of interaction into speech, tracking, gestures, mobile phones, haptic and other technologies such as gaze and facial expression.
Tsandilas et al. presented SketchSliders [21], a tool that provides a mobile sketching interface to create sliders which interact with multi-dimensional datasets on a wall display. In comparison, in this paper, interaction is performed directly on the display rather than using a secondary touch device. Zhai et al. [22] introduced gesture interaction for wall displays based on the distance of the user from the screen. The gestures can be performed in far or near mode. Unlike the techniques described in this paper, the proposed interaction gestures are not directly related to visual analytics tasks. Heilig et al. [23] developed multi-touch scatterplot visualisation on a tabletop display. Sadana and Stasko [24] proposed advanced techniques for scatterplot data selection on smaller touch-based devices, such as tablets and smartphones, whereas this paper focuses on large multi-touch displays.
MultiLens supports various gestures for fluid multi-touch exploration of graphs [25]. The Regression Lens [13] allows the user to interactively explore local areas of interest in scatterplots by showing the best fitting regression models inside the lens. The idea of visualising local regression models is also studied by Matković et al. [26]. Rzeszotarski et al. [27] introduced Kinetica, a tool for exploring multivariate data by physical interactions on multi-touch screens. Kister et al. [25] presented BodyLenses, a promising set of magic lenses for wall displays, which are mostly controlled by body interaction and therefore suitable for interacting with wall displays from a distance.
Interaction Concepts for Collaborative Visual Analysis of Scatterplots on Large Vertically-Mounted High-Resolution Multi-Touch Displays Fig. 2. On the left, a user is drags a Regression Lens with the right hand while adjusting the lens with the left hand. On the right, a user drags a scatterplot with the right hand while panning through the scatterplot matrix with the left hand.
In comparison to this work, the aforementioned studies either focus on a different type of interaction and medium or are not designed for collaborative visual analytics tasks.
Large displays are well-suited to collaboration [28], [29]. Jakobsen and Hornbaek [5] conducted an exploratory study to understand group work with high-resolution multi-touch wall displays. The study suggests that using this kind of display helps users to work more efficiently as a group and fluidly change between parallel and joint work. A large display benefits group working on a shared task, since users can operate on one common physical medium and share information on it.
Morris et al. [10] formalised the concept of cooperative gestures as a set of gestures performed by multiple users and interpreted as a single task by the system. Liu et al. developed CoReach [9], a set of gestures for collaboration between two users over large multi-touch displays. Comparing the use of a large vertically-mounted display against two ordinary desktop displays, Prouzeau et al. [30] concluded that groups obtain better results and communicate better on large, verticallymounted displays.
An experiment by Pedersen and Hornbaek [31] showed that users prefer horizontal surfaces over vertically-mounted displays, but this result was limited to simple single-user tasks and not collaborative tasks with different dynamics. Vertically-mounted displays allow users to obtain an overview of their data by stepping back from the display and make it possible to interact from afar as well as up close. Badam et al. [32] proposed a system for collaborative analysis on large displays by controlling individual lenses through explicit midair gestures.
Although these studies are not directly related to collaborative scatterplot analysis on large multi-touch displays, they do provide valuable insights into the design process of such systems.
Current standard multi-touch interaction techniques are not designed for collaboration on vertically-mounted highresolution displays [9]. Here, both single-user and collaborative interactions are proposed for the analysis of scat- terplots and scatterplot matrices on such devices. Some of the interaction techniques are based on the concept of the Regression Lens [13], which supports real-time regression analysis of subsets of a scatterplot through lens selection and manipulation. With Regression Lens, a user can select a local area in a scatterplot and observe the regression model of selected points [13]. Shao et al. proposed operations to adjust and manipulate the regression model shown in the Regression Lens, such as changing the degree of the regression model or inverting its axes. Figure 1 illustrates some of the suggested collaborative gestures on an 84-inch 4K/ULTRA-HD@60HZ multi-touch LCD monitor produced by Eyevis [33]. The user on the left finds interesting scatterplots and passes them to the user on the right. The user on the right analyses the plots using the Regression Lens [13]. In the rest of this section, four interaction designs for both collaborative and single scatterplot analysis are introduced. Later in Section IV, an implementation of these techniques is demonstrated.
Magic lens techniques like DragMagics [34] and BodyLens [35] are used to explore local regions in a visualisation. An extended version of the basic lens concept provides for more fluid interaction with large multi-touch displays. For instance, as shown in Figure 2, after a region of interest has been selected in a scatterplot using the dominant hand (here the right hand), a toolbox appears next to the other side of the lens (near the non-dominant hand), where the user can use sliders and touch buttons to adjust the lens. For example, the user can change the degree of the regression model. The lens can be dragged with one hand, while being adjusted with the second hand, thus potentially speeding up performance.
A scatterplot matrix consists of pairwise scatterplots arranged in a matrix, with dimensions typically labelled in the diagonal cells. Since the number of dimensions is usually high, panning and zooming within the scatterplot matrix is almost inevitable. With common multi-touch interactions, the scatterplot or dimension label is dragged to the corner of Interaction Concepts for Collaborative Visual Analysis of Scatterplots on Large Vertically-Mounted High-Resolution Multi-Touch Displays Fig. 4. A user selects a scatterplot of interest from a scatterplot matrix by touching and holding the left hand on the scatterplot. Swiping with the right hand then passes the selected scatterplot to the right hand side of the display for more detailed analysis.
the scatterplot matrix for panning. It is not feasible to zoom into or out of a scatterplot matrix while dragging another object. Based on two-handed interaction on tablets [36], a two-handed technique is proposed whereby the dominant hand is responsible for dragging items, while the non-dominant hand performs common operations. As shown on the left side of Figure 2, the user drags a scatterplot around to reorder the plots in the scatterplot matrix. Panning is performed by the non-dominant hand. With this two-handed technique, the interactions needed to reorder scatterplots in a scatterplot matrix can be reduced.
On large vertically-mounted collaborative displays, it is not always desirable to move from one side of the screen to the other to perform a task. Instead, collaborative gestures can be used to pass objects. Based on the ideas of Liu et al. [9], collaborative gestures on scatterplots are proposed. In the right-hand side of Figure 3, the user on the left is analyses a scatterplot, while the user on the right selects another scatterplot of interest. By holding the background of the scatterplot matrix with one hand, and swiping with the other hand, the scatterplot is passed over to the partner. The partner can then decide whether or not to load the scatterplot for comparison. This technique can also be used for other tasks. For example, in Figure 4, the user selects a scatterplot of interest from a scatterplot matrix by touching and holding it with one hand (here, the left hand) and swipes the other hand in the direction of the analysis panel to load that scatterplot for more detailed analysis.
In collaborative analysis, visual feedback plays an essential role. When two analysts work on a vertically-mounted display without proper visual feedback, they need to communicate more and turn their heads more often. A collaborative lens can help ameliorate this issue. As illustrated on the left side
Proof-of-concept interaction techniques for single-user and collaborative analysis of scatterplots and scatterplot matrices have been implemented on a vertically-mounted Eyevis 84inch multi-touch display with a resolution of 3840 × 2160 pixels and a frame rate of 60 Hz. Figure 1 demonstrates a typical setup of the implemented application with two users working on the screen.
The prototype application is written in Java, using JavaFX for the user interface and the TUIO [37] and the TUIOFX library [38] for multi-touch interaction. To enable multiple users to work on the same screen with different widgets and user interface elements at the same time, a concept called focusArea from the TUIOFX library is used [39]. The application follows the widely-used Model-View-Controller (MVC) architecture.
The use case for the prototype application is to improve interaction with the Regression Lens on multi-touch screens. The developed interaction techniques were tested with the well-known car dataset from the UCI Machine Learning Repository [40].
For the interaction technique shown in Figure 1, user A (on the left) and user B (on the right) select two different plots from the shared central area containing the scatterplot matrix. For this technique, the user holds and touches a scatterplot with one hand and swipes to the right or left with the other hand to maximise it. This technique is elaborated in detail in Section III-C. After that, users A and B select an area in the scatterplot separately and toggle the Collaborative Lens option in the Floating Toolbox. As described in Section III-D, each Interaction Concepts for Collaborative Visual Analysis of Scatterplots on Large Vertically-Mounted High-Resolution Multi-Touch Displays user is now able to observe the regression model of the other user in their regression lens. Figure 1 shows two users working side by side on a large vertically-mounted multi-touch display, after creating two separate Regression Lenses and toggling to the Double Lens option. The exact state of the screen is shown in Figure 6. A single Regression Lens with a floating toolbox is visible in Figure 5.
Users of large vertically-mounted high-resolution displays may take up positions at varying distances from the display, and hence may perceive more or less detail in the display. At greater distances from a large high-resolution display, less detail is perceived. Here, perceived pixel density (PPD) is defined as the number of pixels mapped to a single cell on the retina of the user's eye. PPD increases quadratically as distance to the screen increases. The human perceptual system tends to average out too large PPD w.r.t. colour, brightness, and contrast [41], for example a red pixel and a green pixel is perceived as brown.
The perceptual effect of averaging is well known, for instance in the perception of secondary colours as a mixture of two primary colours or in the phenomena of metamerism. More related effects include simultaneous contrast [42], afterimages [43], and the Chubb effect [44]. Without delving too deeply into perception psychology, note that a sophisticated theory for averaging effects are already available and well described. For the purpose of this discussion with respect to large high-resolution displays, it is sufficient to state that the effect of averaging a set of pixels is already exploited in practice by techniques such as image mosaics [41] and halftone techniques [45], as illustrated in Figure 7.
Since PPD and related averaging effects are a function of distance from the display, screen distance can be seen as an interactive parameter which can be exploited for visual data analysis. Three techniques are proposed to apply a perceptionbased level of detail to scatterplots on large vertically-mounted high-resolution displays.
Firstly, the concept of superpixels is similar to image mosaics. A superpixel consists of a set of pixels in a small rectangular area of the screen, for example a regular grid of say 50×50 pixels. The average colour, brightness, and contrast properties of superpixels can be used to visualise data for users farther from the screen. At the same time, the individual colouring of pixels comprising a superpixel can be used to visualise more detailed information for users who are closer to the screen.
Secondly, the concept of a Screen Progressive Visual Glyph (SPVG) utilises the colour, brightness, and contrast values of a glyph to encode different secondary information for closer users. In Figure 8, the scatterplot on the left visually encodes two different classes (brown and cyan) in the data. This is easily perceivable by a distant user. On the right, a user who is closer can make out an additional level of detail: the dots of the scatterplot in fact contain an additional histogram representing the distribution of the related class in the data. In this case, the circles representing the mapped data points are SPVGs. The difference between SPVGs and superpixels is that SPVGs encode different visual details of the same data at different distances. In this way, they could be understood as a data filter concept as well. SPVGs can be placed on the screen on demand and are not restricted to a regular grid, providing greater flexibility.
Thirdly, variational textures are related to halftone techniques. Structural variations of an underlying texture can be used to visually encode fine data details for users who are very close to the screen, while these details will immediately disappear when the user goes further away.
These proposed approaches for level of visual detail align well with Shneiderman's mantra for information visualisation [46]: "Overview first, zoom and filter, details on demand". In this case, distance from the screen is an additional degree of Interaction Concepts for Collaborative Visual Analysis of Scatterplots on Large Vertically-Mounted High-Resolution Multi-Touch Displays Fig. 7. On the left, a multi-image mosaic of the Mona Lisa [41]. On the right, an example of halftone dot sampling [45]. freedom, controlled by each user individually as they move closer to or further away from the display. The approaches are discussed as a concept and not implemented yet.
The concepts described in this paper are first designs of appropriate touch interaction for the visual interactive analysis of scatterplot data on large vertically-mounted high-resolution multi-touch displays. The interactions support small-group collaborative analysis, by exchanging patterns or settings from one user's view to the others. The interaction design is currently based on user selections, but is generalisable to other basic techniques. The interaction techniques have been implemented as a proof of concept. They still need to be evaluated with real users and real tasks as part of future work. Mapping out the design space for this combination of visualisation and display device may well yield further interesting interaction designs.
The idea of exploiting perception-based level of detail for the visualisation of scatterplots on large displays is new. Detailed information can be rendered inside the marks of the plot, becoming perceivable once users are closer to the screen. Again, this is a proof of concept and requires further development and evaluation.
While large high-resolution displays can improve the exploration of large scatterplot spaces, further data analysis support is needed to scale up with the number of data points and dimensions. Traditional techniques like cluster analysis and aggregation can help with scalability. Another relevant line of improvement is to adjust the view to the user's need and situation. In [47], the authors propose using eye tracking to infer user interest and using this information to recommend additional relevant but previously unseen views for exploration. While that work was developed as a desktop application, it might be interesting to incorporate eye-tracking support to recommend views for small collaborative team work on a large display. Moreover, adding group activity recognition and therefore pro-active interaction, can support collaboration by preventing information overload [48].
This paper presented challenges and solutions for collaborative and single-task multi-touch interaction on large verticallymounted high-resolution displays. The techniques presented are well-suited for collaborative analysis tasks with scatterplots and scatterplot matrices. They are potentially generalisable for other data exploration and visual analytics practices but require further implementation and evaluation. Also, perception-based visualisation of scatterplots is introduced as a possible direction for further research.