Representable AI: Towards a Unified View of Core Dimensions for a Visual Framework

Abstract

Data visualization made of multiple visualization techniques, e.g., dashboards and small multiples, is taking the scene as data, AI algorithms and their analysis are becoming more complex and articulated. However, still too little is said about what are the core dimensions of these interactions that may contribute to characterize visualization techniques orchestration in scenarios where humans and AI work together and communicate through visual languages, and what is the diferential in complexity with respect to single charts interaction. Depending on such dimensions, their level, and their combination, interaction may require a cognitively growing efort. The present study aims at giving a unified view of complex visual frameworks in order to identify the invariants of visualization techniques characterization, and proposes a group of necessary and suficient dimensions emerging when visualization techniques are the focus of the design and may be the focus of interaction between humans and AI. The paper identifies and discusses these dimensions, starting from the literature, and giving a characterization of each of them in terms of constituent levels. The framework may be applied to analysis of a range of data visualization tools and approaches, towards their concrete application to a distributed visual cognition framework where humans-AI interactions will take place.

Keywords

• eol>visualization techniques
• multi-chart interaction
• conceptual framework for distributed cognition
• visual AI

1. Introduction and Background

A visualization technique refers to a graphical view of data in the form of charts (static like bar charts and histograms, or interactive like treemaps) allowing users to analyse and visually explore a dataset, the input and the output of an algorithm or even its intermediate steps. Whilst the data might work with multiple chart types, it is often recommended to select the one that ensures a message clear and accurate according to users needs [ 1 ][ 2 ][ 3 ]. And yet, multi-charts systems (that combine two or more views) have been proved useful to show diversity and complementary of information visualization techniques [ 4 ]. Indeed, most recent and widely adopted tools use multi-charts as a means to design online dashboards [ 5, 6 ], animated views [ 7 ] and graph exploration [ 8 ]. Nonetheless, envisioning how visualization techniques may act as the primary communication means between humans and AI is not a simple task. The dangling question here is: how to systematically describe the dependencies when combining multiple charts to support this kind of distributed cognition and communication?

As we shall see, describing visualization techniques and the orchestration of multi-charts is a daunting task. To the best of our knowledge, the literature have investigated diverse aspects that contribute to the orchestration of multi-chart systems (i.e. [ 9 ]) but it still misses a comprehensive framework showing all the dependencies. In this paper we try to combine all previous works found in the literature and then we propose a conceptual framework aimed at support the discussion in the community about the need of methods for describing visualization techniques and their orchestration.

In the next sections we identify conceptual entities for describing the techniques for information visualization. Then we identify four invariant dimensions that might help to explore the dependencies between multi-charts systems and communication through them.

2. Towards a conceptual framework

The conceptual framework presented in this paper moves from a first conceptualization of the interaction of a user with a visualization technique. The entities and relationships involved in the interaction are shown in Figure 1. The basement level of the figure describes the entities in a single chart. A user expresses a goal which might be accomplished by a task through a set of interactions that contribute to the execution of a Viz technique (short name for visualization technique). The user interaction might also trigger special operations for querying and processing raw data so that the resulting processed data maps the visualization technique. The upper layer in Figure 1 shows the orchestration level of multiple visualization techniques. It is important to notice that all entities continuously co-evolve. Next section, we will focus on that specific layer of visualization technique/s orchestration.

3. The Dimensions of the Visual Frameworks

Four invariants that play a major role for describing the orchestration of multiple visualization techniques we identified (based on the literature and on our unifying efort). These dimensions may afect data visualization tools design, their functionalities and limitations, and the rationale behind many multi-charts problem-solving approaches.

3.1. Composition (Space) Levels

Most works in the literature refers to the disposition of charts along the Space dimension ([ 10 ], [ 11 ], [ 12 ]).

3.1.1. No composition

Whenever only one visualization technique is exploited, there is no composition at all. Another example, with more than one visualization technique, is in the “overview + details” case [ 13 ], like in the Gapminder tool, where no composition facility is provided (even if two – or more – visualizations are present in the screen, they refer to the same visualization technique but at different zoom level). The composition can apply according to the other three levels: juxtaposition of visualization techniques, superimposition and nesting. Each subsequent level may contain the previous one.

3.1.2. Juxtaposition

It occurs when two visualization techniques are placed side by side, in whatever orientation (high-to-low or left-to-right). In a multi-charts view, there could be juxtaposition of one or more of these juxtaposed visualization techniques. An example of juxtaposed visualization techniques are simple dashboards where the layout is composed of side-by-side and top-tobottom juxtaposed visualization techniques.

3.1.3. Superimposition

It occurs when two visualization techniques are (partially) overlapping. In geo-maps visualization, diferent layers may be superimposed (e.g., the physical layer and the toponymy layer). 3.1.4. Nesting Nesting two visualization techniques means to include one visualization technique inside the layout of a primary visualization technique, in order to extend the range of information included in one visualization technique with a second visualization technique. Most of the time, nesting includes superposition of elements of two or more visualization techniques. For example, graph links may be nested inside a treemap structure, as in [ 14 ]. Nesting could also apply to the arrangements of elements in a hierarchical layout (see for example [ 12 ]). When multiple charts are in place, nested visualization techniques may also be juxtaposed in diferent guise.

3.2. Synchronization (Time) Levels

During interaction, visualization techniques may be positioned along the time dimension too. Depending on the kind of visualization technique and on the layout structure, the synchronization of views may assume levels of growing complexity.

3.2.1. No synchronization

The basic level is when no synchronization of visualization techniques is present. Typically, this situation arises when only one visualization technique is present in the layout.

3.2.2. Sequential synchronization

Sequential synchronization arises when visualization techniques are visible one after the other (for example, in a multi-page environment, like the one provided in Tableau Stories or dashboardto-dashboard drilldowns, like the ones provided in the Kibana environment [ 5 ].

3.2.3. Parallel Synchronization

In parallel synchronization, visualization techniques are visible all at a time. This is the typical configuration of small multiples visualizations or of dashboards layout (like, for example, in Power BI, Tableau or Kibana environments).

3.2.4. Interleaved Synchronization

When visualization techniques are all present and evolving in time, their synchronization is “interleaved”. This kind of synchronization may be experienced in parallel, and parallel interleaved visualization techniques may be experienced in sequence, depending on the layout structure. An example of the first kind is the Gapminder tool [ 7 ], which provides animated time series where points at diferent time are all present in the same layout.

3.3. Coordination (Tool) Levels

In case a composition exists, it must be coordinated , i.e., visualization techniques can change congruently. For this dimension we adopt a view of coordination as defined in groupware interaction [ 15 ][ 16 ][ 17 ][ 18 ]. Coordination can be done according to one of the level of coordination shown in the next paragraphs.

3.3.1. No coordination

When there is no coordination, the layout may either contain only one visualization technique or the composition is arbitrary, i.e., no congruent changes arise. Many tools with no coordination are available (see for example tools for single charts design or “disconnected” visualization techniques in dashboard design, i.e., where a change in one visualization techniques does not imply a change into another visualization technique).

3.3.2. Manual coordination

In manual coordination, two or more visualization techniques are related by a commonly changing dataset, a commonly changing visual element (e.g., categorical scale) or a common functional element (e.g., the same filter applied to two visualization techniques). An example of manual coordination is, for example, Tableau, where filters can be set to select elements in more than one visualization technique.

3.3.3. Coordination on demand

In coordination on demand, semi automatic functionalities may be activated on diferent visualization techniques. For example, in the Knime a visual tool for creating data analytics pipelines [19], components may relate and be updated congruently on demand (as soon as the pipeline is executed) or may not, depending on the users’ needs and configurations in the system.

3.3.4. Automatic Coordination

Automatic coordination arises when the user control on the coordination of visualization techniques is only partial or absent. An example of this is Gapminder, which provides a template layout where visualization techniques are automatically composed and animated through a timeline, and only the data can be arbitrarily selected by the user (not the visualization and animation mode).

3.4. Integration (Data) Levels

Although not in the foreground with respect to the interactions among visualization techniques, the Integration dimension (data level) included in visualization techniques is present in our framework. Ranging from “no integration at all” to “combination/merging/ parsing” of multiple sources, this dimension was still under-explored and under-defined. This is due to the various complexity in data integration for which many scholars have struggled to find a solution or at least an agreement. Many attempts made to capture, at least descriptively, this complexity are available in the literature about data integration 1. We are not exploring further this dimension, leaving the levels undefined in terms of what is the complexity involved in each of them, but only mentioning them for completeness and inclusiveness. We may briefly touch here only the fact that integration of data may become part and parcel of the growing complexity of visualization techniques design, interaction and evaluation.

1See, for example, the table schematized in http://www.cs.toronto.edu/~yuana/DI-Complexity.html for an overview of data integration complexity. Angela Locoro et al. CEUR Workshop Proceedings [19] Knime, 2022. URL: https://www.knime.com, (accessed January 2022).

80

References

[1] E. A. Bier , S. K. Card , J. W. Bodnar , Entity-based collaboration tools for intelligence analysis , in: 2008 IEEE Symposium on Visual Analytics Science and Technology , 2008 , pp. 99 - 106 . doi: 10 .1109/VAST. 2008 . 4677362 .

[2] C. He , L. Micallef , L. He , G. Peddinti, T. Aittokallio , G. Jacucci, Characterizing the quality of insight by interactions: A case study , IEEE Trans. Vis. Comput. Graph . 27 ( 2021 ) 3410 - 3424 . doi: 10 .1109/TVCG. 2020 . 2977634 .

[3] C. Ziemkiewicz , R. Kosara , Implied dynamics in information visualization , in: Proceedings of the International Conference on Advanced Visual Interfaces , AVI '10 , Association for Computing Machinery, New York, NY, USA, 2010 , p. 215 - 222 . doi: 10 .1145/ 1842993.1843031.

[4] M. Q. Wang Baldonado , A. Woodruf , A. Kuchinsky , Guidelines for using multiple views in information visualization , in: Proceedings of the Working Conference on Advanced Visual Interfaces , AVI '00 , Association for Computing Machinery, New York, NY, USA, 2000 , p. 110 - 119 . doi: 10 .1145/345513.345271.

[5] Kibana , 2022 . URL: https://www.elastic.co/kibana, (accessed January 2022 ).

[6] Tableau , 2022 . URL: https://www.public.tableau.com/s, (accessed January 2022 ).

[7] Gapminder , 2022 . URL: https://www.gapminder.org/tools, (accessed January 2022 ).

[8] A. Menin , R. A. Cava , C. M. D. S. Freitas , O. Corby , M. Winckler , Towards a visual approach for representing analytical provenance in exploration processes , in: 25th International Conference Information Visualisation, IV 2021 , Sydney, Australia, July 5-9 , 2021 , IEEE, 2021 , pp. 21 - 28 . doi: 10 .1109/IV53921. 2021 . 00014 .

[9] D. A. Keim , Information visualization and visual data mining , IEEE Trans. Vis. Comput. Graph . 8 ( 2002 ) 1 - 8 . doi: 10 .1109/2945.981847.

[10] W. Javed , N. Elmqvist , Exploring the design space of composite visualization, in: 2012 ieee pacific visualization symposium , IEEE, 2012 , pp. 1 - 8 .

[11] M. Gleicher , D. Albers , R. Walker , I. Jusufi, C. D. Hansen , J. C. Roberts , Visual comparison for information visualization , Information Visualization 10 ( 2011 ) 289 - 309 . doi: 10 .1177/ 1473871611416549.

[12] X. Chen , W. Zeng , Y. Lin , H. M. Al-Maneea , J. Roberts , R. Chang , Composition and configuration patterns in multiple-view visualizations , IEEE Trans. Vis. Comput. Graph . 27 ( 2021 ) 1514 - 1524 . doi: 10 .1109/TVCG. 2020 . 3030338 .

[13] B. Shneiderman , The eyes have it: A task by data type taxonomy for information visualizations , in: Proceedings of the 1996 IEEE Symposium on Visual Languages, VL '96 , IEEE Computer Society, USA, 1996 , p. 336 .

[14] N. Henry , J.-D. Fekete , M. J. McGufin , Nodetrix: a hybrid visualization of social networks , IEEE Transactions on Visualization and Computer Graphics 13 ( 2007 ) 1302 - 1309 . doi: 10 .1109/TVCG. 2007 . 70582 .

[15] Y. Laurillau , L. Nigay, Clover architecture for groupware , in: Proceedings of the 2002 ACM Conference on Computer Supported Cooperative Work , CSCW '02, Association for Computing Machinery, New York, NY, USA, 2002 , p. 236 - 245 . doi: 10 .1145/587078.587112.

[16] H. Fuks , A. Raposo , M. A. Gerosa , M. Pimental , C. J. P. Lucena , The 3c collaboration model , Encyclopedia of E-Collaboration ( 2008 ) 637 - 644 . doi: 10 .4018/978-1- 59904 - 000-4. ch097 .

[17] C. A. Ellis , S. J. Gibbs , G. Rein, Groupware: Some issues and experiences , Commun. ACM 34 ( 1991 ) 39 - 58 . doi: 10 .1145/99977.99987.

[18] U. M. Borghof , J. H. Schlichter , Computer-Supported Cooperative Work: Introduction to Distributed Applications, Springer-Verlag, Berlin, Heidelberg, 2000 .