Cross-Virtuality Visualization, Interaction and Collaboration

Abstract

Figure 1: Cross-virtuality analytics for interactive visualization of big data by the seamless integration and free navigation between 2D visualization (left), augmented reality (center) and virtual reality (right).

-

Copyright © 2020 for this paper by its authors. Use permi ed under Creative Commons License A ribution 4.0 International (CC BY 4.0). In X-Pro, we investigate novel user-centric methods and techniques for cross-virtuality analytics. Cross-virtuality analytics in our sense aims to enable a seamless integration and transition between conventional 2D visualization, augmented reality and virtual reality in order to provide users with optimal visual and algorithmic support with maximum cognitive and perceptual suitability, depending on their current tasks and needs in the analysis process.

We thus focus on methods and techniques mainly for production data, which promise a new quality of visual analytics along the reality-virtuality-continuum in order to facilitate a completely di erent level of visual and spatial perception as compared to the state of the art. Aside the conception and development of novel visualization techniques, we also concentrate on a close collaboration and interaction of users within this continuum. Regarding analyzing, exploring and modeling of data, evaluations of trends, the detection of patterns and outliers as well as correlations in the data will be of utmost importance. We target to investigate concepts of novel visual metaphors, novel interaction concepts, their mathematical foundations, and evaluate them in terms of their technical feasibility, their cognitive, perceptional and ergonomic usability.

We believe that cross-virtuality analytics has the potential to fundamentally improve data-driven planning, control, optimization and quality assurance.

CCS CONCEPTS

• Human-centered computing Virtual reality; Collaborative interaction; Information visualization; • Computing methodologies Mixed / augmented reality. cross virtuality, interaction, collaboration, mixed reality Reference Format: Andreas Riegler, Christoph Anthes, Hans-Christian Jetter, Christoph Heinzl, Clemens Holzmann, Jodlbauer Herbert, Manuel Brunner, Stefan Auer, Judith Friedl, Bernhard Fröhler, Christina Leitner, Fabian Pointecker, Daniel Schwajda, and Shailesh Tripathi. 2020. Cross-Virtuality Visualization, Interaction and Collaboration. In Cross-Reality (XR) Interaction, ACM ISS 2020 (International Workshop on XR Interaction 2020). , 1

INTRODUCTION

An important step towards more transparent "big data" analysis methods and a greater general acceptance is found in interactive visualization of data and respective tools, especially in the context of promising new technologies in the eld of augmented reality (AR) and virtual reality (VR). Instead of interpreting numerical or symbolic outputs, interactive visualization allows the use of human input for visual information processing and pattern recognition in order to discover and understand trends, patterns and correlations in large amounts of data. For example, such big data applications occur commonly in the production context, through interaction with computer generated visualizations [ 3 ]. Interactive visualization can be thought of as an "assistance system" to enhance human cognition, which, through external visual representation, greatly relieves the cognitive load during analysis and model building and makes knowledge generation and decision support possible. In addition, interactive visualization can also support the necessary preparation or preprocessing of data (e.g. data cleansing, data preparation, data transformation), e.g. by excluding visible errors or gaps in the data from the analysis [ 3 ]. Due to the abundance of data available today and modern methods of data science, data mining and machine learning, the classic visualization of information has developed into visual analytics [ 11 ]. Traditional visual analytics combines automated analysis techniques with interactive visualisation, mainly using conventional workstation based settings, for an e ective understanding, reasoning and decision making on the basis of very large and complex datasets [ 10 ].

We de ne "Cross-Virtuality Analytics" as new possibilities for interactive visual data preparation, modeling and analysis based on uent transitions between novel visualization and interaction techniques across the entire spectrum of the reality-virtuality continuum. The focus on "virtuality" instead of "reality" in our de nition of this term is based on the premise that early phases of the data analytics process are best supported with technologies leaning to the virtuality-side of the reality-virtuality continuum. Nonetheless, smooth transition towards the reality-side of the continuum can play a critical role to integrate data analytics into real-world spatial settings and social environments such as physical movement in a familiar work environment or face-to-face collaboration with co-located team members. 2

IMMERSIVE VISUAL ANALYTICS IN MIXED REALITY (MR) In order for immersive analytics [ 4 ] to be used to its full extent, traditional 2D desktop environments and virtual environments must be seamlessly combined and integrated. Technologies and media used in the eld of immersive analytics on the reality-virtuality (RV) continuum [ 13 ] o er individual advantages and disadvantages: • 2D visualization on touchscreens: allow for familiar visual representations, directness of touch interaction through haptic feedback, very high readability and very high resolution. Furthermore, 2D visualizations provide the opportunity of natural collaboration, especially on tables, walls or multiple tablets for collaborative visualization using tangible interfaces. However, 2D visualizations lack stereoscopic representation, which leads to a low level of immersion. [ 5, 9 ] • Augmented reality: allows for the retention of the natural environment (therefore still natural collaboration), stereoscopic presentation, natural spatial navigation via body movement, spatial organization of content in the physical environment, object-referenced data representation and integration or representation of remote users. However, AR still provides a limited eld of view and poor contrast. [ 2 ] • Virtual reality: allows for stereoscopic presentation, natural spatial navigation via body movement and head tracking, spatial organization of content in any size of virtual environment, signi cantly larger eld of view and integration or representation of remote users. However, users are visually and socially isolated from the real environment and interaction with content does not occur via familiar touch interaction including haptic feedback. [ 1, 6, 15 ]

So far the question remained unanswered as to how these advantages can be combined for analyses along the reality-virtuality continuum and how individual disadvantages can be overcome through intelligent combinations and transitions between the various technologies or media. Therefore, we propose to research a set of visualization, interaction and collaboration techniques as well as the necessary data preparation, analysis and modeling steps needed to productively use them for cross virtuality analytics. 3

VISUALIZATION, INTERACTION AND

COLLABORATION

The possibility of targeted movement within the RV continuum creates a unique new tool or visual-spatial medium for the exploration and analysis of data (see Figure 1). For example, at the beginning of the analysis process, a collaborative visualization of data in 2D for all team members on a non-stereoscopic interactive wall with touch and pen input is possible. Head-mounted displays (HMDs) can also be used to freely place stereoscopic 3D representations of the data outside the wall display in the room (AR), e.g. to enable collaborative analysis of additional dimensions by some team members by means of physical navigation with their own body. Using AR, the physically present team members remain visible and further data, visual metaphors or even additional team members (e.g. experts) connected via internet may be integrated in the room using virtual representations (e.g. for tele-cooperation). Should the analysis of the data require a larger eld of views and the fading out of the real physical environment, this can be realized at any time by using VR HMDs. Spatial AR, i.e. the use of projections on real objects, can be used to transfer annotations created by VR users to real objects [ 16–18 ].

For newly researched methods and procedures of cross-virtuality analytics it is essential, to ensure a spatial and semantic orientation of the users during these transitions between reality, augmented reality, augmented virtuality or virtual reality via perceptual and cognitive "anchor points". For example, the preservation of spatial landmarks across di erent media is desirable, so that physical objects such as screens are not only visible in reality and AR, but also appear in VR through correctly placed virtual representations. Furthermore, the transition has to be uent and keep the users focused on their tasks [ 14 ]. To ensure such a uent work ow across the RV continuum, possible visual transition techniques need to be conceptualized and evaluated. The impact of collaborators during the transition process has to be considered as well. Other factors for such visual and spatial coherence are also consistency in shadowing, lighting, and masking. This also applies to the presence of real or virtual team members in the environment. In terms of learnability and user-centricity, it is still crucial to ensure consistent behavior and interaction techniques in the various stages on the RV continuum. This is the only way to reduce the learning e ort and cognitive load and to show a real added value compared to traditional visual analytics or immersive analytics in experimental user studies. Grasset et al. [ 7 ] have examined the topic of collaboration in the transition between augmented and virtual reality and identied possibilities for transition. Di erent aspects (e.g. user context, information transfer between users) of collaboration between users who are in di erent parts of the RV continuum must be considered.

Visual Analytics and cross-virtuality analytics is also highly attractive for gaining deep, previously unimaginable insights into "rich" volumetric data (spatial data + derived quantitative data) [ 8 ]: Current challenges in this area focus on integrated visual analysis, quantitative data visualization, visual debugging and visual steering. While visual Analytics is increasingly adopted in this eld, crossvirtuality analytics yet plays a minor role in this area, but contains a tremendous potential for future applications. 4

DATA PREPARATION, ANALYSIS AND

MODELING

For cross-virtuality analytics to be successful, however, not only the exploration of suitable visualization and interaction techniques is necessary, but especially new methods for data analysis or data modeling using new algorithmic, statistical, and learning methods of data analysis, data mining or arti cial intelligence must be investigated. As possible use cases for the new methods on the level of visual interaction and data analysis, cross-virtuality analytics can be researched for (1) complex graphs [ 12, 19 ], e.g. in the production and logistics domain and (2) volumetric data from non-destructive testing of production data. The interactive provision and optimization of such models and their results with the necessary precision and reliability in real-time without destructive delays is a particular challenge. Therefore, our approach is not only concerned with basic research in the eld of visualization and human-computer interaction, but also in the eld of information-theoretical and statistical analysis of data and the necessary algorithms for data preparation and classi cation, e.g. by means of data mining or modern machine learning methods. Especially in the production domain, data-driven knowledge generation and decision support can be signi cantly improved for users through the close temporal, spatial and cognitive integration of algorithmic analysis and data or model visualization. 5

CONCLUSION

To sum up, cross-virtuality analytics has the potential to a completely new quality of visual and spatial perception of data as well as a close cooperation between users and algorithms for data modeling and analysis tasks. Across the reality-virtuality-continuum, we aim to enable the seamless integration and transitions between 2D visualization, augmented reality and virtual reality in order to facilitate users with novel interaction and collaboration methods. In this workshop, we would like to share our goals, our approach and our experimental process towards achieving a seamless solution for cross-virtuality applications in our X-Pro project.

ACKNOWLEDGMENTS

This project is nanced by research subsidies granted by the government of Upper Austria.

References

[1] Christoph Anthes , Rubén Jesús García-Hernández, Markus Wiedemann , and Dieter Kranzlmüller . 2016 . State of the art of virtual reality technology . In 2016 IEEE Aerospace Conference. IEEE , 1 - 19 . https://doi.org/10.1109/AERO. 2016 . 7500674

[2] Mark Billinghurst , Adrian Clark, and Gun Lee . 2015 . A survey of augmented reality . Foundations and Trends® in Human-Computer Interaction 8 , 2 - 3 ( 2015 ), 73 - 272 . https://doi.org/10.1561/1100000049

[3] Stuart Card . 2009 . Information visualization. In Human-computer interaction: Design issues, solutions, and applications, Andrew Sears and Julie A . Jacko (Eds.). Taylor & Francis , 181 - 216 .

[4] Tom Chandler , Maxime Cordeil, Tobias Czauderna, Tim Dwyer, Jaroslaw Glowacki, Cagatay Goncu, Matthias Klapperstueck, Karsten Klein, Kim Marriott, Falk Schreiber, and Elliot Wilson. 2015 . Immersive Analytics . In 2015 Big Data Visual Analytics (BDVA). 1-8 . https://doi.org/10.1109/BDVA. 2015 .7314296

[5] Niklas Elmqvist , Andrew Vande Moere, Hans-Christian Jetter , Daniel Cernea, Harald Reiterer, and TJ Jankun-Kelly . 2011 . Fluid interaction for information visualization . Information Visualization 10 , 4 ( 2011 ), 327 - 340 . https://doi.org/10. 1177/1473871611413180

[6] Rubén Jesús García-Hernández, Christoph Anthes, Markus Wiedemann, and Dieter Kranzlmüller . 2016 . Perspectives for using virtual reality to extend visual data mining in information visualization . In 2016 IEEE Aerospace Conference . 1 - 11 . https://doi.org/10.1109/AERO. 2016 .7500608

[7] Raphael Grasset , Julian Looser, and Mark Billinghurst . 2006 . Transitional interface: concept, issues and framework . In 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality. IEEE , 231 - 232 . https://doi.org/10.1109/ISMAR. 2006 .297819

[8] Christoph Heinzl and Stefan Stappen . 2017 . STAR: Visual Computing in Materials Science . Computer Graphics Forum 36 , 3 ( 2017 ), 647 - 666 . https://doi.org/10.1111/ cgf.13214

[9] Hans-Christian Jetter , Jens Gerken, Michael Zöllner, Harald Reiterer, and Natasa Milic-Frayling. 2011 . Materializing the query with facet-streams: a hybrid surface for collaborative search on tabletops . In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . 3013 - 3022 . https://doi.org/10.1145/ 1978942.1979390

[10] Daniel A Keim , Jörn Kohlhammer, Geo rey Ellis, and Florian Mansmann (Eds.). 2010 . Mastering the information age: solving problems with visual analytics . Goslar: Eurographics Association. https://diglib.eg.org/handle/10.2312/14803

[11] Daniel A Keim , Florian Mansmann, Jörn Schneidewind, and Hartmut Ziegler . 2006 . Challenges in visual data analysis . In Tenth International Conference on Information Visualisation (IV'06) . IEEE, 9 - 16 . https://doi.org/10.1109/IV. 2006 .31

[12] Joseph Kotlarek , Oh-Hyun Kwon , Kwan-Liu Ma , Peter Eades, Andreas Kerren, Karsten Klein, and Falk Schreiber . 2020 . A Study of Mental Maps in Immersive Network Visualization . In 2020 IEEE Paci c Visualization Symposium (Paci cVis) . IEEE , 1 - 10 .

[13] Paul Milgram , Haruo Takemura, Akira Utsumi, and Fumio Kishino . 1995 . Augmented reality: A class of displays on the reality-virtuality continuum . In Telemanipulator and telepresence technologies , Vol. 2351 . International Society for Optics and Photonics, SPIE, 282 - 292 . https://doi.org/10.1117/12.197321

[14] Fabian Pointecker , Hans-Christian Jetter , and Christoph Anthes . 2020 . Exploration of Visual Transitions Between Virtual and Augmented Reality . In 4th Workshop on Immersive Analytics: Envisioning Future Productivity for Immersive Analytics at CHI .

[15] Rebekka S Renner , Boris M Velichkovsky, and Jens R Helmert . 2013 . The perception of egocentric distances in virtual environments - A review . ACM Computing Surveys (CSUR) 46 , 2 ( 2013 ), 1 - 40 . https://doi.org/10.1145/2543581.2543590

[16] Joan Sol Roo , Jean Basset, Pierre-Antoine Cinquin , and Martin Hachet . 2018 . Understanding Users' Capability to Transfer Information Between Mixed and Virtual Reality: Position Estimation Across Modalities and Perspectives . In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems . 1 - 12 . https://doi.org/10.1145/3173574.3173937

[17] Joan Sol Roo and Martin Hachet . 2017 . One reality: Augmenting how the physical world is experienced by combining multiple mixed reality modalities . In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology . 787 - 795 . https://doi.org/10.1145/3126594.3126638

[18] Joan Sol Roo and Martin Hachet . 2017 . Towards a hybrid space combining Spatial Augmented Reality and virtual reality . In 2017 IEEE Symposium on 3D User Interfaces (3DUI) . IEEE, 195 - 198 . https://doi.org/10.1109/3DUI. 2017 .7893339

[19] Johannes Sorger , Manuela Waldner, Wolfgang Knecht, and Alessio Arleo . 2019 . Immersive Analytics of Large Dynamic Networks via Overview and Detail Navigation . arXiv preprint arXiv: 1910 . 06825 ( 2019 ).