Review of Virtual Display Layouts in HMD-based Extended Reality Parisa Daeijavad1,* , Frank Maurer1 1 Department of Computer Science, University of Calgary, Calgary, Alberta, Canada Abstract Although industry experts and designers provide guidelines for designing multiple display layouts in extended reality (XR), there is a lack of empirical evidence to support these recommendations. In our study, we review papers that have conducted user research and evaluated display layouts in XR to provide a more structured view of evidence-based guidelines in different applications. Our investigation extends beyond exploring the existing studies in this domain, types of XR environments, and display layouts, providing detailed insights on how specific layout choices impact users’ experience (UX) measures in XR environments. Our research paper provides design suggestions for developers, designers, and researchers, offering a summary of design guidelines for various display layouts in XR, supported by existing empirical evidence. Keywords Extended Reality, Spatial Arrangements, Multiple displays, Human Computer Interaction 1. Introduction This review explores multiple display layouts in extended reality (XR). The identified publications address questions and problems related to the usability, design, and implications of virtual reality (VR), augmented reality (AR) and mixed reality (MR) display layouts for various tasks, and the impact of layout configurations on the user’s performance. In this review paper, we explore the existing studies in this domain, their application, types of XR environments, and display layouts. In addition, to find empirical evidence on how to lay out virtual displays in XR, we report on the empirical outcomes derived from existing user studies in the literature. Figure 1: Examples of XR multiple display layouts in [1, 2, 3]. A) Multiple displays in a shape of sphere [1]. B) Floating planes around the user [3]. C) Vertical and Horizontal arrangements in [2]. RealXR: Prototyping and Developing Real-World Applications for Extended Reality, June 4, 2024, Arenzano (Genoa), Italy * Corresponding author. $ parisa.daeijavad@ucalgary.ca (P. Daeijavad); fmaurer@ucalgary.ca (F. Maurer)  0000-0002-9892-9904 (P. Daeijavad); 0000-0002-0240-715X (F. Maurer) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings Although industry experts and designers provide guidelines for designing multiple display layouts in XR, there is a lack of empirical evidence to support these recommendations [4]. In our study, empirical evidence refers to data and observations from experiments and user studies. We collect studies that highlight evidence-based guidelines and are moving away from relying on expert opinions. The study methodologies used in the reviewed publications contain a range of approaches, including examining specific layout and cognitive features of three-dimensional (3D) space, developing and implementing multiple display layouts in XR environments, and comparing different layouts. In this paper, we reported the results of each study as evidence-based design recommendations, principles and guidelines, with measurements such as cognitive load and presence that have been used for the evaluation of XR display layouts. Through this exploration, we aim to contribute to innovative approaches and established frameworks used in publications, to understand challenges and opportunities in multiple display layouts in XR. 2. Related Reviews Many studies have focused on utilizing multiple displays in non-immersive environments [5, 6, 4]. For instance, Roberts and Al-Maneea [6] reviewed 340 visualization tools from articles published between 2012 and 2018 and evaluated different strategies for laying out multiple displays in these papers. However, we found three review papers in immersive environments [7, 8, 9]. We focused on reviews that aim to provide a more structured perspective on evidence- based guidelines for display layouts in XR. Roberts et al. [7] use case studies to discuss the challenges and opportunities of using multiple views 1 in immersive visualization. Knudsen and Carpendale [8] explored the benefits and challenges of using multiple views in immersive analytics. Ma and Millet [9] focused on designing immersive dashboards, which are interactive interfaces that display multivariate data through coordinated views. In this review, our specific emphasis is on displays that have not been the primary focus in previous reviews. Our goal is to suggest existing empirical evidence on how to layout virtual displays in XR. 3. Methodology For conducting a review we used the four-phase process of: identification of research, paper selection process, inclusion criteria and screening phases [11]. To identify relevant papers we used the following query: ("Augmented Reality" OR "Virtual Reality" OR "Mixed Reality" OR "Extended Reality") AND (display2 AND layout). We included the abbreviation of XR terms (AR, VR, MR, XR) and ran the query on Google Scholar linked to our university’s library resources which has subscriptions to almost all computer science and software engineering databases, including but not limited to IEEE Xplore, ACM Digital Library, Scopus, and WebofScience. Figure 2(a) shows the PRISMA chart of the review process. 1 Multiple views is the general term that has been used in the literature for both multiple visualizations such as 2D and 3D charts and multiple displays [10]. 2 The inclusion of alternative terms for "display", e.g., "screen" or "monitor" did not yield any additional results. Figure 2: (a) Flowchart of the review process. (b) Examples of varied shapes of multiple display layouts: A) Lists, Grids, and Arcs layouts [17]. B) Flat, Semi-circle, and Full-circle arrangements [18]. After running the searches, we extracted papers that were published in English, after 1997. Then, we excluded survey papers and deleted duplicates. Furthermore, we excluded patent descriptions, standards, slideshows, and dissertation (e.g., [12]). Then we established a set of inclusion and exclusion criteria to proceed to the screening phase. We kept papers that were on multiple display layouts in XR and conducted quantitative and qualitative studies for layout evaluation. We excluded papers with only one display, and those lacking a detailed study description. In addition, we only focused on HMD XR and exclude mobile-based and desktop XR studies (e.g., [13]). In the next phase, we performed screening on the studies’ title, abstract, and full text using our inclusion and exclusion criteria. This final phase resulted in 14 unique relevant papers on the topic [14, 3, 15, 1, 16, 17, 18, 19, 2, 20, 21, 22, 23, 24]. 4. Results Reviewed papers: In this section, we closely examine the main focus in 14 publications related to XR multiple display layouts, highlighting their design implications. For instance, [15] and [3], explores design considerations of AR monitors for productivity work and examines the usability of AR monitors compared to physical monitors. [1] focuses on the interaction in multiple VR screens for mobile knowledge workers. In [16] the research examines the cognitive features of 3D space for information displays in MR. [17] compares different display arrangements in the searching process, and [18] investigates the effects of display layout on spatial memory, using abstract tasks. In [19], the authors uses measures such as task load and presence to understand the impact of curvature displays on users’ performance. [2] explores the layout of virtual displays in shared transit AR environments, considering factors such as social effects, and the impact of the number of displays. Mori et al. [21] tested the impact of layout configurations of virtual displays in AR. [22] investigated the passenger experience of display layouts in a simulated VR passenger airplane environment, while [23] identifies key factors to consider in the design and evaluation of XR virtual displays for a workplace environment and "DesignAR" [24] addresses design challenges related to mid-air VR/AR interaction on augmented displays. Display Layouts: The reviewed papers used diverse layouts of displays such as spherical, and cylindrical layouts [1], 3D layouts with attributes like depth distance, information layer number, and target relative position [16], and specific shapes like lists, grids, and arcs [17] (Figure 2(b)-A). Some papers represented arrangements, such as flat, semi-circle, and full-circle [18] (Figure 2(b)-B), while others explore layout classification based on position in 3D environments, such as displays placed according to their position in relation to the user’s head position on both the vertical and horizontal axes in [2] (Figure 1-C). Empirical Evidence: The reviewed papers provide evidence-based guidelines for designing multiple display layouts in XR, such as allowing an information touchscreen display to break out of the screen while transitioning from 2D to 3D can be controlled with the original touchscreen [1]. Three design principles for improving the searching process include using uncommon 3D arrangements, faster scans with grid and arc arrangements, and users’ preferences for grid over arc [17]. However, VR and XR have lower usability compared to the physical environment. [19] reveals higher task load, frustration, anxiety, and eye strain in VR, while lower usability, perceived productivity, and well-being in VR compared to the physical environment [19, 20]. In the context of AR displays, guidelines focus on factors like situational awareness, and social norms. [20] suggests that AR can affect self-confidence and users’ feeling of safety. The guiding principle derived from [2] is that in shared transit, social presence play a significant role in how people positioned AR displays. [21] provides guidelines for the design of AREDs, suggesting the significant dependence of the user’s perception on the AR display layouts. [23] provides guidelines about user preferences for the design of XR virtual displays. For example, one guideline suggests that most users (67%) prefer displays no bigger than the field of view with a scaling factor of 1.5 times the size of a physical display of the same resolution [23]. Furthermore, [24] defines the concept of Augmented Displays as the extension of an interactive surface such as a wall display, and mentions the importance of the alignment of augmented displays with the display itself. Although Pavanatto et al. [3] did not explicitly provide any design principles, their studies show that virtual monitors were slower and required more head-turning than a hybrid combination of both physical and virtual monitors. Wang et al. [16] suggest three principles for designing a 3D spatial information layout for a holographic command cabin in an MR environment. These include determining the optimal depth range of information (3.4 m∼3.6 m), limiting the number of layers to five, and ensuring optimal relative positions between layers (0.2m∼0.35m). [22] identifies five principles for multiple display layouts in MR applications: 1) Users prefer virtual displays within their personal space; 2) Design factors should ensure social acceptance; 3) Horizontal layouts may cause discomfort and physical demands; 4) Vertical layouts are preferred for productivity tasks; and 5) Display preferences can vary based on task type. Designers should consider these principles to avoid negative consequences. Overall, the reviewed papers collectively offer design recommendations for diverse XR applications of multiple display layouts. 5. Conclusion and Future Work In conclusion, our study collected a set of recommended design guidelines for multiple display layouts in XR, supported by empirical evidence. Additionally, our investigation explores the existing studies in this domain, types of XR environments, and display layouts. While these studies provide design implications of multiple display layouts in XR, future research in the domain should address the identified limitations in the reviewed studies including small sample sizes, absence of direct comparisons with similar systems, and a lack of evidence-based guidelines for various layouts. References [1] V. Biener, D. Schneider, T. Gesslein, A. Otte, B. Kuth, P. O. Kristensson, E. Ofek, M. Pahud, J. Grubert, Breaking the screen: Interaction across touchscreen boundaries in virtual reality for mobile knowledge workers, IEEE transactions on visualization and computer graphics 26 (2020) 3490–3502. [2] D. Medeiros, M. McGill, A. Ng, R. McDermid, N. Pantidi, J. Williamson, S. Brewster, From shielding to avoidance: Passenger augmented reality and the layout of virtual displays for productivity in shared transit, IEEE Transactions on Visualization and Computer Graphics 28 (2022) 3640–3650. [3] L. Pavanatto, C. North, D. A. Bowman, C. Badea, R. Stoakley, Do we still need physical monitors? an evaluation of the usability of ar virtual monitors for productivity work, in: 2021 IEEE Virtual Reality and 3D User Interfaces (VR), IEEE, IEEE, Lisboa, Portugal, 2021, pp. 759–767. [4] A. R. Shaikh, D. Koop, H. Alhoori, M. Sun, Toward systematic design considerations of organizing multiple views, in: 2022 IEEE Visualization and Visual Analytics (VIS), IEEE, 2022, pp. 105–109. [5] J. C. Roberts, H. M. A. Al-Maneea, Study of multiple view layout strategies in visualisation, in: IEEE Conference on Visualization: InfoVis, 2018. [6] H. M. Al-maneea, J. C. Roberts, Towards quantifying multiple view layouts in visualisation as seen from research publications, in: 2019 IEEE Visualization Conference (VIS), IEEE, 2019, pp. 121–121. [7] J. C. Roberts, P. W. Butcher, P. D. Ritsos, One view is not enough: review of and encour- agement for multiple and alternative representations in 3d and immersive visualisation, Computers 11 (2022) 20. [8] S. Knudsen, S. Carpendale, Multiple views in immersive analytics, in: IEEE VIS 2017 Workshop on Immersive Analytics, 2017. [9] Q. Ma, B. Millet, Design guidelines for immersive dashboards, in: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, volume 65, SAGE Publications Sage CA: Los Angeles, CA, 2021, pp. 1524–1528. [10] J. C. Roberts, H. Al-maneea, P. W. Butcher, R. Lew, G. Rees, N. Sharma, A. Frankenberg- Garcia, Multiple views: different meanings and collocated words, in: Computer Graphics Forum, volume 38, Wiley Online Library, 2019, pp. 79–93. [11] B. Kitchenham, P. Brereton, A systematic review of systematic review process research in software engineering, Information and software technology 55 (2013) 2049–2075. [12] A. Febretti, Multiview Immersion in Hybrid Reality Environments, Ph.D. thesis, University of Illinois at Chicago, 2017. [13] Y. Kudo, K. Takashima, M. Fjeld, Y. Kitamura, Adaptable: extending reach over large tabletops through flexible multi-display configuration, in: Proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces, 2018, pp. 213–225. [14] S. Butscher, S. Hubenschmid, J. Müller, J. Fuchs, H. Reiterer, Clusters, trends, and outliers: How immersive technologies can facilitate the collaborative analysis of multidimensional data, in: Proceedings of the 2018 CHI conference on human factors in computing systems, 2018, pp. 1–12. [15] L. Pavanatto, Designing augmented reality virtual displays for productivity work, in: 2021 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR- Adjunct), IEEE, Bari, Italy, 2021, pp. 459–460. [16] W. Wang, X. Hong, S. Dang, N. Xu, J. Qu, 3d space layout design of holographic command cabin information display in mixed reality environment based on hololens 2, Brain Sciences 12 (2022) 971. [17] A. R. Ward, R. Capra, Immersive search: using virtual reality to examine how a third dimension impacts the searching process, in: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, Association for Computing Machinery, New York, NY, USA, 2020, pp. 1621–1624. [18] J. Liu, A. Prouzeau, B. Ens, T. Dwyer, Effects of display layout on spatial memory for immersive environments, Proceedings of the ACM on Human-Computer Interaction 6 (2022) 468–488. [19] V. Biener, S. Kalamkar, N. Nouri, E. Ofek, M. Pahud, J. J. Dudley, J. Hu, P. O. Kristensson, M. Weerasinghe, K. Č. Pucihar, et al., Quantifying the effects of working in vr for one week, IEEE Transactions on Visualization and Computer Graphics 28 (2022) 3810–3820. [20] V. Biener, S. Kalamkar, J. J. Dudley, J. Hu, P. O. Kristensson, J. Müller, J. Grubert, Working with xr in public: Effects on users and bystanders, 2023. arXiv:2310.09786. [21] S. Mori, Y. Kataoka, S. Hashiguchi, Exploring pseudo-weight in augmented reality extended displays, in: 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), IEEE, IEEE, Christchurch, New Zealand, 2022, pp. 703–710. [22] A. Ng, D. Medeiros, M. McGill, J. Williamson, S. Brewster, The passenger experience of mixed reality virtual display layouts in airplane environments, in: 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), IEEE, Bari, Italy, 2021, pp. 265–274. [23] Y. Zhang, J. Sun, Q. Ding, L. Zhang, Q. Wang, X. Geng, Y. Rui, Towards workplace metaverse: A human-centered approach for designing and evaluating xr virtual displays, International Journal of Human–Computer Interaction 0 (2023) 1–16. [24] P. Reipschläger, R. Dachselt, Designar: Immersive 3d-modeling combining augmented reality with interactive displays, in: Proceedings of the 2019 ACM International Conference on Interactive Surfaces and Spaces, Association for Computing Machinery, New York, NY, USA, 2019, pp. 29–41.