Exploring the Benefits and Challenges of Data Physicalization S. Sandra Bae1 , Danielle Albers Szafir1,2 and Ellen Yi-Luen Do1 1 University of Colorado, Boulder, ATLAS Institute, 1125 18th St. 320 UCB, Boulder CO, USA 2 University of North Carolina at Chapel Hill, Computer Science, 201 S Columbia St, Chapel Hill NC, USA Abstract Data physicalization has emerged as a new method to represent and interact with data physically rather than digitally. Physical representations afford visual analysis in comparable ways to traditional, desktop- based visualization by introducing new capabilities, such as facilitating tactile manipulation, accessible interactions, and immersion, that are beyond traditional 2D visualizations. However, physicalization has historically been a niche aspect of visualization research due to its unique challenges. This work discusses the current challenges and highlights three areas where data physicalization can aid existing research thrusts: broadening participation, supporting analytics, and promoting creative expression. Keywords data physicalization, challenges, research agenda 1. Introduction Data physicalization—the practice of mapping data to physical form—sits at the crossroads of various domains, including data visualization, tangible user interaction, and design [1]. Data has been traditionally visualized through the desktop model, but current and anticipated advancements in material science and digital fabrication are radically changing how we can possibly represent and interact with data. Data physicalization, as a growing field, is not only introducing new capabilities (e.g., tactile manipulation [2], accessible interactions [3, 4], immersion [5]), but also expanding academic discourse on how we traditionally view data [6]. Despite these advancements, physicalization has historically been a niche aspect of visualization. The challenges of physicalization stem from how the majority of physicalization artifacts are single prototypes. Without broader synthesis, these individual design explorations prevent physicalization from maturing on the field’s broader goals, including fabrication development, theory-building, and ethical and societal impact. To that end, previous research surveyed and cataloged physicalizations through different lenses, including semiotics [7, 8, 9], fabrication techniques [10], and visualization tasks [11]. However, these past efforts often focus on a single perspective rather than reflecting on the broader intellectual foundations of the HCI communities that data physicalization sits upon. From a cross-disciplinary lens, this article Proceedings of ETIS 2022, November 7-10, 2022, Toulouse, France Envelope-Open sandra.bae@colorado.edu (S. S. Bae); danielle.szafir@cs.unc.edu (D. A. Szafir); ellen.do@colorado.edu (E. Y. Do) GLOBE https://sandrabae.github.io/ (S. S. Bae); https://danielleszafir.com/ (D. A. Szafir); https://www.colorado.edu/atlas/ellen-yi-luen-do (E. Y. Do) © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) highlights three promising research areas where data physicalization can aid: broadening participation, supporting analytics, and promoting creative expression. Thus, this article discusses the cross-disciplinary challenges that must be addressed for physicalizations to move from simply vision to being successfully embedded in different domain applications. 2. Background and Motivation To invest resources into addressing the challenges outlined in this article, researchers and target users must see clear benefits in the development of data physicalization. This section briefly explores the need, purposes, and benefits of physicalizing data. 2.1. Physicalization: Post WIMP Interfaces Many data-driven systems use a WIMP (Windows, Icons, Menus, and a Pointer) paradigm and rely on the graphical user interface (GUI) [12]. However, the WIMP and GUI limit how and where users can represent and interact with data (i.e., context) [12, 13]. Notably, these con- straints introduce challenges when users (1) work with multi-dimensional data with an inherent structure and would benefit from physical models (e.g., high-fidelity, material-realistic surgical models) compared to 2D graphical renditions or (2) work in contexts where the traditional GUI is infeasible (e.g., robotic operators supervising in an outdoor field test, doctors operating during surgeries). Data physicalizations provide the opportunity to fundamentally transform human interactions with data. By removing the constraints of the pixels and moving data rep- resentations from flat displays into the physical world, data interactions become more inclusive [14, 15], and we can broaden the ways we can experience and communicate data [16, 17, 18]. Knowing systematically how physicalization differs or fits within existing visualization tools requires a deep understanding. Due to physicalization’s cross-disciplinary nature, this understanding calls for careful considerations that are beyond conventional data visualizations [11, 19]. For example, while all visualizations must consider the expressivity of their designs (e.g., data encoding and data interactions), physicalization designers must also be mindful of the physicalization’s structural and contextual considerations [19]. Each data physicalization artifact (un)consciously reflects different community perspectives and values [19]. For example, physicalizations from the data visualization community mainly focus on how to effectively encode data and support analytical tasks. Physicalizations from the tangible user interface community explore how to amplify the capabilities of the human body and physical world for interaction design, while artifacts from design investigate how to identify and leverage a material’s potential. In short, by developing this deeper understanding, the opportunities data physicalization, as a field, faces might be both expanding (through values of different disciplines) and converging (through the advances in the field). 2.2. Purposes and Benefits: Situating within Existing Research Thrusts Through this cross-disciplinary lens, the opportunities data physicalization affords generally align with three existing research thrusts: broadening participation, supporting analytics, and promoting creative expression. These research thrusts are derived from the authors’ meta-content analysis of tasks presented in Bae et. al’s design space for physicalizations [19]. This work aims to outline the emerging trends of physicalization research since Jansen et al.’s formalization of the field in 2015 [1]. Thrust 1 focuses on understanding how physical representations can enable broader participation when working with data. Thrust 2 focuses on how to employ fabrication techniques to efficiently represent and interact with data and how to leverage the analytical affordances of physical representations. Thrust 3 explores how data acts as a material to create new designs. Thrust 1: Broadening Participation. The use and need for visualizations and data are not just confined to experts anymore. But the design of visualizations holds implicit assumptions about the user’s sensory, cognitive, and motor abilities [15]. Scholarship highlights how exclu- sively digital solutions present challenges and limitations to certain populations (e.g., people with low vision or visual impairments [4, 20], children [21]). These challenges present the need to further investigate how to expand the ways we can interact with data and visualizations. One form of exploration is focusing on non-traditional audiences: children. The novelty of physicalizing data offers an unprecedented method for children to engage with and better understand data. For example, Data is Yours is a toolkit [21] made out of everyday materials (e.g., paper, cardboard). The toolkit explores how constructionist practices can broaden ways of introducing children to data visualization concepts, and in turn, cultivate their data visu- alization literacy (DVL). Past work on children’s DVL has often relied on exclusively digital solutions, where they may come off as “black boxes” to young children and limit their embodied experiences[22]. Physical representations, in contrast, enable children to engage in embodied learning that would not be possible with a 2D screen. Thrust 2: Supporting Analytics. Rooted in visualization, Jansen et al. [1] formally defined physicalization as a research field that focuses on “how computer-supported, physical representations of data (i.e., physicalizations) can support cognition, communication, learning, problem solving, and decision making”. As such, researchers investigated how to leverage physicality for data representation [23, 24, 25], data interaction [26, 27, 2], and social purposes [28, 29]. But there continue to be underlying challenges, namely understanding which datasets would be the most meaningful to analyze and knowing how to analyze them physically. By looking at other disciplines, we see possibilities in how physicality may aid in pre-surgical planning [30] or analyzing star formation [31]. But once a dataset has been chosen, it is important to know how physicalization interfaces can support rich, dynamic interactions. Visualization shows the importance of dynamic interactions, where analysts engage in a series of interactions when exploring a dataset [32]. This reflects findings on how the number of data interactions a system supports affects the richness of the data exploration [19]. Thus, there is a rich opportunity to explore how physicalizations utilizing state-of-the-art fabrication techniques can support analytics and decision-making processes. Thrust 3: Promoting Creative Expression. Engineers, scientists, and artists are building physical artifacts guided by data [33, 34]. The cross-disciplinary space of data physicalization reveals examples of physical artifacts that are encoded with data but do not meet Jansen et al.’s utilitarian definition of data physicalization (Thrust 2). Collectively, these artifacts are examples where data is used as a material source when making. Like a craftsman working with their surrounding materials, data is now another material source and is part of the artisanal spirit when making. For instance, Friske et al. created a data-encoded scarf where “[d]ata became one material among others, no more or less important in design” [34]. The discussion reveals how the authors had to negotiate between how accurately they wanted to encode the data in the physical representation versus embracing the spirit of the material (i.e., yarn). This perspective stands in contrast to Thrust 2, which is more cognitively oriented. In this case, data encoding is prioritized in terms of its effectiveness in helping users carry out their analytical tasks, such as comparing values or estimating correlations [1]. As data continues to be embedded in our world, Thurst 3 highlights how data-driven artifacts can be used for customization [35, 36] or hedonic purposes [37, 38]. 3. Challenges By understanding the benefits of physicalizing data (Section 2.2), we can highlight emerging challenges. The presented challenges are not comprehensive nor do they discuss the downsides of physicalizations (e.g., issues related to sustainability and product lifecycle). Rather they aim to open discussions on where research efforts should be targeted based on current trends. C1: Fabrication: Constructing data physicalizations involves a large design space [19], which includes but is not limited to material choice, scale [39], interactions, and data encoding. This introduces combinatorial possibilities that a user has to consider when designing physi- calizations. In addition, despite material and fabrication advancements, we lack fundamental guidance on how to leverage the analytical affordances of physical representations. This also highlights a technical challenge in understanding how to accurately map data to the given material. Users will need to consider the tradeoffs between material properties (e.g., fluid-based systems [40] and smart materials over electromechanical and magnetic actuators [41]). C2: Interpretation: Currently, there is no formal design language (analogous to Grammar of Graphics [42]) that helps users interpret data physicalizations. Many data physicalizations either use conventional 2D data visualization representations or are idiosyncratic where each encoding is unique to the creator. This starkly contrasts with the field of information visualization where there is an existing design language to help communicate data findings. The two challenges are closely intertwined due to their overlapping focus on data encoding. For instance, interpreting a physicalization will ultimately arise from its fabrication technique. However, understanding the scope of each challenge can help tackle its associated problems. E.g., How would the interpretation of a physicalization change if only the fabrication method changed (e.g., CNC, cast and molding, 3D printing, papercraft)? Through this process, one may uncover different physical affordances or values [43] of each fabrication technique. 4. Conclusion Data is ubiquitous where it is now woven into the fabrics of our everyday life. However, traditional methods of visualizing data digitally limit how we can represent and interact with data. Physicalization aims to expand this effort, but the field faces challenges due to its cross- disciplinary nature. This article discusses the benefits and purposes of physicalizing data to help invest efforts in addressing the outlined challenges. But these challenges will, no doubt, evolve as technology matures, society changes, and applications of physicalization concretize. To facilitate this maturation, this work serves as a discussion point to shape the field’s future. Acknowledgments This material is based upon work supported by the National Science Foundation under Grant No. IIS-2040489, No. IIS-2046725, and STEM+C 1933915. References [1] Y. Jansen, P. Dragicevic, P. Isenberg, J. Alexander, A. Karnik, J. Kildal, S. Subramanian, K. Hornbæk, Opportunities and Challenges for Data Physicalization, Association for Computing Machinery, New York, NY, USA, 2015, p. 3227–3236. URL: https://doi.org/10. 1145/2702123.2702180. [2] C. Perin, M. Le Goc, R. Di Vozzo, J.-D. Fekete, P. Dragicevic, Diy bertin matrix, in: Proceedings of the CHI Workshop on Exploring the Challenges of Making Data Physical, 2015. [3] D. Fan, A. Fay Siu, S. O’Modhrain, S. Follmer, Constructive visualization to inform the design and exploration of tactile data representations, in: The 22nd International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS ’20, Association for Computing Machinery, New York, NY, USA, 2020. URL: https://doi.org/10.1145/3373625. 3418027. doi:10.1145/3373625.3418027 . [4] R. Suzuki, A. Stangl, M. D. Gross, T. Yeh, Fluxmarker: Enhancing tactile graphics with dynamic tactile markers, in: Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS ’17, Association for Computing Machinery, New York, NY, USA, 2017, p. 190–199. URL: https://doi.org/10.1145/3132525. 3132548. doi:10.1145/3132525.3132548 . [5] W. Willett, Y. Jansen, P. Dragicevic, Embedded data representations, IEEE Transactions on Visualization and Computer Graphics 23 (2017) 461–470. doi:10.1109/TVCG.2016. 2598608 . [6] D. Offenhuber, What we talk about when we talk about data physicality, IEEE Computer Graphics and Applications 40 (2020) 25–37. doi:10.1109/MCG.2020.3024146 . [7] J. Zhao, A. V. Moere, Embodiment in data sculpture: A model of the physical visualization of information, in: Proceedings of the 3rd International Conference on Digital Interactive Media in Entertainment and Arts, DIMEA ’08, Association for Computing Machinery, New York, NY, USA, 2008, p. 343–350. URL: https://doi.org/10.1145/1413634.1413696. doi:10. 1145/1413634.1413696 . [8] A. Vande Moere, S. Patel, The physical visualization of information: Designing data sculptures in an educational context, in: M. L. Huang, Q. V. Nguyen, K. Zhang (Eds.), Visual Information Communication, Springer US, Boston, MA, 2010, pp. 1–23. [9] D. Offenhuber, O. Telhan, Indexical visualization–the data-less information display, in: Ubiquitous Computing, Complexity, and Culture, Routledge, New York, NY, USA, 2015, pp. 288–302. [10] H. Djavaherpour, F. Samavati, A. Mahdavi-Amiri, F. Yazdanbakhsh, S. Huron, R. Levy, Y. Jansen, L. Oehlberg, Data to physicalization: A survey of the physical rendering process, Computer Graphics Forum 40 (2021) 569–598. URL: https://onlinelibrary.wiley.com/doi/ abs/10.1111/cgf.14330. doi:https://doi.org/10.1111/cgf.14330 . [11] P. Dragicevic, Y. Jansen, A. Vande Moere, Data Physicalization, in: J. Vanderdonckt (Ed.), Springer Handbook of Human Computer Interaction, Springer Reference, Springer, 2021. URL: https://hal.inria.fr/hal-02113248. [12] B. Lee, P. Isenberg, N. H. Riche, S. Carpendale, Beyond mouse and keyboard: Expanding design considerations for information visualization interactions, IEEE Transactions on Visualization and Computer Graphics 18 (2012) 2689–2698. [13] A. Van Dam, Post-wimp user interfaces, Communications of the ACM 40 (1997) 63–67. [14] S. R. Klemmer, B. Hartmann, L. Takayama, How bodies matter: Five themes for interaction design, in: Proceedings of the 6th Conference on Designing Interactive Systems, DIS ’06, Association for Computing Machinery, New York, NY, USA, 2006, p. 140–149. URL: https://doi.org/10.1145/1142405.1142429. doi:10.1145/1142405.1142429 . [15] K. Marriott, B. Lee, M. Butler, E. Cutrell, K. Ellis, C. Goncu, M. Hearst, K. McCoy, D. A. Szafir, Inclusive data visualization for people with disabilities: a call to action, Interactions 28 (2021) 47–51. [16] O. Shaer, E. Hornecker, Tangible user interfaces: Past, present, and future directions, Foundations and Trends in Human–Computer Interaction 3 (2010) 4–137. URL: http: //dx.doi.org/10.1561/1100000026. doi:10.1561/1100000026 . [17] J. Hollan, E. Hutchins, D. Kirsh, Distributed cognition: toward a new foundation for human- computer interaction research, ACM Transactions on Computer-Human Interaction (TOCHI) 7 (2000) 174–196. [18] Z. Pousman, J. Stasko, M. Mateas, Casual information visualization: Depictions of data in everyday life, IEEE Transactions on Visualization and Computer Graphics 13 (2007) 1145–1152. doi:10.1109/TVCG.2007.70541 . [19] S. S. Bae, C. Zheng, M. E. West, E. Y.-L. Do, S. Huron, D. A. Szafir, Making data tangible: A cross-disciplinary design space for data physicalization, in: CHI Conference on Human Factors in Computing Systems, CHI ’22, Association for Computing Machinery, New York, NY, USA, 2022. URL: https://doi.org/10.1145/3491102.3501939. doi:10.1145/3491102. 3501939 . [20] D. Guinness, A. Muehlbradt, D. Szafir, S. K. Kane, Robographics: Dynamic tactile graphics powered by mobile robots, in: The 21st International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS ’19, Association for Computing Machinery, New York, NY, USA, 2019, p. 318–328. URL: https://doi.org/10.1145/3308561.3353804. doi:10. 1145/3308561.3353804 . [21] S. S. Bae, R. Vanukuru, R. Yang, P. Gyory, R. Zhou, E. Y.-L. Do, D. A. Szafir, Cultivating visualization literacy for children through curiosity and play, arXiv, 2022. URL: https://arxiv. org/abs/2208.05015. doi:10.48550/ARXIV.2208.05015 , accepted to IEEE Transaction on Visualization and Graphics (also appearing in the proceedings of IEEE VIS 2022). [22] M. Resnick, R. Berg, M. Eisenberg, Beyond black boxes: Bringing transparency and aesthetics back to scientific investigation, The Journal of the Learning Sciences 9 (2000) 7–30. [23] A. Everitt, J. Alexander, Polysurface: A design approach for rapid prototyping of shape- changing displays using semi-solid surfaces, in: Proceedings of the 2017 Conference on Designing Interactive Systems, DIS ’17, Association for Computing Machinery, New York, NY, USA, 2017, p. 1283–1294. URL: https://doi.org/10.1145/3064663.3064677. doi:10.1145/ 3064663.3064677 . [24] Y. Jansen, P. Dragicevic, J.-D. Fekete, Evaluating the Efficiency of Physical Visualizations, Association for Computing Machinery, New York, NY, USA, 2013, p. 2593–2602. URL: https://doi.org/10.1145/2470654.2481359. [25] R. G. Raidou, M. E. Gröller, H.-Y. Wu, Slice and dice: A physicalization workflow for anatomical edutainment, in: Computer Graphics Forum, volume 39, Wiley Online Library, 2020, pp. 623–634. [26] F. Taher, Y. Jansen, J. Woodruff, J. Hardy, K. Hornbæk, J. Alexander, Investigating the use of a dynamic physical bar chart for data exploration and presentation, IEEE Transactions on Visualization and Computer Graphics 23 (2017) 451–460. doi:10.1109/TVCG.2016. 2598498 . [27] A. Veldhuis, R.-H. Liang, T. Bekker, Coda: Collaborative data interpretation through an interactive tangible scatterplot, in: Proceedings of the Fourteenth International Conference on Tangible, Embedded, and Embodied Interaction, TEI ’20, Association for Computing Machinery, New York, NY, USA, 2020, p. 323–336. URL: https://doi.org/10.1145/3374920. 3374934. doi:10.1145/3374920.3374934 . [28] L. J. Perovich, S. A. Wylie, R. Bongiovanni, Chemicals in the creek: designing a situated data physicalization of open government data with the community, IEEE Transactions on Visualization and Computer Graphics 27 (2021) 913–923. doi:10.1109/TVCG.2020. 3030472 . [29] A. Thudt, U. Hinrichs, S. Huron, S. Carpendale, Self-Reflection and Personal Physicalization Construction, Association for Computing Machinery, New York, NY, USA, 2018, p. 1–13. URL: https://doi.org/10.1145/3173574.3173728. [30] A. Hosny, S. J. Keating, J. D. Dilley, B. Ripley, T. Kelil, S. Pieper, D. Kolb, C. Bader, A.-M. Pobloth, M. Griffin, et al., From improved diagnostics to presurgical planning: high- resolution functionally graded multimaterial 3d printing of biomedical tomographic data sets, 3D Printing and Additive Manufacturing 5 (2018) 103–113. [31] N. Imara, J. C. Forbes, J. C. Weaver, Touching the stars: Using high-resolution 3d printing to visualize stellar nurseries, The Astrophysical Journal Letters 918 (2021) L3. [32] S. Card, J. Mackinlay, B. Shneiderman, Readings in information visualization: Using vision to think, Technical Communication Quarterly 9 (1999) 347–351. [33] A. Desjardins, T. Tihanyi, Listeningcups: A case of data tactility and data stories, in: Proceedings of the 2019 on Designing Interactive Systems Conference, DIS ’19, Association for Computing Machinery, New York, NY, USA, 2019, p. 147–160. URL: https://doi.org/10. 1145/3322276.3323694. doi:10.1145/3322276.3323694 . [34] M. Friske, J. Wirfs-Brock, L. Devendorf, Entangling the roles of maker and interpreter in interpersonal data narratives: Explorations in yarn and sound, in: Proceedings of the 2020 ACM Designing Interactive Systems Conference, Association for Computing Machinery, New York, NY, USA, 2020, p. 297–310. URL: https://doi.org/10.1145/3357236.3395442. [35] T. Nachtigall, O. Tomico, R. Wakkary, P. van Dongen, Encoding Materials and Data for Iterative Personalization, Association for Computing Machinery, New York, NY, USA, 2019, p. 1–12. URL: https://doi.org/10.1145/3290605.3300749. [36] L. Hale, E. Linley, D. M. Kalaskar, A digital workflow for design and fabrication of bespoke orthoses using 3d scanning and 3d printing, a patient-based case study, Scientific reports 10 (2020) 1–7. [37] R. A. Khot, J. Andres, J. Lai, J. von Kaenel, F. F. Mueller, fantibles: Capturing cricket fan’s story in 3d, in: Proceedings of the 2016 ACM Conference on Designing Interactive Systems, DIS ’16, Association for Computing Machinery, New York, NY, USA, 2016, p. 883–894. URL: https://doi.org/10.1145/2901790.2901886. doi:10.1145/2901790.2901886 . [38] B. Nissen, J. Bowers, Data-Things: Digital Fabrication Situated within Participatory Data Translation Activities, Association for Computing Machinery, New York, NY, USA, 2015, p. 2467–2476. URL: https://doi.org/10.1145/2702123.2702245. [39] I. López García, E. Hornecker, Scaling data physicalization–how does size influence experience?, in: Proceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction, 2021, pp. 1–14. [40] H. Mor, T. Yu, K. Nakagaki, B. H. Miller, Y. Jia, H. Ishii, Venous Materials: Towards Interactive Fluidic Mechanisms, Association for Computing Machinery, New York, NY, USA, 2020, p. 1–14. URL: https://doi.org/10.1145/3313831.3376129. [41] M. Sturdee, J. Alexander, Analysis and classification of shape-changing interfaces for design and application-based research, ACM Computing Surveys (CSUR) 51 (2018) 1–32. [42] L. Wilkinson, The grammar of graphics, in: Handbook of computational statistics, Springer, 2012, pp. 375–414. [43] J. J. Van Wijk, The value of visualization, in: VIS 05. IEEE Visualization, 2005., IEEE, 2005, pp. 79–86.