Reflections about Symbolic vs. Iconic Representations in TUIs Lou Schwartz Thibaud Latour Luxembourg Institute of Science and Technology (LIST) 5, avenue des Hauts-Fourneaux, L-4362 Esch-sur-Alzette, Luxembourg {firstname.lastname}@list.lu ABSTRACT Embedded and Embodied Interactions) only one paper Currently designed Tangible User Interfaces (TUIs) deals with these three keywords [1]. propose both iconic and symbolic tangible objects (TO). Since iconic TOs should enable to interact more naturally The objective of this paper is to exchange reflections about like in the real world and, hypothetically, require less symbolic and iconic representations in TUIs to discuss them learning time than symbolic TOs, some questions arise: and, particularly, to propose some hypotheses on when to Why do symbolic TOs exist? When to use iconic or use one or the other mode. In the following section, some symbolic representation in TOs? This paper discusses these examples of iconic and symbolic Tangible User Interface questions and makes some assumptions on the abstraction Objects (TOs) are exposed. Then, a discussion on the of concepts, the function of TOs (container, token or tool) difficulties to design an iconic TUI is opened. and the context influencing the design choices of TOs. STATE OF THE ART Author Keywords Symbolic and iconic representations of digital data and Symbolic, Iconic, TUI, Tangible User Interface, Design. tools are used in TUIs. This section presents some examples of both of them. ACM Classification Keywords H.5.m. Information interfaces and presentation (e.g., HCI): Examples of symbolic TUIs Miscellaneous. Symbolic TUIs often use boxes and cylinders as TOs to manipulate information and as actuators in different INTRODUCTION domains. Objects used to represent data and actions in an interface can be symbolic or iconic. Chandler defines symbolic and 1 2 iconic dimensions as follows [3]: • Symbol/symbolic: a mode in which the signifier does not resemble the signified but which is fundamentally arbitrary or purely conventional -so that the relationship must be learnt: e.g. language in general (plus specific languages, alphabetical letters, punctuation marks, 3 4 words, phrases and sentences), numbers, Morse code, traffic lights, national flags; • Icon/iconic: a mode in which the signifier is perceived as resembling or imitating the signified (recognizably looking, sounding, feeling, tasting or smelling like it) - being similar in possessing some of its qualities: e.g. a portrait, a cartoon, a scale-model, onomatopoeia, Figure 1: Examples of symbolic TUIs: (1) ReacTable [10]; metaphors, ’realistic’ sounds in ’programme music’, (2) BeatBlocks [7]; (3) Urban planning by Knecht [12]; sound effects in radio drama, a dubbed film soundtrack, (4) GIS by Jones and Maquil [11]. imitative gestures; For instance, musical applications like the ReacTable [10], A third mode is defined by Chandler (index) but is not where the user turns or connects cubes, squares, discs, etc. addressed in this paper. to combine different sound items like synthesizers, effects Semiotics of TUIs is not a well-documented subject. For and samples to compose a music flow, or the BeatBlocks instance, in the ACM digital library, only 3 papers can be [7] where the user places blocks representing sounds in a found with the words ‘iconic’ and ‘symbolic’ and container to compose music. Urban planning also uses ‘tangible’, 22 papers with the words ‘iconic’ and ‘tangible’ boxes to represent the buildings’ location and a disc to and 38 papers with the words ‘symbolic’ and ‘tangible’. In modify time in order to display the shadow of buildings on TEI conference proceeding (Conference on Tangible, the map [12] or to manipulate maps for logistics in a GIS (Geographical Information System) [11] by activating different layers on a base map with squares, manipulating data, to make the boundary between the real world and the the map with a disc to pan or zoom and display more digital world narrower [14]. In this perspective, iconic TOs information about a specific point (using a triangular should allow to interact more naturally like in the real pointer). world. Furthermore, hypothetically, the time needed to learn how to use TOs should be less with iconic than with Examples of iconic TOs Iconic tangible objects, enabling to interact with digital symbolic TOs. Indeed, TOs’ functions are better understood data, look like objects in the real world. when they are iconic than symbolic [1] and users are more involved [9]. Furthermore, the comprehension and transfer in learning experiences are better with iconic than symbolic 1 representations to solve problems of low and high complexity. In addition, for users with low prior knowledge, it is better to use the iconic representation [13]. This is supported by the fact that icons are closer to perception patterns whereas symbols are related to their referent through higher level concepts. This induces two different cognitive processes [18]. 2 Then, iconic TOs should be used more often than symbolic TOs. So, why are there symbolic TOs? In the following some hypotheses or topic discussions are proposed to begin to address this question. Finding the right representation is not so easy What is the right iconic representation to delete a data item 3 4 in a TUI? This type of question is asked at every new interaction mode. As shown for gestural interaction, the answer is not obvious (e.g., for the question of which gesture should be used to drag a data on a digital tabletop, different answers are given [6]). This shows that no symbol is totally universal. This would certainly be the same for the determination of TOs’ manipulations by users. When cultural differences, expertise, sensitivity and innate 5 knowledge are taken into account, this difference is more important. Manage characteristics of the TO A TO is not always only a marker of the presence or Figure 2: Some examples of iconic TUI: (1) CapTUI [2]; absence of data in the digital model, but also the TO gives (2) GuitarHero® [5]; (3) Teegi [4]; (4) I/O Brush [17]; (5) access to the characteristics of data represented by the JabberStamp [16]. object. For instance, in an urban planner, a manipulated TO Some examples of iconic TOs are the tangible drawing tool could be a parking lot that should be placed in the district. in the form of a paintbrush developed by the MIT which The system could enable users to characterize the parking captures any real world color and is used to paint on a lots on several dimensions For instance: the type of parking display [17], or the CapTUI (ruler, protractor and set (aerial, ground or underground), the accessibility (private, square) of Blagojevic and Plimmer [2]. For music and public or public with lots for people with disabilities) or the sound managing we can found GuitarHero® [5] which number of places. Different design approaches can be used: enables one to play guitar by pushing buttons on a quite real it is possible to give as much TO for the same concept that looking guitar and the JabberStamp [16] which proposes a they have crossed characteristics (in this example 3x3xn recorder (symbolized by a microphone) and a speaker to possibilities exist). But an exponential need of TOs will be augment drawings with sounds. Iconic TOs are also used in observed. Users could be confused in front of such a the medical domain with Teegi (tangible EEG Interface) quantity of TOs. Or it is possible to enable users to that shows a patient’s brain activity in real-time and enables characterize the TO by manipulating it. E.g., by using some the user to interact with [4]. bricks to add on the parking lot to increase the number of places, and the color of these bricks could signify that lots REFLECTIONS are private, public or for disabled people. Find the right Why do symbolic TOs exist? iconic representation for the TO and its characteristics is TUIs seek to embody interactions, to give the sensation of not tricky. Mixing symbolic and iconic could be an option. interacting easily as in the real world, to manipulate digital Represent abstract concepts the number of TOs available (give only a reasonable Sometimes, abstract concepts should also be represented, number of TOs to handle) and of cost limits; like in modeling [15]. Modeling languages have tried to represent concepts more appropriately, but for such abstract • (3) A mix of symbolic and iconic TOs could be a good concepts, finding an iconic representation is not obvious or compromise; even possible. • (4) Use of symbolic and iconic could depend on the Representation is not the key subject of experiments kind of TO in regard of a TOs’ taxonomy. For instance, As shown by the low number of referenced publications in containers should be symbolic, tokens should be as ACM digital library on iconic, symbolic and tangible, the iconic as possible and tools could be both; representation of TOs, in these terms, doesn’t seem to be a • (5) A same concept/data should be represented by an trendy research question. Indeed the focus of research was iconic TO in a certain context and by a symbolic TO in more on the validation of this new way of interaction and another context. paradigm. The question should be raised in following years. This reflection raises other questions: how to design a Another question arises: when to use iconic or symbolic usable iconic TO? What is the taxonomy of TOs? When representation in TOs? should symbolic and iconic be used? Could a mix of them When to use iconic or symbolic representation in TOs? be a good compromise? These are questions that we will To answer this question, taxonomy of TOs should be done. address in future works. Some inputs can be found in Holmquist et al [8]. Three ACKNOWLEDGMENTS types of TOs are defined: The authors acknowledge all their colleagues who are • Containers: contain digital information, like a drive. working on this question at LIST, who nourished this They are generic, that means “the physical properties reflection, in particular Valérie Maquil, Séverine Mignon of a container do not reflect the nature of the digital and Annie Guerriero. information it is associated with”. REFERENCES • Tokens: represent particular digital information. 1. Bakker, S., Vorstenbosch, D., van den Hoven, E., “Tokens are objects that physically resemble the Hollemans, G. and Bergman, T. Tangible interaction in information they represent in some way”. tabletop games: studying iconic and symbolic play pieces. In Proc. international conference on Advances in • Tools: are functions to manipulate the digital model, computer entertainment technology, ACM (2007), 163- for instance zoom, pan, rotate and magnify TOs. 170. In regard of this first distinction between TOs: 2. Blagojevic, R. and Plimmer, B. CapTUI: geometric drawing with tangibles on a capacitive multi-touch • Containers should be symbolic, as they are generic. display. In Proc. IFIP Conference on Human-Computer • Tokens should be as iconic as possible, in regard of Interaction, Springer Berlin Heidelberg (2013), 511- other constraints (abstract vs. concrete data, etc.) 528. 3. Chandler, D. Semiotics for beginners. (1994). • Tools could be iconic or symbolic. 4. Frey, J., Gervais, R., Fleck, S., Lotte, F. and Hachet, M., The previous question gives some directions: an abstract Teegi: tangible EEG interface. In Proc. 27th annual concept should be represented by a symbolic TO. A ACM symposium on User interface software and concrete concept with several tunable characteristics could technology, ACM (2014), 301-308. be represented by all iconic TOs or by a mix of iconic and symbolic. 5. Guitar Hero®, Activision Publishing Inc, https://www.guitarhero.com/, last access 03.15.2017. CONCLUSION At first sight, it seems to be natural to have very iconic 6. Hinrichs, U. and Carpendale, S., 2011, May. Gestures in TOs. But, as observed, a lot of symbolic TOs exist. This the wild: studying multi-touch gesture sequences on can be explained by several hypotheses: interactive tabletop exhibits. In Proc. of the SIGCHI Conference on Human Factors in Computing Systems, • (1) Abstract concepts should reasonably be best ACM (2011), 3023-3032. represented by a symbolic TO, especially when there is 7. Hoefslinked, J. and Schiffeleers, S. Beat Blocks, no obvious perception equivalent to the concept; Tangible Sound Interface, • (2) Concrete concepts with several tunable https://www.youtube.com/watch?v=KoI8WH3F_Yk characteristics could be represented by totally iconic (2006), last access 24.02.2017. TOs or by a mix of iconic and symbolic, depending on 8. Holmquist, L. E., Redström, J. and Ljungstrand, P. Token-based access to digital information. In Proc. International Symposium on Handheld and Ubiquitous simulations. In Journal of educational psychology, Computing, Springer Berlin Heidelberg (1999), 234- (2006), 98(4), 902. 245. 14. Marshall, P. and Hornecker, E. Theories of Embodiment 9. Horn, M.S., Solovey, E.T., Crouser, R.J. and Jacob, R.J. in HCI. Sage (2013). Comparing the use of tangible and graphical 15. Oppl, S. Towards intuitive work modeling with a programming languages for informal science education. tangible collaboration interface approach. In Proc. In Proc. SIGCHI Conference on Human Factors in Enabling Technologies: Infrastructure for Collaborative Computing Systems, ACM(2009), 975-984. Enterprises, IEEE (2006), 400-405. 10. Jordà, S., Kaltenbrunner, M., Geiger, G. and Bencina, R. 16. Raffle, H., Vaucelle, C., Wang, R. and Ishii, H. The reacTable*, Jabberstamp: embedding sound and voice in traditional http://mtg.upf.edu/files/publications/9d0455- drawings. In Proc. of the 6th international conference ICMC2005-JordaKaltenbrunnerGeigerBencina.pdf on Interaction design and children. ACM (2007), 137- (1999), last access 02.24.2017. 144. 11. Jones, C.E. and Maquil, V. Towards Geospatial 17. Ryokai, K., Marti, S. and Ishii, H. I/O brush: drawing Tangible User Interfaces: An Observational User Study with everyday objects as ink. In Proc. SIGCHI Exploring Geospatial Interactions of the Novice. In conference on Human factors in computing systems, Geographical Information Systems Theory, Applications ACM (2004), 303-310. and Management, Springer International Publishing (2016), 104-123. 18. Tolar, T.D., Lederberg, A.R., Gokhale, S. and Tomasello, M. The development of the ability to 12. Knecht, K. and König, R. Augmented Urban Model: recognize the meaning of iconic signs. In Journal of Bridging the Gap Between Virtual and Physical Models Deaf Studies and Deaf Education, (2008), 13(2), to Support Urban Design. In Proc. 11th International pp.225-240. Conference on Construction Applications of Virtual Reality (2011), 142-152. 19. Underkoffler, J. and Ishii, H. Urp: a luminous-tangible workbench for urban planning and design. In Proc. 13. Lee, H., Plass, J.L. and Homer, B.D. Optimizing SIGCHI conference on Human Factors in Computing cognitive load for learning from computer-based science Systems, ACM (1999), 386-393.