Reflections about Symbolic vs. Iconic Representations in TUIs

Abstract

Currently designed Tangible User Interfaces (TUIs) propose both iconic and symbolic tangible objects (TO). Since iconic TOs should enable to interact more naturally like in the real world and, hypothetically, require less learning time than symbolic TOs, some questions arise: Why do symbolic TOs exist? When to use iconic or symbolic representation in TOs? This paper discusses these questions and makes some assumptions on the abstraction of concepts, the function of TOs (container, token or tool) and the context influencing the design choices of TOs.

-

• Icon/iconic: a mode in which the signifier is perceived as resembling or imitating the signified (recognizably looking, sounding, feeling, tasting or smelling like it) being similar in possessing some of its qualities: e.g. a portrait, a cartoon, a scale-model, onomatopoeia, metaphors, ’realistic’ sounds in ’programme music’, sound effects in radio drama, a dubbed film soundtrack, imitative gestures; A third mode is defined by Chandler (index) but is not addressed in this paper.

Semiotics of TUIs is not a well-documented subject. For instance, in the ACM digital library, only 3 papers can be found with the words ‘iconic’ and ‘symbolic’ and ‘tangible’, 22 papers with the words ‘iconic’ and ‘tangible’ and 38 papers with the words ‘symbolic’ and ‘tangible’. In TEI conference proceeding (Conference on Tangible, Embedded and Embodied Interactions) only one paper deals with these three keywords [ 1 ].

The objective of this paper is to exchange reflections about symbolic and iconic representations in TUIs to discuss them and, particularly, to propose some hypotheses on when to use one or the other mode. In the following section, some examples of iconic and symbolic Tangible User Interface Objects (TOs) are exposed. Then, a discussion on the difficulties to design an iconic TUI is opened.

STATE OF THE ART Symbolic and iconic representations of digital data and tools are used in TUIs. This section presents some examples of both of them.

Examples of symbolic TUIs Symbolic TUIs often use boxes and cylinders as TOs to manipulate information and as actuators in different domains.

1 3 2 4

For instance, musical applications like the ReacTable [ 10 ], where the user turns or connects cubes, squares, discs, etc. to combine different sound items like synthesizers, effects and samples to compose a music flow, or the BeatBlocks [ 7 ] where the user places blocks representing sounds in a container to compose music. Urban planning also uses boxes to represent the buildings’ location and a disc to modify time in order to display the shadow of buildings on the map [ 12 ] or to manipulate maps for logistics in a GIS (Geographical Information System) [ 11 ] by activating different layers on a base map with squares, manipulating the map with a disc to pan or zoom and display more information about a specific point (using a triangular pointer).

Examples of iconic TOs Iconic tangible objects, enabling to interact with digital data, look like objects in the real world.

1 2 3 4 5

Some examples of iconic TOs are the tangible drawing tool in the form of a paintbrush developed by the MIT which captures any real world color and is used to paint on a display [ 17 ], or the CapTUI (ruler, protractor and set square) of Blagojevic and Plimmer [ 2 ]. For music and sound managing we can found GuitarHero® [ 5 ] which enables one to play guitar by pushing buttons on a quite real looking guitar and the JabberStamp [ 16 ] which proposes a recorder (symbolized by a microphone) and a speaker to augment drawings with sounds. Iconic TOs are also used in the medical domain with Teegi (tangible EEG Interface) that shows a patient’s brain activity in real-time and enables the user to interact with [ 4 ].

REFLECTIONS Why do symbolic TOs exist? TUIs seek to embody interactions, to give the sensation of interacting easily as in the real world, to manipulate digital data, to make the boundary between the real world and the digital world narrower [ 14 ]. In this perspective, iconic TOs should allow to interact more naturally like in the real world. Furthermore, hypothetically, the time needed to learn how to use TOs should be less with iconic than with symbolic TOs. Indeed, TOs’ functions are better understood when they are iconic than symbolic [ 1 ] and users are more involved [ 9 ]. Furthermore, the comprehension and transfer in learning experiences are better with iconic than symbolic representations to solve problems of low and high complexity. In addition, for users with low prior knowledge, it is better to use the iconic representation [ 13 ]. This is supported by the fact that icons are closer to perception patterns whereas symbols are related to their referent through higher level concepts. This induces two different cognitive processes [ 18 ].

Then, iconic TOs should be used more often than symbolic TOs. So, why are there symbolic TOs? In the following some hypotheses or topic discussions are proposed to begin to address this question.

Finding the right representation is not so easy What is the right iconic representation to delete a data item in a TUI? This type of question is asked at every new interaction mode. As shown for gestural interaction, the answer is not obvious (e.g., for the question of which gesture should be used to drag a data on a digital tabletop, different answers are given [ 6 ]). This shows that no symbol is totally universal. This would certainly be the same for the determination of TOs’ manipulations by users. When cultural differences, expertise, sensitivity and innate knowledge are taken into account, this difference is more important.

Manage characteristics of the TO A TO is not always only a marker of the presence or absence of data in the digital model, but also the TO gives access to the characteristics of data represented by the object. For instance, in an urban planner, a manipulated TO could be a parking lot that should be placed in the district. The system could enable users to characterize the parking lots on several dimensions For instance: the type of parking (aerial, ground or underground), the accessibility (private, public or public with lots for people with disabilities) or the number of places. Different design approaches can be used: it is possible to give as much TO for the same concept that they have crossed characteristics (in this example 3x3xn possibilities exist). But an exponential need of TOs will be observed. Users could be confused in front of such a quantity of TOs. Or it is possible to enable users to characterize the TO by manipulating it. E.g., by using some bricks to add on the parking lot to increase the number of places, and the color of these bricks could signify that lots are private, public or for disabled people. Find the right iconic representation for the TO and its characteristics is not tricky. Mixing symbolic and iconic could be an option. Represent abstract concepts Sometimes, abstract concepts should also be represented, like in modeling [ 15 ]. Modeling languages have tried to represent concepts more appropriately, but for such abstract concepts, finding an iconic representation is not obvious or even possible.

Representation is not the key subject of experiments As shown by the low number of referenced publications in ACM digital library on iconic, symbolic and tangible, the representation of TOs, in these terms, doesn’t seem to be a trendy research question. Indeed the focus of research was more on the validation of this new way of interaction and paradigm. The question should be raised in following years. Another question arises: when to use iconic or symbolic representation in TOs? When to use iconic or symbolic representation in TOs? To answer this question, taxonomy of TOs should be done. Some inputs can be found in Holmquist et al [ 8 ]. Three types of TOs are defined:

Containers: contain digital information, like a drive. They are generic, that means “the physical properties of a container do not reflect the nature of the digital information it is associated with”.

Tokens: represent particular digital information. “Tokens are objects that physically resemble the information they represent in some way”.

Tools: are functions to manipulate the digital model, for instance zoom, pan, rotate and magnify TOs. In regard of this first distinction between TOs: • • • • • • • • • • • the number of TOs available (give only a reasonable number of TOs to handle) and of cost limits; (3) A mix of symbolic and iconic TOs could be a good compromise; (4) Use of symbolic and iconic could depend on the kind of TO in regard of a TOs’ taxonomy. For instance, containers should be symbolic, tokens should be as iconic as possible and tools could be both; (5) A same concept/data should be represented by an iconic TO in a certain context and by a symbolic TO in another context.

This reflection raises other questions: how to design a usable iconic TO? What is the taxonomy of TOs? When should symbolic and iconic be used? Could a mix of them be a good compromise? These are questions that we will address in future works.

ACKNOWLEDGMENTS The authors acknowledge all their colleagues who are working on this question at LIST, who nourished this reflection, in particular Valérie Maquil, Séverine Mignon and Annie Guerriero. Containers should be symbolic, as they are generic. Tokens should be as iconic as possible, in regard of other constraints (abstract vs. concrete data, etc.) Tools could be iconic or symbolic.

The previous question gives some directions: an abstract concept should be represented by a symbolic TO. A concrete concept with several tunable characteristics could be represented by all iconic TOs or by a mix of iconic and symbolic.

CONCLUSION At first sight, it seems to be natural to have very iconic TOs. But, as observed, a lot of symbolic TOs exist. This can be explained by several hypotheses: (1) Abstract concepts should reasonably be best represented by a symbolic TO, especially when there is no obvious perception equivalent to the concept; (2) Concrete concepts with several tunable characteristics could be represented by totally iconic TOs or by a mix of iconic and symbolic, depending on

References

1. Bakker , S. , Vorstenbosch , D., van den Hoven, E., Hollemans , G. and Bergman , T. Tangible interaction in tabletop games: studying iconic and symbolic play pieces . In Proc. international conference on Advances in computer entertainment technology , ACM ( 2007 ), 163 - 170 .

2. Blagojevic , R. and Plimmer , B. CapTUI: geometric drawing with tangibles on a capacitive multi-touch display . In Proc. IFIP Conference on Human-Computer Interaction , Springer Berlin Heidelberg ( 2013 ), 511 - 528 .

3. Chandler , D. Semiotics for beginners . ( 1994 ).

4. Frey , J. , Gervais , R. , Fleck , S. , Lotte , F. and Hachet , M. , Teegi: tangible EEG interface . In Proc. 27th annual ACM symposium on User interface software and technology , ACM ( 2014 ), 301 - 308 .

5. Guitar Hero®, Activision Publishing Inc, https://www.guitarhero.com/, last access 03.15 . 2017 .

6. Hinrichs , U. and Carpendale , S. , 2011 , May. Gestures in the wild: studying multi-touch gesture sequences on interactive tabletop exhibits . In Proc. of the SIGCHI Conference on Human Factors in Computing Systems , ACM ( 2011 ), 3023 - 3032 .

7. Hoefslinked , J. and Schiffeleers , S. Beat Blocks , Tangible Sound Interface, https://www.youtube.com/watch?v=KoI8WH3F_ Yk ( 2006 ), last access 24.02 . 2017 .

8. Holmquist , L. E. , Redström , J. and Ljungstrand , P. Token-based access to digital information . In Proc. International Symposium on Handheld and Ubiquitous Computing , Springer Berlin Heidelberg ( 1999 ), 234 - 245 .

9. Horn , M.S. , Solovey , E.T. , Crouser , R.J. and Jacob , R.J. Comparing the use of tangible and graphical programming languages for informal science education . In Proc. SIGCHI Conference on Human Factors in Computing Systems , ACM( 2009 ), 975 - 984 .

10. Jordà , S. , Kaltenbrunner , M. , Geiger , G. and Bencina , R. The reacTable* , http://mtg.upf.edu/files/publications/9d0455- ICMC2005 -JordaKaltenbrunnerGeigerBencina.pdf ( 1999 ), last access 02.24 . 2017 .

11. Jones , C.E. and Maquil , V. Towards Geospatial Tangible User Interfaces: An Observational User Study Exploring Geospatial Interactions of the Novice . In Geographical Information Systems Theory, Applications and Management , Springer International Publishing ( 2016 ), 104 - 123 .

12. Knecht , K. and König , R. Augmented Urban Model: Bridging the Gap Between Virtual and Physical Models to Support Urban Design . In Proc. 11th International Conference on Construction Applications of Virtual Reality ( 2011 ), 142 - 152 .

13. Lee , H. , Plass , J.L. and Homer , B.D. Optimizing cognitive load for learning from computer-based science simulations . In Journal of educational psychology , ( 2006 ), 98 ( 4 ), 902 .

14. Marshall , P. and Hornecker , E. Theories of Embodiment in HCI . Sage ( 2013 ).

15. Oppl , S. Towards intuitive work modeling with a tangible collaboration interface approach . In Proc. Enabling Technologies: Infrastructure for Collaborative Enterprises , IEEE ( 2006 ), 400 - 405 .

16. Raffle , H. , Vaucelle , C. , Wang , R. and Ishii , H. Jabberstamp: embedding sound and voice in traditional drawings . In Proc. of the 6th international conference on Interaction design and children . ACM ( 2007 ), 137 - 144 .

17. Ryokai , K. , Marti , S. and Ishii , H. I/O brush: drawing with everyday objects as ink . In Proc. SIGCHI conference on Human factors in computing systems , ACM ( 2004 ), 303 - 310 .

18. Tolar , T.D., Lederberg , A.R. , Gokhale , S. and Tomasello , M. The development of the ability to recognize the meaning of iconic signs . In Journal of Deaf Studies and Deaf Education , ( 2008 ), 13 ( 2 ), pp. 225 - 240 .

19. Underkoffler , J. and Ishii , H. Urp: a luminous-tangible workbench for urban planning and design . In Proc. SIGCHI conference on Human Factors in Computing Systems , ACM ( 1999 ), 386 - 393 .