Designing augmented interfaces for guided tours using multimedia sketches Margarita Martínez and Guadalupe Muñoz Departamento de Informática, Estadística y Telemática Universidad Rey Juan Carlos C/Tulipán s/n, Móstoles 28933 Madrid, Spain +34 91 664 74 53 {mmartinez, g.munoz}@escet.urjc.es ABSTRACT building reconstructing it from these few fragments. Also, One of the more interesting fields of application for an augmented reality system provides the user with augmented reality is cultural heritage. Designing additional data such as architectural information or augmented reality interfaces in this domain is difficult historical facts that are not available just looking at the because it involves, among other things, the selection and environment. interpretation of multidimensional information, the Though information on these elements can be obtained intervention of persons from multiple disciplines and the through different media, such as Web pages or TV design of the presentation and the interaction with the user. programs, most people likes to visit historic and We present a methodology that takes into account all these archaeological places in situ, combining entertainment with factors and that is supported with an innovative use of cultural enlightenment. One of the main advantages of the multimedia sketches. augmented reality paradigm for the cultural heritage Keywords domain comes from the facility to allow the mobility of the Design methodology; sketches; augmented reality; user while she is walking around the historical place. multimedia narration; mobile computing. On the other hand, if we compare the information given to INTRODUCTION a visitor of an historical place by a human guide with that An augmented reality (AR) system supplements the real provided by the new technologies, we can see that the last world with virtual objects that appear to coexist with the is more complete, and also with a more attractive format real ones [2]. Augmented reality has been applied to (graphics, sounds, three dimensional images). different fields, such as medicine, military training, There already exist prototypes of systems in which the entertainment, or engineering design, but one of the visitor receives this information through a head-mounted domains where it has the potential to be more useful is display or a tablet [19][23]. cultural heritage. The need to integrate virtual and real information implies a The application that we are addressing is that of providing change in the design methodology and design tools with the user with information while they are visiting an respect to other types of interactive systems that has been archaeological site or a monument, creating a personal little explored. In this paper we will present a methodology "touring guide machine". For many people visiting a for designing AR interfaces for cultural heritage that is monument or an archaeological place can be somewhat supported in its different phases by the use of multimedia disappointing, even when accompanied by a tour guide. sketches. Archaeological evidence for a building may consist of little The organization of the paper is as follows: In the first more than a few fragments of wall foundation. For the section we will state the requisites of such a methodology untrained eye it can be difficult to imagine the shape of a and will review the related work. Next, we will explain why to use sketches in supporting the design. In the third section, we will outline the main steps of the methodology we propose and how are multimedia sketches used in each one of them. Finally, we will present our conclusions and our plans for the future. 2. DESIGNING AR INTERFACES FOR GUIDED TOURS In our case we use sketches not only for storyboarding the The design of AR interfaces for guided tours is a complex presentation, but also in the first steps of the design, to process because consists of two kind of designs: 1) the specify requirements, compile and organize the design of the visit: selecting the information to show, information, and selecting and interpreting it. developing its historical interpretation, elaborating a Multimedia sketches also let us see the overlapping of journey, and building a narrative script, and 2) the design computer-generated objects on real ones without treating, of the user interface itself (how to show the information, in the first stages of the design, the problem of position what multimedia means to use, and how the user interacts detection, which is a very desirable characteristic for the with the application) design of any augmented reality application, and a Currently, there are methodologies to design some of these particularly difficult point in the case of AR interfaces for elements separately but none that integrates all of them. guided visits. Mitchell and Economou [18], discuss some methodological Of the various available tools for sketching multimedia approaches. Cohen [8] and McDonnel [17], while applications, we selected DEMAIS [4], mainly because of reviewing the role of the tour guide, discuss also the its capability for expressing dynamic behavior through content of the visit. In [5] Balaguer et al present the user designer's ink strokes and textual annotations. Originally centered design of a visit to an archaeological site by using intended for sketching user interaction elements and scenarios, storyboards, and a video prototype. In the synchronization of time-based information display, we use Archeoguide project [21], the design was based in use them also to help designers to navigate among the different cases [1]. Granollers et al [9] present how the location parts of the information. attribute is taken into account in the design process and the dialogue model. Apart from these advantages, DEMAIS helps us to map the multi-dimensionality of our domain (time, 3-space, cultural We have been working on such an integrated methodology aspects, historical features) into the 4-dimensional world and have stated in [16] the requirements we think it should (time, 3-space) of the historical place visited by the user. meet. Among them, we would emphasize that this methodology must take into account the multiple dimensions of the information to show (spatial, time- A METHODOLOGY SUPPORTED BY MULTIMEDIA dependent, social and cultural aspects, etc.), the most SKETCHES adequate approach to link them to elaborate a narration, The methodology took as starting point the design of film and the method employed to design the appearance of this movies [14], which are a kind of non-interactive information using the means provided by the multimedia multimedia narration. Later, we also took into account the and the augmented reality paradigms. interactivity of these narration for the case of augmented reality and the case of the cultural heritage domain. SKETCHES The methodology is divided in several steps, that are Such a methodology must we applied in close collaboration outlined below. with specialist in multiple domains, such as architects, Selection and interpretation of information archaeologists, or experts in cultural tourism and with end It consists on defining: users. For communication among them and the designers, The target of the application we began using paper storyboards, but quickly we started to use multimedia sketches. In this case the objective is more than presenting information, but instead to transmit a comprehension of the Actually, the case of guided tours is a special case of culture of the visited place. multimedia narration. According to [3] and [13], designers of multimedia applications prefer the development of The target user groups, and their requisites sketches over formal notations, authoring tool or other The typical users will be cultural tourists, but they could prototyping application early in the design process. also be students. They can also be divided in subgroups Sketches allow us to be imprecise, which is needed in the depending on age, previous knowledge, distance between early design when not all the user requirements have been their culture and that of the visited place, and so on. There completely gathered. It is also a fast method to have should be variations on the content, structure and something to check and refine [22]. navigation for each one of these subgroups. With respect to augmented reality interfaces, there is a lack The details of the information to be transmitted of tools for design, particularly for early design. The tool The selection consists in determining which points are presented in [7] covers early design, but it requires going to be visited and what information is going to be programming, which makes it inappropriate for transmitted. As it implies an interpretation of the multidisciplinary design. Other tools, such as [10] and [11], information, which usually means to make hypotheses from are more oriented to development. the known facts, it must be done in close collaboration with domain experts. Though in this type of AR interface the user is allowed to divert from the proposed path, most people prefer to be told Establishing the form of the medium an itinerary (or a small set of them) and a basically lineal First, the hardware where the application will run must be narration [12]. selected: the computer platform, the type of display, the In this phase, we use multimedia sketches mainly as a mean method and devices to detect user position, because many of communication between the different persons aspects of the design of the interface are dependent on intervening in the design, such as experts in cultural them. heritage or cultural tourism, multimedia designers, etc. and Next, the type of media to be used (sound, graphics, as a mean to organize the information obtained. Figure 1 images) for each information element is selected and these shows how an itinerary is designed, using the plane of a information elements are designed in detail. building as a base to lay out the paths and to mark elements of interest. DEMAIS layers are used to organize the As the multimedia contents have to be thoroughly multiple dimensions of information, and DEMAIS dynamic integrated with the physical objects that are seen by the behaviour elements, such as buttons, to relate elements, user, we use multimedia sketches to design the overlapping acting somewhat as hyperlinks (Figure 2). between real and generated elements. In this case, the base for the sketches is photographs and videos of the place When the place has suffered changes along the history, (figure 4). Moreover, they can be used to see the degree of multidimensional diagrams (figure 3) can be used to show visibility of the real elements from each position. these changes. Figure 3. Showing changes along time Figure 1: Designing an itinerary Figure 4: Overlapping real and computer generated interface elements Figure 2. Using sketches to organize and link information In the future, we plan to use DEMAIS also for early prototyping, in order to experiment and evaluate different interaction techniques and forms of presentation. We also want to refine the steps of our methodology, and particularly the transition from multimedia storyboards to production, that is not trivial in this case because presentation must be adapted in function of the position and point of view of the user in each moment. REFERENCES 1. ARCHEOGUIDE Consortium. ARCHEOGUIDE Aplication Scenarios. Deliverable D04. http://archeoguide.intranet.gr/papers/D04-V01- ICOM_Final.doc Figure 5: Designing temporal aspects of the visit 2. Azuma, R.; Baillot, Y.; Behringer, R.; Feiner, S.; Julier, Videos also help to design the temporal aspects of the visit, S.; MacIntyre, B. “Recent Advances in Augmented pacing the oral contents with the time it takes the user to Reality”. In IEEE Computer Graphics and Applications cover a stretch of the itinerary (figure 5). 21, 6 (Nov/Dec 2001), 34-47. As the objective is not only information transmission, but 3. Bailey, B.P. and J.A. Konstan. Are Informal Tools to make the user enjoy the experience of the visit, Better? Comparing DEMAIS, Pencil and Paper, and designing through sketches allow to experiment easily and Authorware for Early Multimedia Design. Proceedings with a low cost with different design options and even of the ACM Conference on Human Factors in artistic expression forms. There is not much experience Computing Systems, 2003, pp. 313-320. with this type of application, so it is particularly important 4. Bailey, B.P.; Konstan, J.A. and Carlis, J. V. “Supporting to evaluate early the different aspects of the design, with Multimedia Designers: Towards More Effective Design the participation of end users. Tools”. Proceedings Multimedia Modeling, 2001. pp. Finally, elements for explicit interaction with the user can 241-250. also be designed by using the dynamic capabilities of 5. Balaguer, A.; Lorés, J.; Junyent, E. and Ferré, G. Demais. "Scenario based design of augmented reality systems Designing the application software. applied to cultural heritage", Proceedings Panhellenic It consists on selecting the software platform and designing Conference on Human Computer Interaction, 2001, the software architecture, the mechanism for information Typorama pub. personalization, etc. 6. Caloini, A., D. Taguchi, K. Yanoo, and E. Tanaka. Prototyping and evaluation Script-free Scenario Authoring in MediaDesc. Though with sketches we can experiment and evaluate the Proceedings ACM Multimedia, 1998. main aspects of the design, there are other factors, such as 7. Ceiger, C., Paelke, V. Reimann, C. Rosenbach, W. And synchronization and integration [20], that affect usability Stoecklein, Testable Design Representations for Mobile and whose evaluation requires a working prototype and a Augmented Reality Authoring, Proceedings ISMAR'02: visit to the place. International Symposium on Mixed and Augmented CONCLUSIONS AND FUTURE WORK Reality, IEEE, 2002. The design of augmented reality interfaces for visiting 8. Cohen, E. (1985): "The Tourist Guide. The Origins, cultural heritage places is a complex process, because it Structure and Dynamics of a Role". Annals of Tourism involves both the design of the visit and the design of the Research 12 (1), pp. 5-29, 1985. multimedia information presentation. It is also a 9. Granollers, T.; Lorés, J.; Raimat, G.; Junyent, E.; multidisciplinary process, because it needs the intervention Tartera, E. Design of augmented reality systems in in the design of persons with different backgrounds, such archaeological sites". Enter the past. Viena 8-12 abril, as multimedia designers, usability specialists, 2003. archaeologists or architects. 10. Grimm P., Haller M., Paelke V., Reinhold S., Reimann In this paper we have presented the steps of a methodology C., Zauner J., AMIRE - Authoring Mixed Reality, First for solving these problems, and the use of dynamic IEEE International Augmented Reality Toolkit multimedia sketches in supporting the different phases of Workshop, September, 2002. Darmstadt, Germany the design, from the selection and interpretation of cultural information to the layout of the presentation. 11. Haringer, M. and Regenbrecht, H.T. A Pragmatic Approach to Augmented Reality Authoring. Proceedings ISMAR'02: International Symposium on 18. Mitchell , W.L. and Economou, , D. "Methodological Mixed and Augmented Reality, IEEE, 2002. Approaches to Developing Content for the Cultural 12. Karat, M-C.; Pinhanewz, C.; Karat, J.; Arora, R. and Grid", MDA Information, MDA, Vol. 5, No. 1, Vergo, J.: Less Clicking, More Watching: Results of the December 2000, pp. 5-10. Interactive Design and Evaluation of Entertaining Web 19. Newman, J.; Ingram, D.; Hopper, A. (2001): Experiences. INTERACT'01: Eight IFIP Tc.13 "Augmented Reality in a Wide Area Sentient Conference on Human-Computer Interaction. pp. 455- Environment" Second IEEE and ACM International 463. IOS Press. Symposium on Augmented Reality (ISAR 2001), 13. Landay, J.A. and B.A. Myers. Interactive Sketching October 2001, New York. for the Early Stages of User Interface Design. 20. Trevisan, D., Vanderdonckt, J., Macq, B. Analyzing Conference Proceedings on Human Factors in Interaction in Augmented Reality Computing Systems, 1995. Systems, Proceedings of ACM Multimedia'2002 14. Manovich, L. "What Is Digital Cinema?": The Digital International Workshop on Immersive Dialectic: New Essays on New Media. Ed. Peter Telepresence ITP'2002 (Juan Les Pins, 6 December Lunenfeld. Cambridge: The MIT Press, 1999. pp. 172- 2002), G. Pingali, R. Jain (eds.), ACM Press, New 192. York, 2002, pp. 56-59. 15. Martinez, P. “Digital realities and archeology: a 21. Vlahakis, V; Ioannidis, N; Karigiannis, J.; Tsotros, M.; difficult relationship or a fruitful marriage?”, 2001 Gounaris, M.; Stricker, D.; Gleue, T.; Daehne, P.; ACM Conference on Virtual reality, archeology and Almeida, L. (2002): “ Archeoguide: An Augmented cultural heritage. pp. 9-16. Reality Guide for Archaeological Sites”. IEEE Computer Graphics and Applications, Vol. 22, no. 5, 16. Martinez, M and Muñoz, G., Requisitos para una pp. 52-60. metodología de diseño de sistemas de visita guiada basados en computación ubicua y realidad aumentada, I 22. Wickens, C. D., Gordon, S. E., and Liu, Y. An Encuentro Internacional de Informática Aplicada a la Introduction to Human Factors Engineering. Addison- Investigación y Gestión Arqueológicas, Córdoba, mayo Wesley (Longman Imprint): New York, NY, 1998. 2003. 23. Zancanaro, M.; Stock, O.; Alfaro, I. (2003): “Using 17. McDonnel, I. (2001): "The role of the tour guide in Cinematic Techniques in a Multimedia Museum tranferring cultural understanding", Working Paper No. Guide”. Museums and the Web 2003, 19-22 March 3, School of Leisure, Sport and Tourism, University of 2003, Charlotte, NC. http://tcc.itc.it/people Technology, http://www.business.uts.edu.au/leisure/research/WP3_ McDonnell.pdf. Sidney.