Obstacles and Perspectives for Evaluating Mixed Reality Systems Usability. Cédric Bach Dominique L. Scapin INRIA INRIA B.P. 105 Domaine de Voluceau B.P. 105 Domaine de Voluceau 78153, Le Chesnay, France 78153, Le Chesnay, France +33 1 39 63 51 09 +33 1 39 63 55 07 cedric.bach@inria.fr dominique.scapin@inria.fr ABSTRACT mixed reality systems, from a user-centred perspective, and The goal of this paper is to survey the main issues with the to discuss potential research avenues. ergonomic evaluation of MRS (Mixed Reality Systems) First, one needs to discuss definitions : MRS are viewed as and to stimulate discussions for future research. A first a subset of VE, with an ergonomic perspective. point concerns definitions and specificities of MRS within Secondly, through a brief account of VE and MRS studies, the « reality / virtuality » continuum, and the incorporation we highlight a few items explaining the lack of current of user issues. Another point concerns the combinatory knowledge on the usability of such systems, mainly related character of the ergonomic knowledge to be applied to to their novelty. MRS entities (reality, « GUIs », « VR », and « MR » spe- cific). A major issue concerns ergonomic evaluation meth- Then, the paper mentions briefly a number of evaluation ods, their current state, their advantages and drawbacks, methods that can be applied to MRS, and focuses on the particularly for user testing. Finally, the discussion points specific methodological challenges with user testing. at various items which may be part of a future research Finally, the discussion offers several items of interest for a agenda, such as the need for more usability data, for generic common research agenda in the area of MRS ergonomics. and well controlled experiments; for common testing plat- forms, for shared recommendations data bases, for design DEFINITIONS and SCOPE and assessment of inspection methods, for common task For MRS, there is no fully agreed-upon definition, so far, taxonomies and for common models of MR entities and but a set of features, e.g., as the ones mentioned in the Call situations, etc.; all of this possibly leading to increased For Papers for this (IUI / CADUI-associated) Workshop: “ knowledge based on shared benchmarks. … integration of the physical and digital worlds in a smooth and usable way. This fusion involves the design Keywords and development of "mixed reality systems", including Mixed reality, virtual environments, ergonomics, usability, augmented reality, augmented virtuality, augmented video, evaluation methods, user testing, inspection methods, er- and tangible systems …”. We definitely agree that: “The gonomic recommendations, research agenda, user-centered diversity of terms used to denote these systems is evidence approaches, interaction models, task models. both of the amount of research activity in the field and the lack of a common conceptual framework for that activity”. INTRODUCTION Obviously, one of the goals of this Workshop should be to Virtual Environments (VE) are being developed fast and progress on definitions and on a conceptual framework. widely, in various contexts (e.g., training, data visualisation, computer-aided design, tourism, art, games, For VE, there are numerous definitions. Most often, they etc.). are techno-centred. A definition that we use for VE is the Mixed Reality Systems (MRS) follow the same path. For one for VR (Virtual Reality) from Loeffler & Anderson both types of environments, just as it was the case for [14] : “Virtual Reality is a three-dimensional, computer- GUIs and for the Web, a large utilisation will depend on its generated, simulated environment that is rendered in real usability. time according to the behaviour of the user.” Facing the question of establishing a research agenda for However, this definition should to be completed on two contributing to more usable MRS, this paper attempts to points: draw a limited picture of the current state of knowledge on - VE can be multi-users (for the evaluation of collaborative VE, see for instance [25]) LEAVE BLANK THE LAST 2.5 cm (1”) OF THE LEFT - VE can be described along a « Real-Mixed-Virtual » COLUMN ON THE FIRST PAGE FOR THE continuum, as described by Milgram [16]. COPYRIGHT NOTICE. That continuum includes in fact a large spectrum of 3D computer-based interactive situations. On that spectrum, a number of terms used in the literature can be located, such as Virtual Reality, Mixed Reality, Augmented Reality, performance, information presentation, commands Desktop or Immersive Environments, Augmented and controls, etc.). Virtuality, etc… it is even possible to incorporate within • Virtual environments are concerned as well by that continuum 3D CAD or VRML objects displayed on cognitive ergonomics, but also by physical « classical » computers, which makes them belong to VR, ergonomics (e.g., in terms of VE behaviour, even though very loosely. presence, cyber sickness, etc). The first point here is a matter of joining the techno-centred • Fusion of the previous elements, which and the user-centred perspectives. The second point deals corresponds to one specificity of MRS: the with the combinatory nature of MRS ergonomic knowledge appropriate correspondence between the various (real, « GUI », « VR », and « MR » specific). real or virtual elements constituting MRS. People might think that there is an enormous difference This problem, also called « continuity », between a 3D object to be interacted with in a regular GUI characterizes the perceptual and cognitive fluidity, environment, and that same object presented in a CAVE, from the users’ point of view, between the real using a 3D mouse. That difference is certainly there, but world and the virtual world. According to Nigay only from a technical point of view, not necessarily from a et al. [17], “The perceptual continuity is verified if user’s point of view. the user directly and smoothly perceives the With an ergonomic perspective, these two situations must different representations of a given concept. be considered through their capacity to support users in Cognitive continuity is verified if the cognitive achieving specified task goals, with efficiency, effectiveness processes that are involved in the interpretation and satisfaction. In some cases, an immersive situation can of the different perceived representations lead to a be beneficial (e.g., learning driving / control operations on unique interpretation of the concept resulting a high-speed train, which involves physical simulation of from the combination of the interpreted perceived the access to the driving instruments), while it may be less representations.”. appropriate in others (e.g., learning specific procedures for Such a concept is a challenge for MRS design that that same train, which involve more cognitive sequence requires optimal matching between various learning steps) where regular GUIs, or even paper instruc- features in terms of presentation, coding, meaning, tions may be more appropriate. and behavior. From an ergonomic point of view, one could state In other words, depending on the task at hand, the useful- that it is not only an issue with interface ness of the support required may be different : in kinetics consistency (e.g., providing information learning, immersion may make sense ; while in cognitive presentation as well as user initiated interactions learning, it may not be as helpful. with the same characteristics across operations and The same point can be made for MRS which are across applications), but also a complex constituted of both real and virtual elements within the compatibility issue, i.e., matching the various same 3D interactive environment. As they can be located MRS individual elements with their referent, but roughly at the midpoint of the Milgram continuum, they also matching all related MRS elements which can be considered as a particular form of VE. However, each other, depending on the tasks and context. their specificity is to stage a form of fusion between real ERGONOMIC KNOWLEDGE ON MRS and virtual worlds. Previous history of HCI concerning ergonomics knowledge Even if MRS can be divided into Augmented Reality and is probably repeating itself : MRS novelty and lack of er- Augmented Virtuality [17] depending on the exact physical gonomic knowledge. or computer-based nature of the tasks and interactions With new technologies, usability concerns start with gen- involved, they relate to combinations of knowledge from eral questions, hypotheses, debates ; then questions are various domains of specialization of ergonomics : sorted and put to the test through experiments ; theories are • Real objects, in the area of physical ergonomics, offered ; data on usability grows rapidly as the technology concerned with human anatomical, anthropomet- becomes widely available ; data is then organized, made ric, physiological and biomechanical characteris- available through manuals, guides ; finally, style guides, tics (postures, materials handling, repetitive architectures, design and evaluation methods are offered, movements, musculoskeletal disorders, workplace tested, compared. layout, safety and health); in the area of ergonom- This has been proven true for GUIS, the Web, ... it should ics of every-day products and consumer products; be the same for VE and MRS. However, there may be ways in some cases, in the area of cognitive ergonomics to accelerate the process by doing user testing early so that and organizational ergonomics. usability knowledge is gained rapidly, rather than having • Classical interfaces (GUIs) mainly in the area of simply the technology perfected without user concern. cognitive ergonomics, concerned with mental Also, as it has been verified that, when moving from GUIs processes, such as perception, memory, reasoning, to the Web, one should not « re-invent the wheel », but (e.g., mental workload, decision-making, skilled apply as much as possible sound ergonomic knowledge that can be transferred to novel environments (information of recommendations available, to the fact that organisation, consistency, level of feedback, etc.). « classical » ergonomic recommendations have not Implementation and usage of VE and MRS is indeed very yet been applied or extended to VE. In addition, recent as it started in the early nineties and only grows at a such recommendations are not presented in a fast pace since three or four years. common unitary format, but often as experimental or « best practice » results ; see for instance That novelty explains partly why the currently available Gabbard [7] [8]; Kaur [12]; Stanney[21] ; generally ergonomic knowledge is relatively scarce, compared to the such documents are organized according to major number of issues that need to be tackled in such complex categories of VE objects. Also, based on an environments. Such environments are relatively unstable in extensive literature review, a set of 170 their implementation and usage ; lots of technical problems recommendations dedicated to VE have been still need to be solved. extracted under a generic format and organized However, some actions can be envisioned to better incorpo- according to usability dimensions [1]. rate a user-centred approach : • Along those lines, with the goal of defining a • The recommendation above to not « re-invent the structured ergonomic inspection method, an wheel », is particularly true for MRS as their adaptation of the Ergonomic Criteria (E. C.) has components are mixed, therefore, ergonomics been proposed and assessed in terms of intrinsic knowledge needs to be gained for the novel, spe- validity [2]. First of all the application of the E. cific MRS issues (such as « continuity »), but cur- C. inspection method requires expertise in VE and rently available ergonomics knowledge should be training on the 20 dimensions covered by these applied to its non-specific elements (e.g., real ob- E. C.. Also, the method needs further evaluation jects, 2D displays, etc.). The amount for VE and extensions to MRS. of knowledge (empirical results, recommenda- A method dedicated to VR, based on the cognitive tions, standards, etc.) in ergonomics about real ob- walkthrough method, has also been designed by Sutcliffe jects, « classical » interfaces is of course enor- [22]. This method is based on a theory of interaction mous, and needs to be carefully investigated in (Norman [18]). This walkthrough analysis method uses terms of applicability depending on context and three models (first on goal-oriented task actions, second on tasks. Also, concerning ergonomic evaluation, a exploration and navigation in virtual worlds, third on number of already available methods should be interaction in response to system initiative) derived from looked at in terms of their applicability and needs the theory. Each stage of the model is associated with for adaptation to the specifics of MRS. generic design properties. The evaluation method consists • Another recommendation above was to incorporate of a checklist of questions using the properties and ergonomic concerns early. That is usually done following the steps of the method. That method could through user testing (with or without hypotheses, possibly be extended to MRS. on one or several contexts, on one system, or There are also methods, mainly for VE, based on through the comparison of several systems, etc.). recommendations, using computer-based support, such as : That is where a number of methodological prob- lems arise. • I-DOVE (Interactive tool for development of In order to identify the obstacles and perspectives Virtual Environments) [11]. The goal of this in methodological terms, the next two sections prototype, currently being developed as a large- deal first with ergonomics evaluation methods and scale web-based application based on several sets secondly with MRS-specific issues when conduct- of recommendations, is to offer context specific ing user testing. guidance for VE development, and alternative ways of searching and browsing, distinguishing ERGONOMIC EVALUATION METHODS user categories. The initial prototype was based on In the context of MRS, which usability evaluation methods users interviews and was later evaluated by expert are available, and how can such environments be evaluation. appropriately evaluated? • Another tool is MAUVE [21]; also developed as a In the literature, there is no usability method yet website, it incorporates design guidelines specifically designed for MRS, except notations such as according to several VE categories such as AZUR++, a notation for describing, and reasoning about navigation, object manipulation, input, output and the design of mobile mixed systems [6]. so on (based on Gabbard’s [8] taxonomy). A There has been already some research on how to guide VE multi-criteria usability matrix is the support for design with usability considerations (e.g., in [26]), to organizing and retrieving recommendations. The consolidate usability dimensions and to design inspection evaluation process is supported in two steps: methods, but much work is needed to extend their scope to “traditional” heuristics stage, and prioritization of MRS, to assess them and to compare them. usability attributes. That capacity of tailoring the evaluation may be interesting for evaluators with • Methods based on heuristics or recommendations are difficult to carry out due to the limited amount different points of view or organizational goals. problems already experienced with a few cases of The tool has not been evaluated yet. user testing of VE systems (see next section). • The last tool, which we know of, is a hypertext- METHODOLOGICAL DIFFICULTIES WITH USER based prototype developed by Kaur [13]. This tool TESTING present 45 generic design properties that specify User testing is certainly the preferred method to be used, the necessary support from the system for particularly in order to alleviate the current lack of available “successful” VE interfaces. Like the previous one, usability data. However, many methodological problems no usability evaluation plan was integrated during require solutions. the development of the prototype. First of all, let us use a metaphor both historical and aero- There is also the question of adapting, at least partly, to nautical. Looking back over a century ago, let us imagine MRS some of the « classical » ergonomics methods. Such that the following question was asked at that time: « which methods are too numerous to be all discussed here (see for flying machine constitutes the best way to move rapidly in instance, [3], [4], [9]). However, one can mention three the air from one point to another : the blimp or the air- categories of methods that are general enough in their planes ? ». In those times, the airplanes trials were just approach to be good candidates for MRS : questionnaires/ starting ; it would have been difficult to find users (i.e., interviews, inspection methods and user testing. A number pilots) able to fly (usually the few pilots were the airplanes of questions must be solved in order to apply them to designers themselves) ; the underlying technology was only MRS. emerging, and often planes had numerous technical prob- • Questionnaire and Interviews allow gathering lems or simply just crashed. If at that time one would have subjective data, often quite important to evaluate conducted comparative performance testing, there is no visual appeal, preferences, aesthetics, missing doubt that the blimp would have won over the airplanes ! functionalities, and also very useful as a means to However, by now, everyone would agree that airplanes are compare or cross-reference performance data. Such better that blimps for long distance transport of passengers. methods are certainly interesting candidates for Making the parallel with HCI technology, « classical being applied to MRS, providing specific lead HCI » would be our blimps and MRS would be our questions for interviews and questionnaire items planes. In several ways, MRS are at the same point as are tailored and validated for such environments. th. planes in early 20 century : For questionnaires, Kalawsky [10] has designed • There are very few experts that can operate them. VRUSE for measuring usability of a VR application in terms of users’ attitude and • The characteristics of tasks that can be performed perception. The 100 questionnaire items are in such environments are still quite vague. organized under 10 usability factors: functionality, • Learning by trying is still the rule. input, output, user guidance, consistency, flexibility, simulation fidelity, error correction, • There are many problems to solve in order to presence and overall system usability. This « fly » those environments : on the technical questionnaire has been tested in terms of aspects (e.g., computer graphics) ; on the reliability (Cronbach's alpha > 0.9). interaction aspects (e.g., devices and modalities) ; and on their usability. • Inspection methods are also good candidates for supporting MRS evaluations. The problem there That state of affairs can explain why often results are is the need for more data (particularly recommen- disappointing [19] when classical graphics environments dations) and more data organization in order to are compared to VE or MRS. cover the range of ergonomic problems related to Before conducting user tests or to compare interactive situa- MRS. Issues regarding recommendations identifi- tions, it is best to : cation and structuring into dimensions need to be • First make sure that the environments are already looked at carefully; the history of HCI has shown sufficiently well-designed so that well-known us- so far that such dimensions can be established and ability deficiencies are removed. This is a sound efficiently contribute to evaluation of GUIs, the precaution helping to focus user testing on the real web, currently with VR ... but specificities of MR « new » usability problems and the test compari- need to be taken into consideration (e.g., task sons on the real usability hypotheses, rather that compatibility, devices consistency for visualiza- obscuring the picture with unwanted usability tion, documents compatibility, innovative help problems, both for the user and for the experimen- systems, etc.). tal data analyst. This can be achieved by careful • User Testing has been the major method in assessment of the design, for instance through ap- ergonomics and will probably remain as important plying available inspection methods and sets of for MRS. However, in order to apply that method recommendations. to MRS, a number of methodological problems • To alleviate as much as possible the various need to be tackled, including of course, the limitations of user testing due to the specificities of VE and MRS. Some of those limitations are and ways to operate the various parts of the environment. described in the next three paragraphs. This also leads to questions on how best to describe the tested situation without coaching too much the subjects if Limitations related to the physical environment one wants to study their “intuitive” performance or One of the major differences between MRS and traditional preferences. interfaces concerns their physical environment. MRS require a more sophisticated environment : the users rarely Along the same line, an additional difficulty is simply that just sit in front of their computer ; they move from one it may be difficult to explain the complexities of MRS, place to another, they talk, they move parts of their body in considering that written instruction may not be sufficient order to interact. beforehand, and impossible during the test when several entry or display devices are used together (the user cannot This raises several problems in evaluation situations in be using an eye-tracker, a data glove, watch a large dis- which it is useful to prepare the experiments in order to play, and at the same time walk through a leaflet of task avoid some disturbances, for instance, if the MRS instructions). In addition, when needed, where to make application area in an office, it is necessary to limit the some help system available ? interaction zones in order to avoid collision with the chairs, tables, or cables ; other constraints can complicate the In some cases, as current technology is unable to support situations, for instance, interaction devices can be an fully and consistently novel interaction paradigms, there is obstacle to data collection : use of stereoscopy does not no way to test those new ideas unless using « Wizard of allow collecting data on video or on a monoscopic Oz » techniques which require trained specialists and spe- monitor, in which case, the evaluator must access directly cific, carefully balanced experimental design. to the stereoscopic data using a tracker or some parallel In other cases, the techniques used for interacting and those application software so that the evaluator does not become used for gathering subjects data obviously conflict ! For another user (even though a passive user) in the scene, instance, the use of thinking aloud cannot work well when which would certainly become a major bias in terms of MRS use (as they may often do) voice recognition as an « presence » [5]. input mechanism ! MRS can also be multi-users and require a large experimen- Limitations related to the subjects tation space or they can be used outdoors, which makes A first problem is that the application of MRS is not al- difficult or even impossible the use of current usability ways directed by application needs, but by the design of laboratories, which are located in limited spaces to facilitate new interaction paradigms, which makes difficult the speci- data extraction. fication of precise and accurate task and user requirements. MRS using video projection can very easily augment the It is therefore quite difficult to generalize because the users room temperature in the usability laboratories, which can profile is often ill-defined ; sometimes, it is even the tech- become an important, often underestimated bias in the ex- nical people that developed the MRS who are tested ! periments. Another difficulty is that, at the current stage, it still im- Difficulties in the set up of user testing possible yet to practically distinguish, as it is usually done First of all, the complexity of interactive situations with with “classical” HCI, the subjects in terms of experience MRS may necessitate more resources that usual user with MRS (e.g., novices vs. experts). That holds true as testing : several evaluators may be needed to be able to well for the skills of human factors specialists in charge of extract the interesting data (e.g., checking on performance, the evaluations ! on various modalities, on various media, etc.). Also, the Experimental design may also encounter difficulties in the use of video (several ones, simultaneously on various types number of subjects needed to cover the many potential of events) may be mandatory, as user behavioural sequences variables involved in the MRS. For instance, if one wants are more complex to extract and describe. « simply » to compare various combinations of interactive In addition to the complexity of software programming for devices associated to an MRS, such as 3 devices for each such environments [23], setting up experiments may also one of 3 user channels MRS (e.g., voice, gesture, eye- be more complex as it requires more technical specialists to gaze), one needs 27 different testing situations (i.e., 9 calibrate, tailor several types of technologies and various combinations x 3 channels) in order to have all subjects software supporting the complexity of MRS. participate to all possible combinations, which can become quite exhausting for the subjects and therefore potentially In case of devices or system breakdown, it may be more inconclusive for the experiment, due to fatigue, learning difficult and more time consuming to restart the devices or curve, etc. system, which may jeopardize the outcome and measures in the user testing. Therefore, it is even more important for And if one decides to set up the experimental design with MRS to be as stable as possible for the duration of the non-repetitive measures, and associating different subjects’ evaluation experiments. groups to each situation, for instance at a minimum of 5 subjects per cell, then the number of subjects required Another obstacle which is specific to MRS (and to all (135, i.e., 27 x 5) becomes fast very large and consequently novel applications) is that users may not know at all the quite complex to recruit and manage, not mentioning the way the situation works, which means extra time and caution in learning experimental requirements, task goals, additional difficulty of making sure that the subjects’ Also, this would facilitate the design and share of population is homogeneous. common recommendations databases. Regarding Other limitations can be identified for the subjects such as that, it would also be useful to share some the consequences of « cyber sickness ». Such limitations mechanisms allowing the usable organization, are both ethical (concerning the decision to run the user storage, and retrieval of such sets of testing) and practical (concerning the post-experimental recommendations, using dedicated software (e.g., arrangements). multiple guidelines bases management tools comparable to MetroWeb [15]). In terms of ethics, one can wonder if it is legitimate to expose subjects to physiological problems that can be • Under the assumption on the combinatory charac- serious, such as fainting fit or ataxia. In some countries, ter of the ergonomic knowledge to be applied to such as France, a law covers experimental situations of that MRS (real, « classic » & « web » computer-based, nature and requires the experimental setting to incorporate « VR », and « MR specific »), an effort should be the presence of certified physicians. made to distinguish the problems and to apply so- lutions from existing knowledge, for instance, on Concerning the management of subjects potential disorders physical parameters of the MRS situations (e.g., after the experiments, protocols are available, e.g., Stanney noise, lighting, temperature, workload, perception, [20], to make sure that a secure environment is provided to cognition, etc.), using knowledge related to diagnose and to avoid potential incidents or accidents. Of physiology, psychophysiology, anthropometrics, course, it is obvious that, in the event of « cyber etc., and of course software ergonomics. In that sickness », performance measures are completely biased, area, a number of results on GUIs, Web, etc., are unless it is the goal of the experiment to test such obviously applicable (e.g., information presenta- disorders! In any case, it is mandatory to have ways, in an tion in 2D graphics, navigation on the internet, experiment, to diagnose early manifestations of such etc.), at least partly, to MRS situations. sickness, in order to limit its physiological consequences. Also, issues related to social psychology DISCUSSION ET PERSPECTIVES and organizational ergonomics may apply, as such This paper has presented a number ergonomic issues related systems may concern organizational overall activi- to the evaluation of VE and extensions to MRS. ties, cooperative work, virtual organizations, etc. In terms of measurements, one may also want to Among these, it may be fruitful for the future usability of look at other measures than performance, such as MRS to investigate and to coordinate a number of research preferences, or even, levels of addiction (e.g., in avenues. games), etc. In terms of ergonomic knowledge, a common effort • Inspection methods are good candidates as should be pursued to ensure that MRS are, at all stages of complementary methods to user testing (for design, evaluated for usability; that such data is sound and instance, just before user testing). Ergonomic generic; and that, whenever possible, that knowledge is Criteria (op. cit.) have been proposed to VE as a made available as recommendations and shared for further basis for ergonomic inspection, but it has been design and evaluation. necessary not only to modify one criterion In terms of methods, several points can be made: (“Significance of codes and behaviour”) and to • For user testing, it would be useful to study the add two new ones (“Grouping-Distinguishing design of user testing protocols in order to allevi- items by behaviour” and “Physical workload”, but ate the various biases identified. also to adapt their definitions, justifications, to add more illustrative examples and counter- • It is particularly important to make sure that user examples, in order to take into account testing of MRS concerns well identified usability recommendations specific to VE. These criteria are questions rather than just testing the environment also been currently tested in terms of compared as it is. Caution should be exerted in order to efficiency (i.e., the number and quality of “clean-up” MRS from well-known problems usability problems diagnosed) towards expert before testing real ergonomic problems on new evaluation and user testing. usability interaction paradigms. These Ergonomic Criteria could be candidate for • In addition, organizing some kind of efficient further adaptation to the specifics of MRS, communication within the MRS research providing the empirical data on such environments community may help to cross-reference and become available on a large scale. One can already increase common knowledge on usability, e.g., consider that more criteria might be needed simply from the various user tests performed in different due to the fact that MRS concentrate many laboratories. This would certainly help in the ergonomics issues from the Reality to the generalization of usability results and lead to Virtuality; not mentioning the specific issue of commonly agreed generic recommendations. “continuity”. Using common testing platforms and testing In terms of MRS-specific questions, a large place should protocols would help to compare and share results. be considered for the issue of « continuity ». That issue, which is certainly a major usability property [24] needs to Teleoperators and Virtual Environments. Vol. 11, n°4, be further investigated in order to better define the concept, 2002, pp. 404-424. its components, and distinguish the concept of 6. Dubois, E., Gray, P.D., Nigay, L., ASUR++: a Design « continuity » from other dimensions related to guidance, Notation for Mobile Mixed Systems. Interacting With the structuring of information, the compatibility with the Computers, Special Issue on Mobile HCI, Paterno, F. tasks or common practice, the consistency of information (ed), 2003. presentation, of modalities, of procedures, etc. 7. Gabbard, J.L. Researching Usability Design and Another issue that seems to carry new types of problems is Evaluation Guidelines for Augmented Reality (AR) Sys- the design of help systems ; how should they be organized, tems. 2001. Available: which type of support should they use, etc. http://www.sv.vt.edu/classes/ESM4714/Student_Proj/cla In terms of models, it would be interesting to look at and ss00/gabbard/ to coordinate the characteristics of the models used for 8. Gabbard, J.L., & Hix, D. 1. A taxonomy of usability MRS (e.g., task models, interaction models, formal mod- characteristics in virtual environments. 1997. Avaible : els, architectures, etc.), and to identify, when useful, the http://csgrad.cs.vt.edu/~jgabbard/ve/taxonomy/ potential communication mechanisms between these mod- els ... this has been a recurrent issue for GUIs (also in 9. ISO/ TS 16982. Ergonomics of human-system interac- terms of system lifecycle development processes); it should tion – Usability methods supporting human-centred de- not be different for MRS. sign. 2000 Other issues are worth discussing as well, such as the need 10. Kalawsky, R., VRUSE – a computerised diagnostic for application and task models, for agreed upon extended tool : for usability evaluation of virtual/synthetic envi- task taxonomies, for instance, in order to be able to com- ronment systems. Applied Egonomics, Elsevier (ed) pare evaluation results, to generalize from one application n°30, 1999, pp. 11-25. domain to another one. 11. Karampelas P., Grammenos D., Mourouzis A. & Partly linked to that issue is the need for shareable classifi- Stephanadis C. Towards I-dove, an interactive support cations of MR elements, of common MR objects models tool for building and using virtual environments with (and a shared vocabulary), not only to facilitate the analysis guidelines. Proceedings of HCI, 22-27 june 2003, of MRS, but also both to compare results within the scien- Crete, Greece, vol. 3, pp. 1411-1415. tific community and to facilitate the design of strategies for 12. Kaur, K. Designing virtual environments for usability. inspections methods and heuristics. Ph. D. Thesis, City University, London, 1998. Discussions on these various items, together with other 13. Kaur, K. Designing Usable Virtual Environments. issues talked about during the Workshop on "Exploring the 1997. Demo available: design and engineering of Mixed Reality Systems", should http://www.soi.city.ac.uk/~dj524/demtool/frame.htm lead to some insight for future research and contribute to 14. Loeffler, C.E., and Anderson, T. The Virtual Reality increased knowledge on shared benchmarks. There is cer- Casebook. New York: Van Nostrand Reinhold, 1994. tainly lots of work at hand for this workshop, but also for the following ones! 15. MetroWeb: Available: http://www.isys.ucl.ac.be/bchi/research/metroweb.htm REFERENCES 16. Milgram, P., Takemura, H., Utsumi, A. and Kishino, 1. Bach, C., Scapin, D. L. Recommandations ergonomi- F. in SPIE 94, Vol. 2351, Telemanipulator and ques pour l’inspection d’environnements virtuels. (Rap- Telepresence Technologies, 1994, pp. 282. port de contrat). Projet EUREKA-COMEDIA, INRIA 17. Nigay, L., Dubois, E., Renevier, P., Pasqualetti, L. Rocquencourt, France, 2003. and Troccaz, J. Mixed Systems: Combining Physical 2. Bach C. & Scapin D. L. Adaptation of Ergonomic Cri- and Digital Worlds. Proceedings of HCI, 22-27 june teria to Human-Virtual Environments Interactions. in 2003, Crete, Greece, vol. 1, pp. 1203-1207. Interact'03. IOS Press. 2003. pp. 880-883. 18. Norman, D. A., Cognitive engineering. In D. Norman 3. Bastien, J. M. C., Scapin, D. L. Les méthodes ergo- and S. Draper (eds), User centered system design: New nomiques : de l’analyse à la conception et à perspectives on Human Computer Interaction (Hillsdale l’évaluation. Traité d’ergonomie, P. Falzon (Ed.), NJ: LEA), 1986, pp. 31-62. Masson. 2003 19. Porcher Nedel, L., Dal Sasso Freitas, C. M., Jacon 4. Bastien, J. M. C., et Scapin, D. L. Évaluation des Jacob, L. and Soares Pimentas M. Testing the Use of systèmes d'information et Critères Ergonomiques. In Egocentric Interactive Techniques in Immersive Virtual Systèmes d’information et Interactions homme-machine, Environments. In Proceedings of INTERACT’03 (Zu- C. Kolski (Ed.), Hermès. 2001. rich, September 2003), IOS Press, pp. 471-478. 5. Bowman, D.A., Gabbard, J.L., and Hix, D. A Survey 20. Stanney, K. M., Kennedy, R. S. and Kingdon, K. Vir- of Usability Evaluation in Virtual Environments: Clas- tual Environment Usage Protocols. in Handbook of Vir- sification and Comparaison of Methods. Presence: tual Environments, LEA Publishers, 2002, pp721-730. 21. Stanney, K.M., Mollaghasemi, M., and Reeves, L. 24. Trevisan, D., Vanderdonckt, J. and Macq, B. Continu- (2000), Development of MAUVE, the multi-criteria ity as a Usability Property. Proceedings of HCI, 22-27 assessment of usability for virtual environment system. june 2003, Crete, Greece, vol. 3, pp. 1268-1272. (Final Rep., Contract N°. N61339-99-C-0098). 25. Tromp, J. G., Steed, A. & Wilson, J. R. Systematic Orlando, FL: Naval Air Warfare Center Training Usability Evaluation and Design Issues for Collabora- Systems Division. 2000. tive Virtual Environments. Presence, Vol. 12, n°3, june 22. Sutcliffe, A. G., and Kaur, K. D. Evaluating the usabil- 2003. pp. 241-267. ity of virtual reality user interfaces. Behaviour & Infor- 26. Wilson, J. R., Eastgate, R. M., & D'Cruz, M. mation Technology, vol. 19, n°6. 2000. pp. 415-426. Structured Development of Virtual Environments. in 23. Träskbäck, M., Koskinen, T.and Nieminen, M. User- Handbook of Virtual Environments, LEA Publishers, Centred Evaluation Criteria for Mixed Reality 2002, pp. 353-378. Authoring Application. Proceedings of HCI, 22-27 june 2003, Crete, Greece, vol. 3, pp. 1263-1267.