Abstract and Concrete Interaction with Mixed Reality Systems The case of the mini screen, a new interaction device in Computer-Assisted Surgery Benoit Mansoux, Laurence Nigay Jocelyne Troccaz Laboratoire CLIPS-IMAG Laboratoire TIMC-IMAG BP 53 I. I. I. S. -Faculté de Médecine 38041 Grenoble cedex 9 38706 La Tronche cedex {mansoux, nigay}@imag.fr jocelyne.troccaz@imag.fr ABSTRACT INTRODUCTION In this paper we focus on the design of mixed reality (MR) In this paper, we focus on the design of Mixed Reality systems. We propose two design spaces that can be useful (MR) systems. We focus on MR systems that assist a user in a top-down (abstract to concrete) design method for MR in performing a task on a physical object (a class of MR systems. The first design space consists of an organized systems called “Augmented Reality” in [2]). One of our framework of abstract interaction situations for describing main application domains for such MR systems is mixed systems. Each situation is depicted by an ASUR Computer Assisted Surgery (CAS), in the context of a diagram and describes the exchange of information between multidisciplinary project that involves the HCI and the the entities involved in a mixed system. The situations are CAS research groups of the University of Grenoble. The abstract because they are independent of the interaction main objective of CAS systems is to help a surgeon in modalities (both interaction languages and devices). The defining and executing an optimal surgical strategy based abstract interaction situations are illustrated by several on a variety of multi-modal data inputs. MR systems play Computer-Assisted Surgery (CAS) systems. Such a a central role in the CAS domain because the key point of a framework is useful for the designer in order to CAS system is to "augment" the physical world of the systematically explore the set of possibilities at an early surgeon: the operating theater, the patient, the surgical stage of the interaction design, without being biased by a tools etc., by providing pre-operative information during particular technology. With the interaction situation the surgery. MR systems are now entering many surgical described, the designer can then focus on the modalities to specialties and such systems can take on the most varied be used: both passive and active modalities can be elected. forms. Although many CAS systems have been developed This design stage consists of concretizing the interaction and provide real clinical improvements, their design is ad- situation by selecting the modalities. For this stage of the hoc and principally driven by technologies. design, we propose a design space that characterizes the In this context, our research aims at providing elements possible usages of one particular innovative interaction useful for the design of usable MR systems by focusing on device for CAS systems: a mini screen. We illustrate the the interaction between the user and the MR system. We complementarity of our two design spaces by presenting present two design spaces that can be useful in a top-down two CAS systems that embed a mini screen for different design method for MR systems. The first design space, purposes in the interaction: one system is based on a presented in the second section of the paper, consists of an localized mini screen fixed on the surgical tool while the organized framework of abstract interaction situations for other involves the surgeon handling the mini screen on top describing MR systems. This first result is useful at an of the patient’s body. early stage of the design of MR systems: indeed it enables Keywords the designer to systematically explore the set of Mixed Reality, Computer Assisted Surgery, Design Space, possibilities without being biased by the available Interaction Device, Mini screen technologies. While this first design space focuses on abstract interaction (i.e., independent of the interaction technologies), our second design space, presented in the third section of the paper, characterizes the possible usages of one particular interaction device, a mini screen. Our two design spaces are therefore complementary and address different stages of a top-down design method of MR systems: abstract versus concrete interaction. Before presenting our two design spaces, we first clarify the two interaction design steps, i.e. the design of the abstract and concrete interaction. ABSTRACT AND CONCRETE INTERACTION Entities (R and A) Relationships We call interaction situation, an abstract description of the characteristics characteristics interaction involved in an MR system. Such a description - Perceptual/Action sense - Interaction language is independent of the interaction modalities. We define in and location [7] a modality as the coupling of a physical device with an interaction language. After describing the interaction Table 1: Characteristics of ASUR entities and situation, the following step in the design consists of relationships. concretizing the abstract situation by choosing the modalities: the description of the interaction is then Two levels of abstraction in describing concrete. For describing the abstract and concrete interaction using ASUR interaction, we use the ASUR notation [2][3]. In the In [3], we explained how we use the ASUR notation during following paragraph, we summarize the main characteristics the requirements definition phase for describing usage of the notation. We then describe how to use the ASUR scenarios and during the external specification phase for notation for describing the abstract and concrete interaction. describing the concrete designed interaction. Going one ASUR notation step further, we define here two levels of abstraction in ASUR [2][3] stands for "Adapter", "System", "User", "Real describing the interaction in an MR system, as part of a objects". In user-centered MR systems are described in top-down (abstract to concrete) method for designing the terms of entities (A, S, U, R) taking part in the interaction interaction. Interaction situations are described using and the relations between those entities. Between the user ASUR at the most abstract level. Nevertheless, for (U) and the computer system (S), the adapters bridge the analytical reasons, we describe the two levels of interaction gap between the physical world and the digital one. They description in reverse order, from the concrete one to the could be input adapters (Ain) (e.g., a mouse, a localization abstract one. mechanism) or output ones (Aout) (e.g., a video projector, (1) The most concrete description is the final stage of the audio speakers). Physicality is one key feature of MR external specification phase. Interaction is fully depicted by systems: real objects are involved in the task. Within the a set of ASUR entities and relations that are described by ASUR notation we distinguish physical objects that are the ASUR characteristics. The interaction modalities tools (Rtool) for performing the task, from the ones that are (devices and languages) are therefore chosen: we distinguish the objects of the task (Robject). two types of modalities in an MR system, the active and Three kinds of relationship between two ASUR entities are passive modalities. Active and passive modalities are identified: defined for the MR systems we are concerned with in this paper: the object/target of the main task is physical, for • Exchange of data is represented by an arrowed line example the patient for CAS systems. between two ASUR entities (AÆB). • For inputs, active modalities are used by the user to • Physical activity triggering an action: a double-line issue a command to the computer such as a pedal to arrow (AfiB) denotes the fact that when the entity A move a laparoscope in a CAS system. Passive modalities meets a given spatial constraint with respect to entity B, are used to capture relevant information for enhancing the data will be exchanged along another specified realization of the task, information that is not explicitly relationship (CÆD). expressed by the user to the computer ("perceptual user • Physical collocation is represented by a non-directed interfaces" [9]). For example, in our CASPER (Computer double line (A=B). This refers to a persistent physical ASsisted PERicardial puncture) system, presented in proximity of two entities. Figure 1, a system for computer assistance in pericardial Finally, the ASUR entities and relationships are described punctures, a passive modality is used for tracking the by a set of characteristics. Table 1 presents some of them. position of the puncture needle. For example the first characteristic induced by the use of a • For outputs, active modalities, conveying information real object (R) or an adapter (A) is the human sense from the computer to the user, imply that the user involved in perceiving data from such an entity or in explicitly switches attention from her/his current task performing actions using such entity. The most common focus to a new focus in order to perceive the provided used ones are the haptic, visual and auditory senses. A information. For example in our CASPER system, second characteristic is the location where the user has to visual guidance information during the puncture task is focus with the required sense, in order to displayed on a screen. While using CASPER (Figure 1), perceive/manipulate the real entity as well as to manipulate the surgeon must always shift between looking at the the adapter or perceive the data provided by it. In addition screen and looking at the patient and the needle (i.e., the one characteristic of a relation between two ASUR entities task environment). As opposed to active modalities, is the interaction language used to express data carried by passive output modalities convey information to the user the relation. If we refer to our definition of a modality [7] that is integrated in her/his task environment, for as the coupling of a physical device with an interaction example displaying anatomical information onto the language, the device is described by an ASUR entity while patient’s body during a surgery. For the case of passive the interaction language is a characteristic of the relation output modalities, the user does not have to switch from this entity (device) to another ASUR entity. attention from her/his current task focus in order to is represented by two mobile crosses, while one stationary perceive the provided information. cross represents the planned trajectory. A complete description of the concrete interaction in ASUR can be found in [2]. (2) A more abstract level of description of interaction consists of focusing on the exchange of information between the involved entities during interaction. By doing so, we describe what we call the interaction situation. Interaction modalities are not yet chosen but the elementary tasks are identified. The role of the adapters are therefore defined (for example, a localization mechanism, a data presenter) but the concrete adapters (physical devices) as well as the forms of the data conveyed along the relation are not yet defined. In addition the physical setting is not yet defined, physical relationships between entities are not decided. In conclusion, such level of description consists of an ASUR diagram: Fig. 1: CASPER in use during the intervention. • without characterization of the entities and relations. In Figure 2, we illustrate this level of interaction description, by presenting the ASUR diagram of the • with one kind of relation: exchange of data (AÆB). CASPER system. During the surgery, CASPER assists the Figure 3 illustrates this level of description using our surgeon (U) by providing in real time the position of the CASPER system. Figure 3 is therefore a more abstract puncture needle (Rtool) according to the planned trajectory. description of the interaction described in Figure 2. Two adapters (Ain, Aout) are necessary: The first one (Aout) is the screen for displaying guidance to the surgeon, and the second one (Ain) is dedicated to tracking the needle position S and orientation as well as the patient’s body (Robject). The localization of the needle is possible within a predefined volume near the patient’s body. Such a constraint is represented in Figure 2 by an ASUR relation fi (physical A1in A2in Aout U activity triggering an action). S Rtool Ain Aout U Robject Robject : Patient Rtool Robject : Patient Rtool : Puncture needle Rtool : Puncture needle U : Surgeon U : Surgeon Ain : Cameras+diodes A1in : Localizer Aout : Screen Robject A2in : Localizer S : Computer System Aout : Data presenter S : Computer System Fig. 2: ASUR diagram of the concrete interaction in CASPER. For a complete ASUR description, the Fig. 3: ASUR description of the abstract interaction in diagram is completed by the characteristics of each CASPER. As opposed to Figure 2, the interaction entity and relation (see [2]). modalities as well as the physical relationships are not yet defined at this stage of the design. The concrete interaction description of Figure 2 is not complete. The ASUR diagram is completed by the In a top-down (abstract to concrete) design method, the characteristics of the identified entities and relations. For designer first focuses on the interaction situation (i.e., example the interaction language (one of the characteristics) abstract description of the interaction) and will then select used to convey the guidance information on screen (Aout) the modalities for concretizing the interaction. Our first must be described: Using CASPER, in the same window design space identifying a set of interaction situations is on screen, the current position and orientation of the needle therefore useful at an early stage of the interaction design for reasoning on the interaction without being biased by the For these two situations that involve passive modalities, interaction technologies. Our second design space we suggest that the user and the object of the task are characterizes the possible usages of one particular physically together. In the case of telesurgery for example, innovative interaction device (output adapter) for CAS the surgeon (user) and the patient (object of the task) are systems: a mini screen. This second design space is distant. Such situations are described using ASUR by therefore useful for designing concrete interactions adding an ASUR chain that comprises the computer system involving a mini screen. (S) between: INTERACTION SITUATION DESIGN SPACE • the user (U) and the tool ([Rtool, R object]) for Class III- Our design space is made of interaction situations that are input, independent of the interaction modalities. A situation is • the user (U) and the object of the task (Robject) for Class dedicated to a particular task. For example, in Figure 3, the IV-input. diagram depicts the interaction situation for the task of The ASUR chain to be added is either: pericardial puncture while using CASPER. A situation describes both the abstract input and output interaction. (a) (AinÆSÆAout) Our framework is composed of input and output situations. (b) (RtoolÆAinÆSÆAout) Our approach for establishing the framework of interaction situations draws from our distinction of active and passive The two ASUR chains differ by the way the user interacts modalities. with the computer system (S). The two chains (a) and (b) respectively correspond to Class I-input and Class II-input. Input interaction situations We therefore obtain four classes: For inputs (user to computer), we identify four situations, two of them involve active modalities while the other two Class III-input-a (U and Robject distant): involve passive modalities. UÆ(AinÆSÆAout)Æ[Rtool, Robject]ÆAinÆS (1) The two situations, Class I-input and Class II-input, Class III-input-b (U and Robject distant): involve active modalities. In these situations, the user UÆ(RtoolÆAinÆSÆAout)Æ[Rtool, Robject]ÆAinÆS explicitly issues a command to the computer system. The user must switch attention from the task’s focus (Robject) to Class IV-input-a (U and Robject distant): a new focus in order to interact with the computer. As a UÆ(AinÆSÆAout)ÆRobjectÆAinÆS consequence, in the ASUR diagram that depicts these two situations, there is no R object involved. Without R object, the Class IV-input-b (U and Robject distant): two remaining possibilities are: UÆ(RtoolÆAinÆSÆAout)ÆRobjectÆAinÆS Class I-input: UÆAinÆS For example, the input interaction situation of the Class II-input: UÆRtoolÆAinÆS telesurgery system described in [5] belongs to Class III- input-b: The surgeon (U) remotely controls a slave robot The first situation (Class I-input) depicts a classical (Aout), that holds the surgical tools (AoutÆ[Rtool, R object]), interaction with a computer, for example using a mouse. by manipulating force-feedback arm-mounted tools The second situation (Class II-input) describes the case (UÆRtoolÆAin). where the user manipulates a physical object (Rtool) to interact with the computer via an adapter that captures the Output interaction situations manipulations. Examples of such input situations are the For outputs (computer to user), we identify four situations, physical icons that are physical handles to digital objects, two involving active modalities and two involving passive “coupling the bits with everyday physical objects and ones. This is the symmetric case of input situations. architectural surfaces” [6]. Class I-output and Class II-output correspond to situations (2) We identify two situations that involve passive involving active modalities. The user must switch attention modalities. The user is performing a task in the physical (explicit action of the user) from her/his current task focus world on an R object while the computer captures relevant (Robject) to a new focus in order to perceive the provided information for enhancing the realization of the task, thanks information carried by the active modalities. The ASUR to passive modalities. Two situations are possible whether diagrams of these two situations therefore do not comprise the user manipulates R object using a tool ([Rtool, R object]) or an entity Robject. directly manipulates Robject. Class I-output: SÆAoutÆU Class III-input: UÆ[Rtool, Robject]ÆAinÆS Class II-output: SÆAoutÆRtoolÆU Class IV-input: UÆRobjectÆAinÆS A Class I-output example is the CASPER output situation A Class III-input example is the CASPER input situation described in Figure 3: During the puncture task, the described in Figure 3: During the puncture task, the surgeon perceives guidance information displayed on a surgeon is handling the puncture needle (Rtool) that touches screen. An example of Class II-output situation would the patient body ([Rtool, R object]). Both the needle and the correspond to a CAS system that displays information on patient are localized by the system via adapters. the wall of the operating theater: Although a surface of the physical environment is used for displaying information (Rtool), it implies that the surgeon consciously switch possibilities at an early stage of the interaction design, attention from the environment of the task (the operating without being biased by a particular technology. With the field) to the wall in order to perceive the information. interaction situation described, the designer can then focus (2) As for inputs, two output situations involve passive on the modalities (device and language) that are passive or modalities. These situations describe the cases where the active according to the situation, as well as on the physical user is perceiving the information provided by the system setting (physical relations described in ASUR). From an within her/his task environment (Robject). The ASUR abstract interaction situation, several concrete interaction diagrams that describe these two situations therefore solutions can be designed. In the following paragraph, we involve an Robject. focus on concrete interaction involving a particular device: a mini screen. Class III-output: SÆAoutÆ[Rtool, Robject]ÆU CONCRETE INTERACTION INVOLVING A MINI Class IV-output: SÆAoutÆRobjectÆU SCREEN The output situation using the PADyC (Passive Arm with The transition from interaction situation to concrete Dynamic Constraints) system [8] belongs to Class-III- interaction is difficult because the set of possibilities in output. Indeed using PADyC, the surgeon is handling a terms of modalities (device and language) is huge. As a surgical tool that is linked to a passive arm (Aout). The first step for accompanying this transition, we propose a programmable arm enables us to provide haptic guidance design space that describes the possible modalities that information (touch feedback) to the surgeon while involve a mini screen. performing the surgery. Another output situation of this Small devices are increasingly being used in MR systems class that involves a mini screen will be described in the as in [10], and offer new interaction techniques, like the last section of the paper. Embodied User Interfaces defined in [4]. For CAS systems, A Class IV-output example is the situation using the a small screen is an innovative device. second version of CASPER [2] that involves a see-through Beyond standard technical features of an LCD screen like head-mounted display (HMD), instead of a screen as in the size, weight, resolution, frame rate, number of colors, first version of CASPER (Figure 1). Thanks to the HMD, luminance, viewing angle, and thickness, we propose a the surgeon directly perceives the guidance information design space based on more interaction-centered displayed on top of the patient. Another example is the characteristics, that are inspired from our situation design Image Overlay system [1] presented in Figure 5. The space. As shown in Figure 4, our framework is comprised guidance information is displayed onto a see-through of four dimensions, namely Input, Output, Manipulation surface located in between the surgeon and the patient’s and DOF. body. Such an interaction situation belongs to Class IV- output. Input The same reasoning as the one for inputs can be applied for The Input dimension is used to characterize how the screen studying the case where the user and the object of the task is used by the user to convey information to the computer are distant. The two chains to be added to Class III-output system. Five values are identified along this dimension: and Class IV-output, in between Robject and U are: none, tactile, pressure, acceleration, localization. The value none means that the screen is not used as part of an input (a) (AinÆSÆAout) modality. Tactile is the common input modality with a (b) (AinÆSÆAoutÆRtool) PDA (tactile screen). Moreover sensors can be embedded within the device. Thus pressure or acceleration can be For example: detected as in [4]. Finally the localization of the screen can Class IV-output-a: SÆAoutÆRobjectÆ(AinÆSÆAout) ÆU be known by the computer system thanks to a tracking One example of such a situation will be the following one: mechanism. a telesurgery system displays anatomical information on Output top of the patient’s body (SÆAoutÆRobject), while a camera The Output dimension is used to describe how the device (Ain) facing the patient’s body enables the distant surgeon conveys information to the user. We focus here on visual (U) to see on her/his screen (Aout) the image of the patient data but other non visual interaction languages can be used, enhanced by the anatomical information. including haptic feedback. Along this dimension, two Completeness of the situation design space values are identified showing whether the displayed data are For each input as well as output situation, we described all dependent on the screen's position or not. For instance, if the combination possibilities of ASUR entities, making the screen is tied to a tool handled by the surgeon, and it the design space complete. Nevertheless for each situation conveys guidance information, then the output data may be the described ASUR chain is the minimal one. While dependent on the screen's position: the displayed data concretizing the abstract situation, some ASUR entities change according to the screen's positions over the patient's may be inserted in the minimal chain. body. Other kind of data (e.g., blood pressure, body temperature) may be independent on the screen's position in The completeness of the framework makes it a useful tool that same case. for the designer to systematically explore the set of stationary translation rotation free (DOF) none none indirect tactile Screen position dependent data direct pressure Manipulation acceleration Captions Screen position independent data : Guidance system localization : Overlay system Input interaction Output interaction Fig. 4: Mini screen design space. Manipulation Guidance system The Manipulation dimension expresses the context of use An immediate usage of the mini screen consists of using it of the screen. Two values, direct and indirect, are identified as an output adapter to display guidance information. We along this dimension. The manipulation is direct if the user therefore obtain the same situation as in CASPER holds the device. The manipulation is indirect when the presented in Figure 3 (Class III-input and Class I-output). device is bound to another entity (e.g., an automatic arm), While concretizing the interaction, we decided that the mini which itself is manipulated by the user. screen will be tied directly to the tool (e.g., a drill) or a Degree of Freedom (DOF) tool guide if the tool itself is too fragile (e.g., a needle). The DOF dimension is used to describe the number of The ASUR description of the concrete interaction therefore different ways in which the screen can move. The screen includes: (Aout=R tool). Within the mini screen design space, can be stationary, move only in translation or in rotation, this design decision is described by (none, screen position or accept free motions. Those values are always based on a independent data, indirect, stationary) as shown in Figure referential. For instance, if a screen is tied to a surgical tool 4. Taking this design decision was driven by the need to (e.g., a drill), the screen is stationary in the tool's reduce the perceptual discontinuity as defined in [2] and referential, but freely mobile in a more global referential: experimentally observed in CASPER. Linking the screen Its position and orientation are therefore tool-dependent. and the tool may indeed reduce the perceptual That referential is often defined thanks to the context of use discontinuity. (Manipulation). As a CAS system to integrate our prototype, we have Two CAS systems involving a mini screen chosen puncture applications, either pericardial or renal. We present two usages of a mini screen that we designed Guidance information in these systems is limited and easy and are currently developing. They correspond to different to represent (tool direction, tool orientation, and tool interaction situations as well as characterizations within our depth). mini screen design space. Overlaying data system As on-going work, we are studying the interaction Another possible usage of a mini screen consists of not situations of other types of MR systems and not only the reducing it to an output adapter only, as in the previous ones that assist a user in performing a task on a physical system, but allowing it to be manipulated as a tool by the object as in CAS systems. surgeon. The mini screen can then be used as a magnifying During the workshop we would like to discuss the glass or "magical lens" on top of the patient’s body. Our completeness of the mini screen design space and apply our design is inspired from the Image Overlay system [1] interaction situation design space for describing the presented in Figure 5. As opposed to the interaction interaction situations of the presented systems. situation of the Image Overlay system where the surface is an output adapter (Aout), the mini screen in our system is ACKNOWLEDGMENTS both an Aout and a R tool. Indeed the surgeon is no longer This work is supported by the French Minister of Research manipulating surgical tools but the mini screen. The under contract MMM. Special thanks to C. Marmignon for interaction situation therefore belongs to Class III-output as the CASPER picture and to G. Serghiou for reviewing the opposed to Class IV-output for the Image Overlay system. paper. For inputs, the interaction situation corresponds to the REFERENCES same one as in CASPER. The mini screen as a tool (Rtool) 1. Blackwell, M., Nikou, C., DiGiola, A.M., and Kanade, is localized by an input adapter. T. An Image Overlay System for Medical Data While concretizing the interaction, the same localizer (Ain) Visualization. Proceedings of MICCAI'98, (1998), can be used for both the patient and the mini screen (as in LNCS 1496, Springer-Verlag, 232-240. CASPER). We fixed the value “translation on top of the 2. Dubois, E., Nigay, L., and Troccaz, J. Consistency in patient’s body” along the dimension DOF, as shown in Augmented Reality Systems. Proceedings of EHCI'01, Figure 4. The ASUR description of the concrete interaction (2001), IFIP WG2.7 (13.2) Conference, LNCS 2254, therefore includes: (Rtool fiR object). Springer-Verlag, 117-130. 3. Dubois, E., Gray, P., Nigay, L. ASUR++: a Design Notation for Mobile Mixed Systems. Interacting With Computers (2003), IWC, 15(4), Elsevier Science, 497- 520. 4. Fishkin, K.P., Moran, T.P. and Harrison, B.L. Embodied User Interfaces : Towards Invisible User Interfaces. Proceedings of EHCI’98, (1998), IFIP WG2.7 (13.2) Conference , Kluwer academic, 1-18. 5. Green, P. S., Jensen, J. F., Hill, J. W., Shah, A. Mobile Telepresence Surgery. Proceedings of MRCAS'95 (1995), International Symposium on Medical Robotics and Computer Assisted Surgery, Wiley, New-York, 98-103. 6. Ishii, H., Ullmer, B. Tangible Bits: Towards Seamless Interfaces between People, Bits and Atoms. Proceedings of CHI’97 (1997), ACM Press, 234-241. 7. Nigay, L., Coutaz, J. A Generic Platform for Addressing the Multimodal Challenge. Proceedings of Fig. 5 : Image Overlay system. CHI'95 (1995), ACM Press, 98-105. CONCLUSION 8. Troccaz, J., Delnondedieu, Y. Semi-Active Guiding In this paper we presented two design spaces for MR Systems in Surgery: A Two-DOF Prototype of the systems. Passive Arm with Dynamic Constraints (PADyC). Mechatronics (1996), 6(4), Elsevier Science, 399-421. • The interaction situation design space is useful for the designer in order to systematically explore the set of 9. Turk, M., Robertson, G., Eds. Perceptual user possibilities at an early stage of the interaction design, Interfaces. Communications of the ACM (2000), 43(3), without being biased by a particular technology. ACM Press, 32-70. • The mini screen design space helps the transition 10. Wagner, D., and Schmalstieg, D. First steps Towards between the abstract and concrete interaction by Handheld Augmented Reality. Proceedings of ISWC03 characterizing possible usages of one particular innovative (2003), IEEE International Symposium on Wearable interaction device for CAS systems: a mini screen. Computers, IEEE Computer Society.