Design of Perceptualization Applications in Medicine Jonas Forsslund, Eva-Lotta Sallnäs Pysander Karl-Johan Lundin Palmerius School of Comp. Science and Communication Norrköping Visualization and Interaction Studio Royal Institute of Technology Linköping University 100 44 Stockholm, Sweden 601 74 Norrköping, Sweden +46 8 790 60 00 +46 11 36 33 26 jofo02@kth.se, evalotta@kth.se karlu@itn.liu.se ABSTRACT appropriate decision” [23]. We suggest that applying a We are in this position paper presenting the experiences we User Centered Design method that also addresses aspects have from three medical application projects. A user of collaborative work is very well suited to fill these needs. centered design methodology have been applied in order to BACKGROUND ground the design in requirements gathered from field We will here give a brief description of User-centered studies of professional medical environments. Methods design and how we applied it, perceptualization theory and used have been interviews, user observations in the work the field of surgical simulators. context and cooperative evaluations of prototypes. With a particular focus on haptic (touch) feedback, we are Application of UCD in the medical domain exploring how novel medical applications can benefit from A method is qualified as a User Centered Design method as feedback to more senses than vision and how needs can be defined by ISO 13407 if the method includes four distinct revealed and transformed into effective design. design activities that form one development iteration [11]: Keywords 1. understand and specify the context of use User centered design, perceptualization, haptics, medical 2. specify the user and organizational requirements applications 3. produce design solutions INTRODUCTION 4. evaluate design against requirement In this position paper we will report on the application of The iterative nature of this process leads to gradual User Centered Design as a valuable method when improvements of the product, as well as allowing for early developing medical applications that aim at exploiting the changes of the direction of application development. benefits of perceptualization techniques, i.e. visualization extended to audio and haptic feedback. We propose that To understand and specify the context of use contextual grounding medical software design in user requirement inquiry [4] is used which involves field studies with data gathered from field studies and prototype-based user observations of the intended users workplace and on-site studies are an effective and efficient way of improving interviews to elicit user needs. Much emphasis is put on complex medical procedures of today. This will be understanding how the future users of the system carry out discussed based on the following three cases: their tasks today, and the field studies are rather designed Case 1: Oral surgery simulator to get a broad description of the workspace than of Case 2: Liver surgery planning obtaining detailed requirements. In an ethnographic study Case 3: Heart simulation of multi-disciplinary medical team meetings, Kane et al. The radiologist and researcher Ratib [23] argues that ”It is [13] show the importance of the radiologist and pathologist important to convey to device manufacturers that a wider being able to point to specific areas in the medical adoption of multi modality imaging techniques such as information (radiology images and pathology samples), as PET/CT in clinical routine will be properly enhanced only it is an essential part when presenting their statements to if the technology has an effect on the whole process of the other participants in an inter-disciplinary meeting. This patient management and not just on achieving higher kind of information is fundamental to consider in a design diagnostic accuracy” and that “an important step in the process in order to develop a system with good validity. In process of patient management is the collegial discussion our studies annotations are made of video recordings and between interpreting and referring physicians, surgeons, audio recordings of interviews. The results of the analysis and oncologists who review the images together to make an of the data gathered in field studies together with concurrent technical feasibility studies informs the requirements and design recommendations. These recommendations are in turn used as a basis for LEAVE BLANK THE LAST 2.5 cm (1”) OF THE LEFT implementation of lo-fi or hi-fi prototypes. The primary COLUMN ON THE FIRST PAGE FOR THE evaluation methods used are cooperative evaluation and COPYRIGHT NOTICE. observations of groups of collaborating users. Cooperative 42 evaluation is a method where the user is given a task to optimal integration. This means that the addition of touch is solve with the prototype, and the user and the evaluator is of increasing importance with decreasing quality of the allowed to discuss the interface during the session. The visual impression. While some applications may see only user is also encouraged to “think aloud” [20]. A rather new marginal improvement in user performance and approach to “think aloud” is evaluating users that understanding others may depend entirely on the successful collaboratively solve tasks which allows the researcher to integration of the haptic modality. Also, with the increasing observe the complex and often more realistic use of a size, detail and complexity of data presented in volume system [19]. The users discuss the task with each other visualization environments it is of increasing interest to while using the system and that result in a more natural augment the visual impression with information obtained kind of “think aloud” data that most probably does not add via other complementary sensory channels. Thus, the as much to the cognitive load of the users as traditional integration of haptics with volume visualization has “think aloud”. The analysis of the dialogue provides potential to significantly increase the speed and accuracy of information about problems in the design of the system. volumetric data exploration as well as improved This method also reveals how well the system supports interactivity. social interaction. It has also been shown that the design of the haptic Perceptualization feedback is of utmost importance. Choosing the wrong Perceptualization is an emerging research field that haptic representation of a feature in data may result in poor expands visualization to utilizing several senses such as understanding or even misinterpretation of the features at hearing and touch. Just as the purpose of visualization is hand [22]. This further stresses the importance of user mainly insight, and not necessarily realism, the purpose of centered design in the application development. perceptual-ization is insight and to afford exploration of Surgical simulation data in an effective and efficient way. The research Simulation of surgical procedures is a popular and presented here aims at exploring how advanced interaction important application of perceptualization technology such devices can support medical professionals analytical work as haptic feedback. The goal with a simulator is usually by providing feedback to more senses than vision. high realism that makes the person using it feel that it is a One example is making it easier to analyse the images from real patient or object she is handling. Meijden and Schijven ultrasound-based diagnostic medical imaging. Already [18] review a number of studies with VR-simulators and today colours in Doppler echocardiography represent results indicate that haptic feedback is especially useful information such as the direction and velocity of blood when it comes to achieving psychomotor skills. Haptic flow. Researchers have however shown that temporally feedback has been implemented in applications for surgery distributed events such as velocity and direction are easier simulation, bone drilling and virtual prototyping [3, 21]. to perceive kinesthetically (touch modality) than by vision Simulation-based training of laparascopic procedure has and even better by hearing the information [15]. In a proved to improve the performance of novice surgeons as perceptual perspective colour coding is might not an well as ensure they reach required skill level prior to optimal way of representing that kind of information. practice in real operations [1, 17]. The conventional The most spread use of haptic feedback systems in medical apprentice-based training of surgery is being challenged, applications are simulators for training surgical procedures since supervised practice in operating theaters are and robot assisted surgery [21]. Haptic sensing is defined expensive and occupies teachers for long time. In 2009 as “The use of touch in combination with motor behaviours United Kingdom implemented the European Working Time to identify objects” [2]. With a haptic feedback system it is Directive to limit surgeons working hours to 48 per week. possible to feel the shape, weight, texture, friction and This have led to complaints by the president of Royal stiffness of an object and for example collisions between College of Surgeons who argue that trainees do not gain objects. In collaborative virtual environments it is also enough experince anymore [10]. possible for each user to feel the other’s persons pulling In the field of medical simulation for training a distinction and pushing forces on shared objects or the other persons can be drawn between scenario mannequins and task cursor. trainers. The scenario mannequins often consist of a full Our sense of touch and kinaesthetics is capable of scale human mannequin with some simulated behavior and supplying large amounts of intuitive information about the it is often remotely controlled by a technician that can location, structure, stiffness and other material properties of suddenly invoke a heart stop or something similar. The objects. Providing feedback to more human senses in scenario is prepared in beforehand by the teacher and interfaces makes it possible for humans to access more of technician and the purpose is often to teach team skills in their brain capacity [15] or more popularly stated, it critical situations. The task trainers, on the other hand, increases their cognitive bandwidth. It has been shown [5] often simulate a specific procedure or are designed to that the integration of senses in multimodal environments is improve the student’s fine motor skills. Many of the done implicitly in the perceptual system by statistically laparoscopy trainers are of this type. According to Johnson 43 [12] this type of simulators builds “on an understanding of visual and virtual haptic model co-located and that allows medical practice as being made up of constellations of the user to feel the model where she sees it (figure 1). In discrete skills that can be learned separately and out of the image, a ray-cased based volume rendering of bone and context, and then put together in the examination or teeth are projected within an artificial 3D face model. The operating room to create a complete medical procedure”. purpose of the face model is to limit the view as is the case That implicit knowledge or situation-based experience is in real life. A physical head model (mannequin) is also important, is also argued by Giles when he states that the used where the haptic device is located to limit the physical simulators (task trainers) only cover a small part of the work space of the haptic device and to give the correct surgical curriculum, even though he acknowledge their hand support. With the haptic feedback device, the user can value and fit for purpose [10]. feel the shape of the teeth and resistance and vibrations while drilling with differences depending on material such CASE 1: ORAL SURGERY SIMULATOR The first case in this paper concerns a surgery simulator for as bone, enamel and dentin. A segmentation map of the teaching. The purpose of this project is to allow for dental volume also keeps track of where the user should and students to practice surgical extraction of wisdom teeth in a should not drill, and which parts can be removed with the risk-free virtual environment together with a supervising tools (drill and elevator). A state-machine progress the user teacher. through the procedure. These features are all designed based on data from field studies [7]. Results The work has resulted in an open architecture and open source software, as well as a particular simulation model for training of surgical extraction of wisdom teeth. Results from cooperative evaluation sessions showed important design considerations such as shading and coloring of the teeth, physical hand support and positioning of dental instruments. Shading is an important clue for seeing texture and depth in the rendering [6]. In an independent course intervention study of the oral surgery simulator conducted by Karolinska Institutet in Huddinge, 73% of the 60 course participants very much agreed on that this simulator training should be a permanent part of the course [24]. Preliminary results from the latest evaluation suggest that the best benefit from the simulator is the opportunity for the teacher and student to discuss the procedure freely while performing it repeatedly. The simulator is used as a mediating artefact that makes it easier for the teacher and Figure 1 Oral surgery simulator the students to contextualize theory, makes instructions Design process more concrete and makes it possible to teach tacit To form a mental model of what was possible to implement knowledge like what forces to apply, drilling angles and within reasonable time, a technical feasibility study was depth etc. conducted in advance. Previous work, such as a temporal CASE 2: LIVER SURGERY PLANNING bone surgery simulator [3] shows that haptic feedback Our second case is a visualization tool for supporting enabled virtual reality based simulations can well be used decision-making. In the liver surgery planning project, the for training of bone drilling tasks. To find out what is the objective is to explore multi modal technologies for most important aspects of the procedure that is new to the enhancing communication in medical multidisciplinary students, a contextual inquiry method was applied that team meetings concerning patient specific liver surgery involved observations, interviews and experimentation. planning. These studies revealed some of the tacit knowledge the surgeons depend on such as haptic perception of different Design process tooth and bone material. In these multidisciplinary team meetings, a patient case is introduced by a surgeon and discussed while a radiologist Application presents radiological diagnosis along with the patients' The oral surgery simulator consists of a physical and visual medical images as shown in figure 2 [9]. The images are model and provides haptic and audio feedback. The mostly contrast-enhanced computed tomography (CT) but monitor is aligned in a way such that the user looks with also magnetic resonance imaging (MRI) and ultrasound stereoscopic shutter glasses through a mirror that makes the 44 images are displayed. Although they occasionally display visual and haptic rendering of iso-surfaces of the same pre-computed (non-interactive) 3D volume renderings, volume (figure 3). With a haptic feedback device, the gray-scale 2D slices are the standard way of presenting. surgeon is able to feel the size and shape of a pre- segmented tumor, and distances to contrast-enhanced tissues. The two views are linked in such a way that the radiologist can scroll the stack of slices and point with the mouse cursor, which is shown simultaneous in the 3D view along with the position of the current slice. The position of the haptic device proxy is also displayed in the 3D view. Figure 2 Multi-disciplinary team meeting An understanding of the context of use has been obtained by observation analysis of video recordings of real meetings. The radiologists use their standard workstation for this demonstration, but utilize scrolling and pointing Figure 3 Collaborative surgical planning tool with the mouse cursor to indicate regions of interest. The discussion is often related to the location and dimension of The purpose of the haptic rendering in this case is not to one or several tumors in relation to blood vessels and other simulate the feeling of performing surgery, but rather as an organs. The surgical audience has to mentally reconstruct additional perceptual channel to explore the dataset. In the the region of interest in the 2D images to the 3D anatomy, prototype, a binary classification based on the attenuation which can be challenging. When the radiologist is absent, level in the images is used, and all voxels (after filtering) as when a surgeon is revisiting the case report before a above a threshold are perceived as hard material. surgery, the surgeon has to depend on their ability to To support several haptic feedback devices, a custom perceptually link the verbal descriptions in the reports even network haptic device proxy has been implemented, where though some markings in the images exist. For this reason, each haptic device is controlled by a dedicated computer have our research group proposed an interactive radiology sending position and receiving calculated force over a local report system where verbal statements can be directly network with a measured ~800 Hz update rate. linked to image annotations [14]. Evaluations of the system with expert users are currently Furthermore, our observations showed a need for improved performed and it is too early to report on results from visualization and for new ways for the audience to interact those. Suggested future work is to evaluate the concept in with the data to enhance communication between the natural setting, the medical multidisciplinary team radiologists, surgeons and other specialists. In one case, a meetings concerning patient specific liver surgery surgeon uttered "oh, is it that big?" referring to a tumor, planning, as well as improving the feasibility of multiple half-way through the discussion, although the radiologist haptic devices. had shown images and attempt to state the size prior to the discussion. Gestures were also made in the air by the CASE 3: HEART SIMULATION surgeons in the audience that indicates a need to point at In the third project the added value is investigated of specific parts of the patient images, to get the radiologist to including multi modal feedback in a simulation of a human show more of some specific data or zoom in on details or to heart that is based on ultra sound data with the purpose to show where the surgeon planned to cut in order to take out support experts that perform physiological examinations. In the tumor. These pointing behaviors were however very this recently started project the requirements are currently imprecise as the surgeons did not have access to any other gathered for the design of a future simulation-based means of interaction than gesturing with their hands. diagnostic tool for clinical physicians. Design process Application To allow for a richer visualization while leaving the Based on a finite element method simulation of blood radiologist with a familiar interface we developed a pressure and velocity inside a geometrical model of left software solution that combined visualization of 2D slices ventricle of a human heart, derived from a arbitrary patient, of a Computed Tomography volume, with a stereographic the goal is to be able to predict heart dysfunction of a specific patient given certain attributes as input to the 45 simulation. Both the parameter input and the resulting disbelief, this is a requirement for users to accept the simulation should be perceptualized to allow a trained simulated artefact as real enough for the purpose, in this physician to draw quantitative as well as qualitative case learning a surgical procedure. conclusions regarding a patient’s health . The transition of focus from the simulator working as a It is too early to report on results from evaluations of the stand-alone surgical trainer to primarily being an artefact system. However, findings regarding the context of the that mediates and enhances the discussion between the clinical physician’s workplace show conspicuous student and the teacher suggests the need for additional differences between dedicated machines and workstations. alternative design solutions. Future research will include The ultrasound machines used (GE Vivid 7 Dimension) functionality that specifically support the dialogue between provides sonification of ultrasound data that might assist in the teacher and student such as non-realistic rendering of detecting abnormal heart rhythms. The machine also the tooth and jawbone. Future evaluations will focus on provides an interface with physical knobs, sliders and how knowledge gained from training on such simulation buttons. However, for post-processing and analysis, the would transfer to skills performed in the operating room. physicians use a conventional keyboard-mouse When the focus is on supporting communication the whole workstation, even if the software comes from the same design process shifts towards designing for mediation of vendor. dialogue and then designing specific functionality for The interface differences motivate further investigation, mediated communication becomes important. Making that especially regarding useful sonification and tangible focus shift is rather novel in the area of simulation and interface properties. Interpretation of ultrasound data is a perceptualization but is inevitable when applying a user complex practice that involves both quantitative centered approach today as most systems are used by estimations and qualitative conclusion generation. We are groups of people as well as by individuals. currently exploring the technical feasibility of CONCLUSIONS perceptualizing both visually and haptically the geometrical We have demonstrated the potential of applying user deformation of the simulation as well as simulated blood centered design methods to design and engineering of flow. medical applications that strive to exploit the benefits of DISCUSSION perceptualization, with a particular focus on haptic While all three of the presented cases in this paper are feedback. By following the proposed method, engineers related to surgery and uses similar technology, only the oral could arguably design innovative systems that have larger surgery simulator is focused on training. In the heart impact on real work. In a recent issue of Communications project, the application data comes from a simulation but of the ACM, the future of haptic feedback is discussed with the application itself should not be considered a simulation the underline “After more than 20 years of research and since the data is pre-computed. It is important to development, are haptic interfaces finally getting ready to distinguish between a data-altering simulation and a pre- enter the computing mainstream?” [25]. Several haptic computed simulation based perceptualization where the scholars are interviewed and the speculation is that haptics goal is to perceive the data that has been generated as a might take the same path as touch screens, that has existed result from a simulation. for a long time but suddenly - with the iPhone - took off as The simulator we have developed for training surgical a mainstream technology. Professor Colgate states the extraction of wisdom teeth is primarily a skill trainer, but in requirements for such to happen: “The technology has to our studies of how the teacher supervising students uses the be sufficiently mature and robust, there has to be an active simulator results indicate that the learning it supports marketplace that creates competition and drives down depend a lot on the communication between the student costs, and it has to meet a real need.” [25]. We hope this and the teacher that is mediated by the simulator. This is in paper has contributed with examples of how a user accordance with the theories of Johnson regarding what she centered design process can support system designers in calls reconstitution to “create medical practice out of finding the required real needs of the professional users in simulator practice” [12]. That is, during the simulation the the medical domain. participants, a student and a teacher, help each other to ACKNOWLEDGMENTS mentally create a patient out of the simulated patient and to We thank colleagues at the Royal Institute of Technology make a surgeon of the student including the embodied for their contributions to the work presented in this paper. knowledge that is needed (the way the student is positioned We also thank the radiologists and surgeons at the etc). Johnson mentions a situation with a laparoscopy Karolinska Institute that have given us important insight simulator where the teacher explains the relative position of into their work practice as well as feedback on our system the instruments by pointing to his own leg. The same job design. This work has been funded by the KTH Innovation, could be achieved by using a mannequin leg but that would Swedish Governmental Agency for Innovation Systems be more expensive and the point is that the teacher’s and the School of Computer Science and Communication. gestures were sufficient. Also known as suspension of 46 REFERENCES 14. Kapourkatsidou, C. (2010). Visualization of Findings 1. Aggarwal, R., Grantcharov, T. P., Eriksen, J. R., in Radiologists’ Statements based on 2D and 3D Blirup, D., Kristiansen, V. B., Funch-Jensen, P., and Images. Master’s Thesis in Computer Science (30 Darzi, A. (2006). An evidence-based virtual reality ECTS credits) at the Interactive Systems Engineering training program for novice laparoscopic surgeons. Master Programme Royal Institute of Technology. Annals of surgery, 244(2):310-314. 15. Lederman, S. J. & Klatzky, R. L. (2009). Haptic 2. Appelle, S. (1991). Haptic perception of form: Activity perception: A tutorial. Attention, Perception, & and stimulus attributes. In M. Heller, & W. Schiff, Psychophysics. 71(7), 1439-1459. (Eds.), The psychology of touch. (pp. 169-188). New 16. Lee, J-H., and Spence, C. (2008). Assessing the Jersey: Lawrence Erlbaum Associates, Inc. benefits of multimodal feedback on dual-task 3. Agus, M. (2004). Haptic and Visual Simulation of performance under demanding conditions. Bone Dissection. PhD thesis, Dept. of Mechanical Proceedings of the 22nd British HCI Group Annual Engineering, University of Cagliari, year 2004 Conference on HCI 2008. Liverpool, United Kingdom, 4. Beyer, H., and Holtzblatt, K. (1998). Contextual pp. 185-192. Design: Defining Customer-Centered Systems. San 17. Felländer-Tsai, L. and Wredmark, T. (2004). Image- Francisco: Morgan Kaufmann. ISBN 1-55860-411-1 guided surgical simulation - a proven improvement. 5. Ernst, M. (2006). A Bayesian view on multimodal cue Acta Orthop Scand, 75(5):515–515. integration. Perception of the human body from the 18. Meijden, O.A.J., and Schijven, M.P. (2009). The value inside out Volume: 131, Issue: Chapter 6, Publisher: of haptic feedback in conventional and robot-assisted Oxford University Press, New York, Pages: 105-131 minimal invasive surgery and virtual reality training: a 6. Flodin, M. (2009). The importance of shading in current review. Surgical Endoscopy, 23(6),1180-1190. volume rendered visualization in a multimodal 19. Moll, J., and Sallnäs, E-L. (2009). Communicative simulator for operative extraction of wisdom teeth. Functions of Haptic Feedback. In Proceedings of Master's Thesis in Interactive Media Technology, Haptic and Audio Interaction Design, Dresden Royal Institute of Technology Germany September 10-11. HAID 2009. LNCS 5763, 7. Forsslund, J. (2008). Simulator for surgical extraction Springer, 2009, 1-10. of wisdom teeth. Master’s Thesis in Computer Science, 20. Monk, A., Wright, P., Haber, J., and Davenport, L. Royal Institute of Technology (1993). Improving your human-computer interface: a 8. Forsslund J., Sallnäs, E-L, and Palmerius, K-J. (2009). practical technique. A volume in the BCS Practitioner A user-centered designed FOSS implementation of Series, Prenice-Hall, ISBN 0-13-010034-X. bone surgery simulations In proceedings of World 21. Morris D, Sewell C, Barbagli F, Blevins N, Girod S, Haptics Conference, 2009. and Salisbury K. (2006). Visuohaptic Simulation of 9. Frykholm, O., and Groth, K. (2010). Shared Bone Surgery for Training and Evaluation. IEEE visualization of patient information. Technical report Transactions on Computer Graphics and Applications, NADA, April 2010. (Note: Position paper, Workshop November 2006, pp. 48-57. on Interactive Systems in Healthcare (WISH), in 22. Lundin Palmerius, K-J and Forsell, C. (2009). The conjunction with CHI2010) impact of feedback design in haptic volume 10. Giles, J. A. (2010). Surgical training and the european visualization. in Proceedings of World Haptics working time directive: The role of informal Conference, 2009 pp. 154-159. workplace learning. International Journal of Surgery, 23. Ratib, O. (2004). PET/CT Image Navigation and 8(3):179-180. Comm-unication The Journal of Nuclear Medicine, 11. ISO. (1999). Human-centred design processes for Vol. 45, No.1 (Supl), January 2004 interactive systems (ISO 13407:1999). International 24. Rosén, A., Fors, U., Zary, N. Sejersen, R., Sallnäs Organization for Standardization. Pysander, E-L., and Lund B. (2009). Can students 12. Johnson, E. (2007). Surgical Simulators and Simulated learning improve in oral surgery with different Surgeons: Reconstituting Medical Practice and simulator techniques? Poster at Karolinska Institutets Practitioners in Simulations. Social Studies of Science utbild-ningskongress, March 2009. 2007 37:585 25. Wright, A. (2011). The touchy subject of haptics. 13. Kane, B. & Saturnino, L. (2006). Multidisciplinary Commun. ACM , 54 , 20-2 Medical Team Meetings: An Analysis of Collaborative Working with Special Attention to Timing and Teleconferencing. In Journal of CSCW, 15, 501-535. 47