=Paper=
{{Paper
|id=None
|storemode=property
|title=Surgical simulators integrating virtual and physical anatomies
|pdfUrl=https://ceur-ws.org/Vol-727/eics4med3.pdf
|volume=Vol-727
|dblpUrl=https://dblp.org/rec/conf/eics/CarboneCFFM11
}}
==Surgical simulators integrating virtual and physical anatomies==
https://ceur-ws.org/Vol-727/eics4med3.pdf
Surgical simulators integrating virtual and physical
anatomies
Marina Carbone, Sara Condino,
Vincenzo Ferrari Mauro Ferrari, Franco Mosca
Centro EndoCAS Centro EndoCAS
Università di Pisa Università di Pisa
+39 (0) 50 995689 +39 (0) 50 995689
name.surname@endocas.org name.surname@med.unipi.it
ABSTRACT
According to literature evidences, simulation is of utmost
importance for training purposes and for innovative Virtual Reality (VR) simulators virtually reproduce the
surgical strategies assessment. Nowadays the market offers surgical scenario and allows the user to interact with the
mainly two kind of simulators: rubber anatomies or virtual anatomy through different interfaces that could be
environments, each one with advantages and drawbacks. surgically realistic or not and that can or can’t embed some
In this paper we describe a strategy to develop patient- kind of haptic feedback. Even if during last decade many
specific simulators using a hybrid approach: silicone companies proposed virtual simulators, well described
models of abdominal organs sensorized with technical challenges must be still overcome to permit
electromagnetic coils, to acquire deformations, coupled varied training in a realistic computer generated
with a virtual scene. As demonstrated, this approach allows environment. These challenges include the development of
to mix benefits of a real interaction with the physical realistic surgical interfaces and environments, and most of
replicas with the possibility to enrich the virtual all the modelling of realistic interactions between objects
visualization with add-ons and features difficult to obtain in and rendering of the surgical field [17]. Excellent results
the real environment. are anyhow reached in the VR simulation of endoscopies
[7; 10; 18] or endovascular treatments [12; 20], where the
involved anatomies are simple tubular structures and there
Keywords are no complex tasks to simulate.
Patient specific simulator, hybrid simulation, segmentation,
silicone phantom, surgical training, abdominal surgery Simulation using physical objects usually involves plastic,
rubber and latex models arranged in boxes. These objects
are used to render different organs and pathologies and
INTRODUCTION allow to perform specific tasks such as cutting, suturing,
Recent developments in minimally invasive surgery, both grasping or clipping structures. The repetitive performance
traditional and robotic, have strongly promoted the of a single task allows the trainee to develop the hand-eye
development of simulation technologies in order to help coordination and the motor skills before entering the real-
surgeons in the acquisition of the required psychomotor patient setting. The actual interaction with simulated
skills. anatomy can be considered the principal advantage of
Medical simulators are rapidly evolving from primitive physical simulator that, on the other hand, are limited by
plastic mannequins to machines with embedded technology being restricted to single or few standard anatomical
and, recently, computer assistance capable of creating structures and by requiring to buy a new phantom (usually
realistic physiological and patient scenarios. Consequently expensive) for each destructive trial. Physical simulators
many types of simulators of varying complexity have been can also be employed as testing environment for the in-
developed and marketed. The existing trainers can be vitro assessment and validation of innovative surgical
essentially divided into two groups: virtual reality (VR) and technologies (like surgical instruments, robots or
physical simulators, while a third innovative approach to navigation) [4; 6; 8].
the simulation is now finding its space in market and In the last years to overcome limits of the two former
research: hybrid simulation[2; 14]. described approaches a new concept of simulation has been
developed: hybrid simulation. It combines synthetic models
with VR, deploying for example mixed-reality, to bridge
the gap between the synthetic mannequin and the computer.
Copyright © 2011 for the individual papers by the papers'
This avoids some of technical difficulties associated with
authors. Copying permitted only for private and academic
reproducing the feel of instruments and of human tissue in
purposes. This volume is published and copyrighted by the
a complete virtual environment, while still allowing access
editors of EICS4Med 2011.
to the advantages of computer simulation in particular for
13
the trainee performance evaluation, the possibility to enrich As first simulator we selected an healthy patient,
the scene with virtual elements and to give instructions for anonimized, dataset. The dataset has been segmented to
the surgical tasks execution [9]. This kind of simulators obtain organs frontiers. For this purpose we used a semi-
require sensors to quantitatively evaluate the trainee’s automatic tool previously developed in our lab: the
performance. EndoCAS Segmentation Pipeline[5] integrated in the open
This paper describes a fabrication strategy to build patient- source software ITK-SNAP 1.5 (www.itksnap.org) [21].
specific hybrid simulators mixing patient specific synthetic The whole segmentation procedure is based on the
anatomies with virtual reality features. The idea is to neighbourhood connected region growing algorithm that,
overcome the limit imposed by standard anatomy, starting appropriately parameterized for the specific anatomy and
from the elaboration of radiological images to develop a combined with the optimal segmentation sequence
simulator including realistic synthetic organs paired with proposed, allows optimal segmentation results. The results
electromagnetic position sensors and enriched with of a complete upper abdomen segmentation are shown in
consistent virtual model of the entire abdomen. Figure 1a.
MATERIALS and METHODS
The goal of the present work is to define a strategy to
manufacture patient specific silicone organs and pair it with
sensors in order to build a physical test bed enriched by a
virtual environment in the direction of an hybrid simulators
for abdominal surgery.
The simulator is to be used for surgical training, with the
chance of surgical performance evaluation, but also as
testing environment to assess innovative surgical
technologies like surgical robots or surgical navigators.
The development of the simulator starts from the
segmentation and surface extraction of anatomical
components of interest from real medical image data sets.
The obtained 3D virtual models are then employed on one
side to build the graphic interface, on the other side as
starting point to design the moulds for the silicone organs
models.
A commercial torso phantom (CLA® OGI Phantom) is
used to enfold synthetic organs models in a realistic Figure 1: 3D models of the upper abdomen and its segmentation in the
environment (14). Moreover supporting structures are segmentation software.
designed to guarantee the correct positioning of synthetic
models inside the commercial mannequin and replicate
space constraint and relationships between organs. Fabrication of synthetic organs
The class of silicone rubbers, which allows an easy
In this work NDI Aurora® electromagnetic (EM) tracking
reproduction of objects with complex shape, and an agarose
sensors have been used (Aurora® 5DOF Sensor, 0.5 mm x
hydrogel, which closely mimic the mechanical properties of
8 mm, 2 m) to sensorize organs[3; 16].
soft tissues [1], have been selected to fabricate the synthetic
organs.
Physic simulator fabrication More in particular the employed silicones are RTV-TIXO,
The fabrication steps is divided into two principal phases: and GSP 400 from Prochima® while an agarose powder
x Images acquisition and elaboration for the 3D from Sigma [19] (Type I-A Low EEO) is used for the
virtual models extraction hydrogel preparation. We set up two fabrication procedures
to reproduce different anatomical sensorized structures,
x Fabrication of the sensorized synthetic organs respectively sensorized hollow organs and sensorized solid
organ.
Image acquisition and elaboration Regarding hollow organs, for example stomach and
The virtual environment is obtained through the gallbladder, a process has been studied to embed sensors
segmentation of actual radiological datasets. In this first inside the organ wall, between two layers of silicone. In the
phase it lays the key to obtain non standard anatomies and following is detailed the procedure for fabricating a
to choose real anatomies to build up surgical theatre sensorized gastric model.
challenging for the trainee.
First the positions of 8 Aurora electromagnetic sensors
have been identified on the 3D virtual model in function of
14
the clinicians needs. Then it has been fabricated a mould the mould parts. Then, after silicone curing, Aurora sensors
replicating the gastric lumen, with holes in correspondence have been positioned in correspondence of the predisposed
of planned sensors positions. Figure 2 shows the gastric screws. A new layer of RTV TIXO silicone has been
mould with planned, in virtual Figure 2a, and actual screws applied to properly cover sensors. When the silicone cured,
positioning used for an exact sensors positioning Figure 2b. after removing screws, the mould has been closed, ensuring
In Figure 2c, a first layers of silicone RTV TIXO has been the proper alignment of the two mould parts and using
applied on the gastric model; after the silicone curing, additional silicone to attach the two silicone shells.
Aurora sensors have been positioned between each couple
of screws; the thin screws have been removed from the
rigid gastric model and a final layer of GSP 400 has been
applied, Figure2d.
Figure 3: a) Designed mould for the liver reproduction. In red dotted
circles. b) Selected positions for eight Aurora sensor; c) Prototyped
mould after silicone injection. d) Final silicone liver front (sx) and
back (dx).
Finally the prepared agarose gel has been injected into the
closed mould. The final result can be seen in Figure 3d.
In order to guarantee the correct positioning of synthetic
organ models inside the commercial mannequin it has been
Figure 2: Silicone stomach fabrication and sensorization: a) virtual
decided to fabricate a supporting structure, that fits
position for sensors, b) prototyped mould with screw to locate sensors’ perfectly inside the commercial mannequin, and allows to
position, c) first silicon layer and sensors deposition, d) final stomach insert synthetic organs models respecting their actual
model. anatomical location in the patient.
RTV TIXO has been chosen to fine reproduce gastric folds, At this aim, after positioning some radio opaque markers
the outer layer of the model instead has been fabricated on the mannequin, another CT scan has been executed, then
using GSP 400 that allows to obtain a more uniform and a registration between patient images and mannequin ones
smooth surface. has been performed and finally the segmentation obtained
from patient CT images has been loaded on the mannequin
The solid organs have instead been fabricated building greyscale images.
mould where to inject silicone or hydrogel. In the following
is detailed the procedure for fabricating a sensorized liver This allowed to segment the empty space between the
model. In particular the agarose powder has been mixed in mannequin abdominal cavity and the organs models and
water, heated until almost boiling, and then poured into the thus to extract the 3D model of a supporting structure for
designed mould. Since liver Young modulus varies around patient silicone organs that fits perfectly inside the
20 KPa [15] an agarose concentrations of 0.5 % has been commercial mannequin abdomen.
used for obtaining gel with a consistent elastic modulus [1]. Then the segmented model has been refined to optimize its
As showed in Figure 3a,b the mould is composed of two shape and allow an easy positioning inside the mannequin
joinable external shells that are the negative copy of the 3D and an easy insertion of the organs. Finally the designed
liver model. The positions for 8 Aurora sensors have been supporting structure has been fabricated using the 3D
identified on the 3D virtual model of the liver, Figure 3c printers.
shows the assembled mould. A set of abdominal walls has been built to complete the
The process of fabrication started with the application of a simulator. Such walls have been added in order to simulate
layer of silicone RTV TIXO in the internal surface of both
15
the pneumoperitoneum during robotic or traditional laparoscopic interventions.
The 3D model of the organs are visualized inside the
software.
It is important to underline that the virtual environment is
enriched respect to the real one by the possibility to add all
abdominal segmented structures, i.e. vessels and kidneys.
Color information are added to virtual model using vertex
coloring techniques in order to increase the realism of the
virtual scenario.
The physics mannequin is registered with the virtual
anatomy with a point based registration algorithm. This is
necessary to align the reference frame of the aurora
localizer, that read the sensors inside the mannequin, with
the CT reference frame in which the virtual anatomy is
referenced.
The transformation between CT and Aurora reference
frames is computed using the radiopaque artificial markers
Figure 4: Assembled mannequin covered (up), the phantom organs positioned on the commercial mannequin. Marker positions
inside the mannequin (down left) ant the virtual used to obtain are acquired with the Aurora digitizer. Then the registration
internal organs (down right)
matrix is calculated through a least square error algorithm.
The covers are fabricated in thermoformable plastic Starting the simulation the Aurora localizer starts reading
material modelled in the right shape. They are provided position information coming from sensors.
with some soft silicone windows in strategic positions to Each sensors position is registered to find its coordinates in
allow the insertion of the instruments access ports. the mesh reference frame; these coordinates are then
In Figure 4 it is showed the mannequin with 4 organs considered as “control points” to apply the deformation
inside: liver gallbladder stomach and pancreas. The organs function for reproducing the deformation actually imposed
are correctly arranged thanks to the supporting structure[3]. to the organs.
The class of Free Form Deformations methods are the most
Design and build of the graphic interface for the hybrid spread methods to modify the shape of geometrical objects
environment when described with vertices and faces [11]. The inquire on
A software interface that acquires signals coming from the deformation strategies to be followed is broad and literature
embedded sensors and emulates organs deformations on a is very rich about this field. Different decision has to be
virtual scenario (Figure 5) has been implemented to show taken for different organs according to its morphology.
the potentialities offered by hybrid simulation. At this moment we implemented deformation only for the
The software is written in c++ and deploys the openSG stomach. We implemented a point based deformation
opensource libraries to deal with openGL window and the method[13]. As said each sensors position is used as
Qt libraries to build the interface. control point for the mesh of the organ to be deformed.
When a sensors moves a Gaussian distribution function is
evaluated at each mesh vertex, and its displacement is
calculated with this distribution function. The 3D
coordinates of each vertex on the mesh are then coherently
updated, changing the shape of the 3D organ model, and
hence deforming it.
Below the mathematical description of the method is
showed.
ૡ
ࢊ
࢚࢞ ൌ ࢚ି
࢞ ൫࢙࢚ ࢞ െ ࢙ ࢞ ൯ ࢋି ࣌
ୀ
ૡ
ࢊ
࢚࢟ ൌ ࢚ି
࢟ ቀ࢙࢚ ࢟ െ ࢙ ࢟ ቁ ࢋି ࣌
ୀ
ૡ
ࢊ
࢚ࢠ ൌ ࢚ି
ࢠ ൫࢙࢚ ࢠ െ ࢙ ࢠ ൯ ࢋି ࣌
Figure 5: Graphic Interface and texturized virtual anatomy ୀ
rendering. ሬሬሬሬԦ െ ሬሬሬሬԦ
ࢊ ൌ ቚ ࢙ ቚ
16
Figure 6: Example of real time deformation of the virtual environment. The stomach is highly deformed so in virtual it is highlighted in red to
underline the entity of deformation
where
Ԧ௧ is the position of a mesh vertex at the instant t CONCLUSIONS
ݏ௧ is the position of the sensor n at the instant t In this work we describe how to develop surgical
simulators using a new paradigm.
n is the sensor number (in our case from 1Æ8)
In particular it is shown a strategy to build up a complete
dn is the Euclidean distance between the mesh vertex and hybrid simulator for surgical training.
the sensor n
Regarding the physical phantom the strategy easily allow to
σ is the standard deviation of the distribution. modularly build surgical scenarios. The mannequin was
showed to clinicians that confirmed the high degree of
The latter parameter describes the amplitude of the realism and the correct arrangement of organs inside the
gaussian bell and in this application it somehow reflects the abdomen.
material property of the organ describing how much wide Regarding the correspondence between real and virtual
the deformation is. The Gaussian distribution of the deformation real-time performances have been reached.
݀ʹ
݊
distances,ࢋെ ࣌ ǡ is evaluated for each mesh vertex and each At this moment only a simple deformation for the stomach
sensor “off line” when the mesh is loaded. So that, during is implemented but an integration of more complex
the simulation, the amount of computational load to be functions is planned. The aim is to reach integration of
done on the fly is reduced and the simulation is speeded up enough functions in order to simulate a complete
because it’s only needed to check precomputed values in a intervention.
local area only. For example next steps will regard the development of
Steering the σ parameter we obtained a simulator that virtual deformation for liver and gallbladder in order to
reproduce virtually the physical interaction with the simulate a complete colecistecthomy.
anatomy (Figure 6). This type of simulator overcomes the limits imposed by the
Moreover in order to add preliminary metric features to the use of standard anatomies and represents the first step for
developing more complex hybrid platforms, that links
simulator we inserted a visual effect that colours the
benefits coming from having physical scenario to interact
deformed part in function of the deformation entity.
with (mostly in terms of force feedback) with virtual
This is to virtually transmit if a deformation is too strongly elements that enrich the realism of the simulation and can
imposed and furthermore represent the first step to go offer to trainee a complete environment to learn surgery
towards bleeding anatomies and more complex virtual from a single task to more complex ones.
features.
17
While a complete evaluation as for this training purpose is 9. Kneebone, R. (2003). Simulation in surgical training:
currently underway, initial feedback from clinicians using educational issues and practical implications. Med
the system has been positive. The winning strategy to build Educ, 37(3), 267-277.
simulators not starting from standard anatomies but 10. Koch, A. D., Buzink, S. N., Heemskerk, J., Botden, S.
describing a wide variety of anomalies and pathological M., Veenendaal, R., Jakimowicz, J. J., et al. (2008).
scenarios is very encouraged from surgeons. Expert and construct validity of the Simbionix GI
Mentor II endoscopy simulator for colonoscopy. Surg
Endosc, 22(1), 158-162.
ACKNOWLEDGMENTS 11. LaGreca, R., Raffin, R., & Gesqui`ere, G. (2007).
The research leading to these results has received funding Punctual constraint resolution and deformation path on
from the European Community's Seventh Framework NURBS. Paper presented at the GraphiCon'2007,
Programme (FP7/2007-2013) under grant agreement num. Russia, Moscow, June 23-27, 2007.
224565 (ARAKNES Project) 12. Lee, J. T., Qiu, M., Teshome, M., Raghavan, S. S.,
REFERENCES Tedesco, M. M., & Dalman, R. L. (2009). The utility of
1. Bin Luo, R. Y., Peng Ying, Michael Awad, Michael endovascular simulation to improve technical
Choti, Russell Taylor. (2006). Elasticity and performance and stimulate continued interest of
Echogenicity Analysis of Agarose Phantoms Mimicking preclinical medical students in vascular surgery. J Surg
Liver Tumors. Paper presented at the Bioengineering Educ, 66(6), 367-373.
Conference, 2006. Proceedings of the IEEE 32nd 13. Liu, Q., & Prakash, E. C. (2006). Cyber surgery:
Annual Northeast Parameterized mesh for multi-modal surgery
2. Champion, H. R., & Gallagher, A. G. (2003). Surgical simulation. Journal of Visualization, 9(4), 373-380.
simulation - a 'good idea whose time has come'. British 14. McGaghie, W. C., Issenberg, S. B., Petrusa, E. R., &
Journal of Surgery, 90(7), 767-768. Scalese, R. J. (2010). A critical review of simulation-
3. Condino, S., Carbone, M., Ferrari, V., Ferrari, M., & based medical education research: 2003-2009. Medical
Mosca, F. (2010). Building patient specific synthetic Education, 44(1), 50-63.
abdominal anatomies. An innovative approach for 15. Nava, A., Mazza, E., Furrer, M., Villiger, P., &
surgical simulators from physical toward hybrid. Reinhart, W. H. (2008). In vivo mechanical
submitted to The International Journal of Medical characterization of human liver. Medical Image
Robotics and Computer Assisted Surgery. Analysis, 12(2), 203-216.
4. Enquobahrie, A., Gobbi, D., Turek, M., Cheng, P., 16. NDI. http://www.ndigital.com.
Yaniv, Z., Lindseth, F., et al. (2008). Designing 17. Seymour, N. E., & Rotnes, J. S. (2006). Challenges to
Tracking Software for Image-Guided Surgery the development of complex virtual reality surgical
Applications: IGSTK Experience. Int J Comput Assist simulations. Surg Endosc, 20(11), 1774-1777.
Radiol Surg, 3(5), 395-403. 18. Shirai, Y., Yoshida, T., Shiraishi, R., Okamoto, T.,
5. Ferrari, V., Cappelli, C., Megali, G., & Pietrabissa, A. Nakamura, H., Harada, T., et al. (2008). Prospective
(2008). An anatomy driven approach for generation of randomized study on the use of a computer-based
3D models from multi-phase CT images, IJCARS endoscopic simulator for training in
Volume 3, Supplement 1 / June. esophagogastroduodenoscopy. Journal of
6. Ferrari, V., Megali, G., Troia, E., Pietrabissa, A., & Gastroenterology and Hepatology, 23(7), 1046-1050.
Mosca, F. (2009). A 3-D Mixed-Reality System for 19. Sigma Chemical, I. http://www.sigmaaldrich.com.
Stereoscopic Visualization of Medical Dataset. Ieee 20. Tedesco, M. M., Pak, J. J., Harris, E. J., Jr., Krummel,
Transactions on Biomedical Engineering, 56(11), 2627- T. M., Dalman, R. L., & Lee, J. T. (2008). Simulation-
2633. based endovascular skills assessment: the future of
7. Grantcharov, T. P., Carstensen, L., & Schulze, S. (2005). credentialing? Journal of vascular surgery : official
Objective assessment of gastrointestinal endoscopy publication, the Society for Vascular Surgery [and]
skills using a virtual reality simulator. JSLS, 9(2), 130- International Society for Cardiovascular Surgery,
133. North American Chapter, 47(5), 1008-1001; discussion
8. K. Yang, Z. L. S., A.P. Kencana, V.A. Huynh, M. 1014.
Rasouli, S. J. Phee. (2010). Enhancement of Spatial 21. Yushkevich, P. A., Piven, J., Hazlett, H. C., Smith, R.
Orientation and Haptic Perception for Master-Slave G., Ho, S., Gee, J. C., et al. (2006). User-guided 3D
Robotic Natural Orifice Transluminal Endoscopic active contour segmentation of anatomical structures:
Surgery (NOTES). Robotics Automation and significantly improved efficiency and reliability.
Mechatronics (RAM), 2010 IEEE (28-30 June 2010), 15 Neuroimage, 31(3), 1116-1128.
- 18
18