In this paper we introduce a new method to track deformable structures with the use of single retro-reflective markers. In addition, the robustness of the presented technique is evaluated under different situations that might be faced during a navigated open liver surgery. Among them one can highlight, deformation, presence of unexpected markers, occlusions, simultaneous use of single markers and instruments, etc. A workflow for incorporating the presented method in the clinical scenario is also proposed. The results show that our method is able to track structures with deformations even in more demanding scenarios where partial occlusions are present. The deformation per marker can be visualized by a color code.
and developed to assist open liver surgery was augmented with surface tracking capability. The same system was used to perform a set of experiments to evaluate different conditions that might be faced while tracking the liver in the OR. A surgical workflow considering the standard OR procedure constraints was also designed to demonstrate the practical use for our method.
Physical Markers
Surface Tracking Algorithm The NDI Vicra is a position sensor based on stereo cameras that is capable of returning the pose transformation of marker shields with pre-defined geometry, typically used to track the surgical instruments or rigid structures, plus the position of stray markers present in the scene. Although, this device can precisely identify and track a given marker shield geometry, it cannot ensure that a stray maker is returned in the same order or with the same label for two different frames. Consequently, without post-processing the stray markers' information returned by the camera there is no way to know the correspondence between markers of different frames. In addition, whenever two or more markers are aligned along an axis parallel to the cameras baseline, the position detected by stereo cameras is not unique. Thus positions of markers not really present in the scene, i.e. phantom markers, might be returned along with the positions of the real markers. Therefore, as a first step before using the stray markers information, we perform a phantom marker removal. Subsequently, an initial configuration with a label for each marker to be tracked is defined, allowing the actual tracking of the surface to begin. The surface tracking is realized in three steps: marker re-labeling to re-identify each marker label in the current frame, geometrical checking to ensure the markers were correctly identified, and pose estimation to calculate the surface rigid transformation and its deformation vectors. A detailed description of each step is given bellow: Phantom marker removal. It is performed using geometrical constraints imposed by the position sensor stereo construction. Every two markers (A and B) that have the same height (within a tolerance) when projected to a plane located in front of the position sensor and with normal equal to the position sensor direction are likely to produce at least one phantom marker (C). A marker C is considered as a phantom candidate when it is close to the plane formed by A, B, and the position sensor origin. For every A, B and C markers meeting these requirements, a voting function changes the probability of a marker being phantom or real. The probabilities of marker C being a phantom and markers A and B being real are incremented every time the lines collinear to the segments AC and BC pass near to either of the stereo cameras origins (camera location within the position sensor coordinate system) and the line collinear to the segment AB passes further away from both stereo camera origins. Finally, the markers that have a higher probability of being phantom than of being real are removed. Initial configuration definition. A set of visible markers are defined as the initial surface configuration and a label is given to each marker. This initial arrangement imposes the geometrical constraints between the marker set to be tracked. Marker re-labeling. To re-label the stray markers returned by the camera in the same way they were defined in the initial configuration, the distance (plus a tolerance) between each marker in the initial surface configuration is used every time all markers are visible. Otherwise, a stray marker in the current frame is labeled as the closest (within a maximum distance) visible marker in the last frame. In order to recover invisible markers, before looking for the closest stray marker in the current frame, the last rigid surface pose transformation is applied to the initial configuration of each marker that were invisible in the last frame. Geometrical checking. A final check ensures that the distance between identified visible markers matches (within a tolerance) the initial marker configuration distances, therefore avoiding a false label assignment. In case of false match or insufficient minimum number of visible markers, the surface is set to invisible. Pose estimation. The actual surface pose transformation is estimated by minimizing the square distance between each marker in the initial surface configuration to its corresponding re-labeled stray marker [6]. The final deformation vector is calculated by subtracting the current marker position from its initial configuration in the current rigid pose.
acquired and the liver is segmented to generate a 3D model containing liver surface, vessels and tumor (MeVis Distant
Once the liver is stable and the area to be resected is exposed, the single markers are attached to its surface with biocompatible cyanacrylate glue (tested on ex-vivo livers). A total of 6 markers are placed in order to keep a good tradeoff between trackability and resection working space. In order to avoid more damage to the liver, the markers are mostly placed in the part to be ressected. The system guides the surgeons during marker placement to prevent them from being placed in bad configuration (e.g. too close to each other, lying in a line, or symmetrically configured). Once the single markers are in place, the surgeon performs a registration of the real liver with its virtual 3D model. The registration is accomplished by defining 4 landmarks in the virtual model and digitizing them on the real liver with the trackable resection device (i.e. CUSA). This information is then used to identify the single markers initial configuration (assumed to be within the digitized landmarks), and to calculate the transformation between the 3D model and the single marker set. After this initial setup, the liver surface can be virtually co-displayed with the CUSA by means of a rigid transformation and a color code representing the deformation of each single marker (see Fig. 1b). Once the instrument guidance is no longer necessary, the reaming markers can be removed by a small resection.
up with silicon and broke off after the model was dry resulting in silicon model with the shape of a real liver (see Fig. 1a).
Tracking Experiments: For each tracked frame, the error per marker (current marker position minus marker initial configuration position in the current pose), the global error (average error per marker), the surface pose translation within the plate CS, and the rotation similarity between the plate CS and the surface pose were measured. The rotation similarity was used to identify flips of the model during tracking of the surface, and was calculated by adding the dot product of the trackable plate normal and the surface normal to the dot product of the trackable plate x axis and the surface x axis. This defines a value between [2,-2], where 2 means the initial condition in which the rotation of both coordinate systems are totally similar and -2 means they are totally inverted. A flip would be represented by an abrupt drop in the value. The tracking algorithm was parameterized to keep the whole surface visible whenever at least 3 of the markers were visible. While tracking the surface, the liver phantom was tested under different conditions including tracking with deformation (up to 30 mm per marker), total and partial occlusions, recovering from occlusions, occlusion with deformations, presence of unexpected markers, simultaneous tracking of the surface and other marker shields, and 10 different initial configurations. 3
74 degrees [-33°,41°]
>150 degrees [-78°,72°]
127 degrees [-60°,67°]
The results of the tracking experiment showed a quite stable algorithm without flips or jumps of the tracked model. Excluding the situations in which less than 3 markers were visible or the deformations were beyond the defined limits (30 mm), the model was always visible without flickering. For each of the 10 different initial marker configurations, the algorithm was able to identify the stray markers and track the surface. As it can be seen in Fig. 1c, the model can be tracked with very low error when no deformation is introduced. The error is only slightly higher at the end of the curve due to small deformations produced by gravity when rotating the model. Fig. 1d illustrates the tracking robustness against partial occlusions. It shows that the algorithm can track even when some of the markers (one first and then three) are occluded. It also shows that the algorithm recovers the occluded markers when they are no longer hidden.
jumps or flips were produced in the pose translation or in the rotation (Fig. 1f) respectively. This indicates a smooth visual tracking without flickering of the model. The deformation error is well represented in the model by a color code with green (light gray) being 0mm and red (dark gray) 10mm, see Fig. 1b. Although not shown in the figures, the combination of deformation and occlusion was also not a problem. The presence of other marker shields or unexpected markers did not disturb the tracking in general. However, for the particular case when one of the single markers was occluded and an unexpected marker got close enough to the hidden marker location, a jump was realized in the error causing a visual flicker.
50 100 time [camera frames] 150
50 100 time [camera frames]
150 b) d) 18 ]14 m [m10 r o rr 6 e 2 0 18 14 10 6 2 0 marker 0 marker 1 marker 2 marker 3 marker 4 marker 5 global error [mm] pose translation [mm] rotation similarity a) c) 18 ]14 m [m10 r o rr 6 e 2 0 18 ]14 m [m10 r o rr 6 e 2 0 marker 0 marker 1 marker 2 marker 3 marker 4 marker 5 marker 0 marker 1 marker 2 marker 3 marker 4 marker 5
50 100 time [camera frames] 150
50 100 time [camera frames] 150 e) f) Fig. 1: a) Liver phantom with single markers. b) Virtual liver with the virtual markers spheres. The color of the markers, green (light gray) to red (dark gray), represents the deformation from 0 to 10mm respectively. c) Error per marker while moving the phantom without deforming. d) Error per marker while occluding. e) Error per marker while deforming the phantom. f) Global error, pose translation and rotation similarity. 4
This paper presented an algorithm to track deformable structures, and evaluated it under different conditions. In addition, a workflow was proposed for surgical use of our surface tracking method. The results showed the robustness of our method under likely and unlikely situation that might be faced in the OR. The phantom marker removal technique proved to be very robust on eliminating phantoms generated when two real markers were aligned, nevertheless it could not remove all phantoms produced when three or more real markers were aligned. Our experiments also demonstrated that a initial configuration which is suitable for tracking can be found without much effort from the surgeon. In the tested cases, none of the 10 tried initial arrangements presented detection or tracking problems. It is worth mentioning that these experiments did not intend to evaluate how well the markers configuration represents the surface or its errors, but to evaluate the robustness of the tracking algorithm against different initial distances between the markers. The deformation tests showed that the system is able to track and display, in real time, surfaces with deformations up to 30mm, and this range was considered sufficient for the target scenario. The algorithm is also able to track during partial occlusions and to recover occluded single markers, which is a high requirement in a surgical scenario where several instruments and surgeons might temporarily get in the field of view of the camera. As a downside of this method, flickering occurred when an unexpected marker took the place of an occluded surface marker. However, this particular case is very unlikely in the OR and therefore is not considered a problem for the surgeon. Moreover, the error introduced by the unexpected marker can be visualized when displaying the deformation and is eliminated as soon as the surface marker is no longer occluded. Another drawback from this approach is that all markers have to be visible in order to recover from a total occlusion.
The encouraging results of this study motivated us to further investigate the applicability of our method in a real surgical scenario. In the future, this approach is intended to be used as basis for breathing gating. 5