Depth-sensing cameras (e.g. Kinect or Creative Gesture Camera) are exploited in many computer vision and augmented reality applications. They can also serve as a key component of natural user interaction via virtual keyboard, body pose or hand gestures. We integrated both these functions proposing the “Plane of Interaction” (POI) which is a solid flat surface placed on the reference plane (table top, floor). Calibrated camera/projector system automatically identifies the position of POI surface, projects the virtual menu buttons onto it and recognizes which button was “clicked” by hand (fingertip). The proposed POI was tested with the camera/projector prototyping setup. POI allows natural and quite robust interaction in this specific environment.
Human-computer interaction is one of the most progressive research areas of computer science because it simplifies the control of electronic devices and opens the door for new potential users. Natural User Interface (NUI) represents the latest stage trying to exploit gestures, voice commands, gaze tracking, brain computer interface based on the analysis of EEG signals etc. Probably the most popular NUI outcome are touch gestures applied within last few years into the touch tablets and smart phones.
Low-cost depth-sensing camera Microsoft Kinect [
Projection-based augmented reality [
In addition to the interaction with an object, sometimes is required also the interaction controlling the program itself (e.g. change the mode of operation). The use of a mouse or a keyboard would be complicated and unnatural in this situation. Therefore we created the specific object (plane of interaction) which is a natural part of augmented reality environment but its function is to control the program via the projected virtual buttons. 2
There are many application areas, in which a pair Kinect/projector can be used for augmented reality. Many researchers developed different augmented/virtual reality graphical user interfaces over the time. They differ in different hardware requirements (such as haptic pens, special virtual glasses, etc.) and features.
Szalavari in his dissertation [
Poupyrev et al. in [
The similar approach was used by Geiger et al. in [
Benko et al. in [
Our main motivation was to find a simple and easy way how to interact with the program during experiments with augmented reality environment using the depth sensor instead of the mouse or keyboard. Proposed “Plane of Interaction” (POI) is a part of scene and exploits the same depth-sensing camera as the basic program. From the technical point of view, POI is a planar surface which can be placed anywhere inside the field of view of the camera, it should be automatically identified by the camera and exploited as a virtual touch sensor.
In this section we will describe how to acquire, calibrate and extract the important information using the POI. In this chapter we will describe how to acquire, calibrate and extract the important information using the POI. 3.1
For prototyping purposes and for experiments with
projector-based augmented reality we constructed the
setup shown on Fig. 1. Projector P and 3D camera C share
the same mount attached to the massive stand. The
translation and possible rotation between them compensates
calibration software (see subsection 3.4 – Calibration). Plane
of Interaction is a solid planar plate, which can placed
anywhere inside the field of view of the camera.
We exploited projector BENQ MX613ST (aspect ratio 4:3,
through ratio cca. 1.0) and Microsoft Kinect [
The camera and the projector have different poses
(translated and rotated relative to each-other). They have
the different resolution and we have to take into account
also the recalculation of pixels to length units
(millimeters). All of these problems should be solved by the
following geometrical transformation
xP = Tx(xC, yC, zC)
yP = Ty(xC, yC, zC)
(1)
where Tx and Ty are linear transformations. It means that
each 3D point (xC, yC, zC) measured by the 3D camera is
transformed into the 2D point (xP, yP) displayed by the
projector. This problem is similar to “Bundle adjustment”
method in [
VP0 = AVC xP0
r11
yP0 = rr2311
zP0
1 0
To transform the point from the 3D coordinate system of
the projector (VP0 = (xP0, yP0, zp0)T) to the 2D coordinate
system of the projector (VP = (xP, yP)T), we can use the
following equations
xP =
c1xP0
zP0
yP =
c2yP0
zP0
where c1 and c2 are unknown coefficients related to the
ratio of the projector and scaling from millimeters to
pixels. Further expressing of equations 4 and modification
of matrix A leads to a system of 11 linear algebraic
equations (more details can be found in[
In practice, we need cca. 30 pairs of corresponding points to achieve reasonable precision. We have developed a special program which simplifies their acquisition. We project a grid to the surface and place a small disk into all grid intersections (see Fig. 2). The program on each intersection automatically acquires the depth image of the disk, finds its contour and fits the contour by the model of an ellipse. The center of such a disk determines coordinates (xC, yC) and the height of the disc above the reference plane is zC. Coordinates of the projector points (xP, yP) are the known grid intersections.
As the positions of the camera and the projector are stable, the calibration should be performed only once. Then the calibration data are saved as a part of the configuration file. The calibration coefficients are the same for the POI rectangle and for the rest of the depth image.
It should be noted, that to obtain optimal precision of the transformation, not all pairs of corresponding points should lie in one plane. If all pairs of corresponding points lie in one plane, then the error raising from inaccurate transformation between the 3D Kinect camera and the projector raises with the absolute distance of the measured 3D points from the plane in which pairs of corresponding points were acquired. In that case some serious inaccuracy can be seen on the image projected to the POI surface if the POI surface (plane) is not quite near to the plane in which the pair of corresponding points lie. The image of menu buttons is projected onto the surface of POI (see Fig. 3 Right). The goal is to identify the button rectangle touched by a fingertip.
It should be noted, that menu buttons should be projected onto the surface regardless of the orientation of the surface (see Fig. 3 Right). So if the surface is rotated, then we must ensure proper orientation of the menu buttons, too.
The first step in our approach is to detect POI surface. After that we try to find the smallest possible rectangle (minimal area), which will contain all points from POI surface. We also calculate its rotation.
Once this is known, we calculate the affine transform between in-memory image, which is not rotated and the projected image, which is rotated according to the calculated angle. Let us denote this affine matrix asM. To calculate M we need three 2D corresponding points between in-memory image and projected image. Those points are depicted on Fig. 3 as points A, B and C, where superscript S denotes source image and superscript D denotes destination image. Clearly, all six points (AS, BS, CS, AD, BD, CD) can be calculated automatically.
Now the depth sensor watches the hand above the ROI surface from the top. The reference plane (see Fig. 4) is labeled R, the POI is h units above the R. Two threshold levels T1 and T2 represent the range of sensitive distances from POI. 3D points having z-coordinate (depth) from the interval <h + T1, h + T2> create a binary image representing a rough approximation of the fingertip (in our experiments we used T1 = 5 and T2 = 20 millimeters). As stated above, the POI surface can be rotated, so to find out which button is “clicked”, we first transform the binary image of the fingertip approximation to the in-memory menu image coordinates using the inverse transform of matrix M. The result of this step is depicted on Fig. 5 (Middle). To eliminate noise (the outline of the hand and other fingers), we apply the OPEN morphology operation to the binary image. In our experiments we used 3 × 3 rectangle as a kernel with an anchor point in the center of the rectangle and OPEN morphology operation was applied in two iterations. The difference can be seen on Fig. 5 (Middle and Right). To finally find out which virtual button is “clicked”, we count non-zero pixels in each virtual button rectangle from the binary image and we select the one, which has the maximum number of white pixels in its rectangle. To eliminate recognition of some small region as a fingertip, we use another threshold value Tf and we “select” some virtual button only if the counted white pixels exceed the threshold value Tf .
In our experiments we used Tf = 150 pixels. In case we expect some smaller fingers (such as fingertips of children), the threshold value Tf should be set to something between 70 and 100 pixels.
It should be noted that the orientation of the hand is not critical to this approach, which is clearly an advantage. It is possible to place the hand from any direction to the POI surface and as we stated previously, the POI surface can be freely rotated on the reference plane.
Science museum (called also “Theme Park”, “Science center”, “Discovery Center”) is a modern type of museum where most of exhibits are interactive. One such museum was opened last year in our city where our group participated in the construction of several exhibits. Augmented reality sandbox is one of them, which exploits the depthsensing camera.
The construction of our interactive sandbox was
inspired by [
Conditions in the science museum are very specific: • Most of visitors are groups of children requiring simple, robust and self-explanatory control of the exhibits. • No extra hardware such as keyboard, mouse, cables etc. is acceptable. • The virtual menu should exploit the same camera as the exhibit itself. • Periodical innovation and upgrade of exhibits is the necessary condition to achieve repeating visits of the same people. • The museum is open daily for more than 100 of visitors per day so the time for the installation and testing of exhibits in real conditions is limited.
From the beginning, our sandbox operated only in a single mode as described above. However, we plan to implement new features and modes in the near future. POI concept described in this paper allows the easy way how to switch between various modes of operation by “clicking” the virtual buttons by hand. 5
We proposed a simple system of interaction with program exploiting depth-sensing camera called “Plane of Interaction”. POI is the integral part of virtual reality environment with specific function – recognize the position of finger (fingertip) and return the index of “clicked” rectangle representing the virtual button.
The proposed system works in real-time. In our experiments we achieved more than 30 frames per second, which is enough for real-time augmented reality applications. In fact, because the FPS of the Kinect depth-sensing camera is 30, more than 30 FPS is not needed.
We tested POI on the prototyping system with satisfactory results. Currently, we test POI in real conditions of the science museum with augmented reality sandbox exhibit. 6
The augmented reality sandbox installed in the science museum was continuously tested by real visitors. Although the anonymous survey declared mostly high and very high satisfaction of visitors, practical experience revealed some problems and showed new possible improvements. The calibrated Kinect/Projector/Sandbox system can be easily extended to create other augmented reality applications.
Anyway, technological aspects are only one side of the coin because they must be in balance with other exhibits in the science museum. Therefore any significant changes in exhibits must be consulted with other authors and designers.
The functionality of the POI can also be extended, so
it can be used in the similar way as default touch screens
and displays widely used today. We plan to extend the
functionality of the POI concept described in this paper
with the following features:
• The function of menu must be very simple – usually
just select the mode of operation.
• Multi-select. This feature enables users to use more
than one finger to perform a multi-select. It can be
utilized to selected more objects at once or, for
example, to “check” multiple virtual check-boxes.
• Select and move. This feature is very similar to the
widely used drag & drop feature and enables users to
select some object (or with the combination with the
previous features to select multiple objects at once)
and to move it somewhere else. The “select” part
of this feature will be exactly the same concept as
the one described in this paper, which is similar to
the mouse down event. To simulate mouse up event,
we just check for moving the fingertip (or fingertips)
away from the POI surface.
• Recognition of basic gestures. If it is possible to
simulate “click and move”, then we can use the points
obtained during the “move” phase as a gesture and try
to recognize it. Basic gestures (such as swipe
fingertip left/right/up/down) can be recognized quite
easily. To recognize more complicated gestures one can
use Dynamic Time Warping algorithm [
It should be noted, that not all improvements listed above are directly connected to the augmented reality sandbox described in section 4. Some of the improvements are planned to be used in the future in our other projects. 7
The plane of Interaction is the alternative to control the exhibits in science centers with the majority of children visitors. It is simple, self-explanatory and robust. For interaction it does not require any extra hardware only this included in the exhibit.
This work was supported by Slovak research grant agencies APVV (grant 0526-11).We thank to US Steel Košice as the main contributor of Steel Park science museum and “City of Košice” as co-founder.