International Symposium on Ubiquitous VR 2007 1 Hand Gesture Recognition for Table-Top Interaction System Sejong University, Game Interface Research Center Hyoung Il Park and Jong Weon Lee Abstract—Hand interface is well known for its intuition and convenience. User can use their real hands to interact with the system without any assistant device. In this paper, we propose a method which uses vision-based hand interface for table-top interaction system. To do this, first, we find skin region in camera image. But the result might include false-regions which are likely to skin-color, so we use region-based segmentation. To use PCA, a method of hand gesture recognition, we have to split detected region into hand region and arm region, then eliminate hand yaw angle component. We demonstrated the (a) (b) usefulness and possibilities of our method by developing 'omok' game and testing in table-top interaction system. Figure 1. (a) The original image, (b) detected skin region B. Detecting Hand Region from Skin Region Index Terms—hand recognition, skin detection, table-top interaction system, principal components analysis. After having detected the skin region, we have to separate hand region and arm region for hand gesture recognition. We I. INTRODUCTION base on the thickness variable of skin region. The point where the thickness changes most is the wrist point. To T able-top interaction system using hand interface has been an active research topic in recent years. Its main identify the most thickness variable, we do the following steps: advantage is to allow users to play game or work like the way they do in real life. Without any required devices for input 1) Finding the orientation vector, centroid point and interface, the system becomes very convenient. To obtain this rectangle boundary of the skin region (see Fig. 2 (a)). interface, a method of hand gesture recognition which is 2) Determining the two intersection points of the based on detected hand region is necessary. In relation to this orientation vector with the rectangle boundary using work, K. Oka, Y. Sato, H. Koike use fixed-size search positive orientation vector and negative orientation window to separate arm from whole image depending on the vector (see Fig. 2 (b)). distance from the camera to a user's hand [1]. R. Lockton, A 3) Determining the middle points of two lines connecting and W. Fitzgibbon use a wristed band for detecting wrist the centroid and each intersection point. By our position [2]. J. New, E. Hasanbelliu and M. Aguilar use a experimental research, the wrist position definitely lies method that depends on hand size [3]. We propose a hand between these two middle points (see Fig. 2 (c)). This gesture recognition method which is independent on hand can help us to save the detection time. size or the distance between camera and hand. Our approach 4) Determining interval points in the two lines connecting is to base on the thickness variable of skin region. the centroid with the two middle points (see Fig. 2 (d)). 5) Basing on the obtained interval points, calculating the thickness of skin region by calculating the distance II. HAND REGION DETECTION IN IMAGE between two intersection points of the cross vector and A. Skin region Detection the boundary of the skin region (see Fig. 2 (e)). 6) We could find out the most thickness variables by To extract a skin region from the whole image which finding the maximum difference between two comes from camera, we use YCrCb color-space. consecutive thicknesses. Transformation simplicity and explicit separation of After finding out the most thickness variable, we could draw luminance and chrominance components of YCrCb make it the wrist line and identify which is the hand region and attractive to skin modeling [4]. While detecting a skin which is the arm region by comparing the thicknesses on the region, the result could contain some false-regions which are two sides spitted by the wrist line. We know that hand region likely to skin region. We use labeling method to group pixels has bigger thicknesses than the arm region. So we could into regions. Then we assume the largest area is the skin eliminate the arm region which is unnecessary for hand region (see Fig. 1). gesture recognition (see Fig. 2(f)). International Symposium on Ubiquitous VR 2007 2 new images. IV. EXPERIMENT AND FUTURE WORK A. Overview of ‘Omok’ game The table-top interaction system consists of a display screen on its top and a camera above which points perpendicularly to the table’s screen (see Fig. 4 (a)). The (a) (b) application we developed to demonstrate the usefulness and possibilities of our proposed method is 'Omok' game. The game rule is that the first player who aligns 5 balls is the winner. Players just need to point their index finger to the expected position to make a playing step and use another gesture for undo action (see Fig. 4 (b)). (c) (d) (a) (b) Figure 4. (a)Table-top interaction system, (b) The developed application. (e) (f) Figure 2. (a) detected centroid point, rectangle boundary, (b) B. Future work intersection points, (c) Middle points, (d) Interval points, (e) calculated The proposed method still requires much improvement. thicknesses, (f) wrist line The first problem is detecting skin region. If the background C. Yaw Component Elimination of image includes skin-like regions which are bigger than After obtaining two end points of the wrist line, we can hand region, then skin detection will fail. And if users wear a define a wrist vector as a vector which goes from the left end long sleeve which covers the wrist point detection will fail point to the right end point. Then we calculate the angle too. We will overcome these problems by improving our between the basic vector (the unit vector of the Ox axis) and method. the wrist vector. Using this angle, we could rotate the hand region to compare with the trained images to identify which ACKNOWLEDGMENT trained image is the most similar to the user hand for PCA This work was sponsored and funded by Korea Game method. Development & Promotion Institute as a Korean government project (The Ministry of Culture and Tourism). REFERENCES [1] K. Oka, Y. Sato, H. Koike, "Real-Time FingerTip Tracking and Gesture Recognition" In Proceedings of the Automatic Face and Gesture Recongition, pp.429-434, 2002. [2] R. Lockton, A. W. Fitzgibbon. Real-time gesture recognition using deterministic boosting, Preceedings of British Machine Vision Conference, 2002. [3] J. New, E. Hasanbelliu, and M. Aguilar, “Facilitating user interaction (a) (b) with complex systems via hand gesture recognition”. Figure 3. (a) The image have a yaw angle, (b) Eliminated yaw angle image ACMSE'03.Knowledge Systems Laboratory, Jacksonville State. [4] V. Vezhnevets, V. Sazonov, A. Andreeva, “A survey on pixel-based skin III. HAND GESTURE RECOGNITION USING PCA color detection techniques”, GRAPHICON03, pp. 85-92., 2003 [5] F.Torre and M. J. Black, "Robust principal component analysis for PCA (Principal Components Analysis) method has been computer vision", Int. Conf. on Computer Vision, volume I, pp. 362?349, proven useful for solving problems such as face and object 2001. recognition, tracking, detection and background modeling [5]. Therefore we use this method. It is separated into two parts: an off-line and an on-line part. The off-line part is performed in order to find the transformation matrix and generate a classifier, all based on a set of training images. The on-line part uses the transformation matrix and the classifier computed off-line to transform and classify any