Gesture Determination for Hand Recognition Taher KHADHRAOUI Fouazi BENZARTI Abdulrahman Hamid AMIRI LSTS Laboratory LSTS Laboratory ALARIFI LSTS Laboratory National School of National School of KACST, CRI National School of Engineers of Tunis Engineers of aarifi@kacst.edu.sa Engineers of Tunis (ENIT), Tunisia Tunis (ENIT), Tunisia (ENIT), Tunisia khadhra.th@gmail.com benzartif@yahoo.fr hamidlamiri@yahoo.com Abstract—Gestures are the dynamic movements of hands within Previous work was done to classify a hand into one of a set of a certain time interval, which are of practical importance in signed words (sign language) [2]. Matthew Tang has proposed many areas, such as human–computer interaction, computer the hand gesture recognition using Microsoft’s Kinect [1]. vision, and computer graphics. Computer vision and pattern recognition techniques [3], This paper demonstrates the feasibility of a new method of hand- involving feature extraction, object detection, and geometry recognition based on parameters derived from the contour of the hand. The contour can be modelled by parameters, classification, have been successfully used for many gesture or features, that can capture more details of the shape of the recognition systems. Xia Liu and Kikuo Fujimura have hand than what is possible with the standard geometrical proposed the hand gesture recognition using depth data [7]. features used in hand-geometry recognition. The set of features There is another efficient technique which uses Fast Multi- considered in this paper consists of the spatial coordinates of Scale Analysis for the recognition of hand gestures as certain landmarks on the contour. suggested by Yikai Fang, Jian Cheng, Kongqiao Wang and Hanqing Lu [6]. Keywords-component; hand recognition; feature extraction; The principles and background of some of these popular tools gesture determination used in gesture recognition are discussed in [8]. I. INTRODUCTION Hand is a natural and powerful means of communication III. PROPOSED METHOD that conveys information very effectively. Hand gesture The proposed method is summarized in figure 1. It uses recognition is an important aspect in Human-Computer four main steps: interaction, and can be used in various applications, such as virtual reality and computer games. Research on hand gestures can be classified into three Input categories: sensor-glove-based analysis, vision-based analysis, Image and analysis of drawing gestures [1]. Pattern recognition consists of 1) identifying the pixels in the image that constitute the hand we’re interested in, 2) Hand Contour extracting features from those identified pixels in order to classify the hand into one of a set of predefined poses, and 3) recognizing the occurrence of specific pose sequences as gestures [1]. In this paper we focus our attention to vision Landmark based shape extraction of hand first part. After second part we Contour Points propose a real time hand gesture recognition system. Experiments have been conducted to validate the performance of the proposed system. As it is easy to develop other hand Pattern Matching (Compare gestures, the proposed system has good potential in many regions with gesture patterns) applications. II. RELATED WORK Gesture Many methods for hand gesture recognition using visual Determination analysis have been proposed for hand gesture recognition. Before we can perform gesture recognition, we need to Figure 1. Framework of gesture recognition identify roughly where the hand is located in our images. Typical image analysis and in particular gesture recognition finger-roots. Both of these features are used to increase the usually consists of several steps: robustness of the system. A. Hand Contour This step is also known as hand detection. It involves detecting and extracting hand region from background and segmentation of hand image. Hand model features (see Figure 1) are extracted from the segmented hand region represented by its boundary contour. Different features such as skin colour [4], shape, motion and Figure 3. Landmark Contour Points anatomical models of hand are used in different methods. Different methods for hand detection are summarized in this The ASM searching algorithm uses an iteration process to find paper. Some of them are. the best landmarks which can be summarized as follows: Colour: Different colour models can be used for hand  Initialise the shape parameters b to zero (the mean detection such as YCbCr, RGB, YUV, etc. shape) Shape: The characteristics of hand shape such as topological  Generate the shape model point using features could be used for hand detection. Learning detectors from pixel values: Hands can be found  Find the best landmark z by using the feature model from their appearance and structure such as Adaboost  Calculate the parameters b’ as algorithm. 3D model based detection: Using multiple 3D hand models multiple hand postures can be estimated.  Restrict parameter b’ to be within If |b’ - b| is less than a threshold value, then the matching process is completed; else b = b’, and return to step 2 C. Pattern Matching Using SIFT features for object matching is very popular, Figure 2. Hand Contour and seems to be a reliable choice for solving the problem of illumination and pose variability. The SIFT descriptor is highly distinctive and partially invariant to variations. In order to make the ASM shape model rotation invariant, the B. Landmarks Contour Points gradient orientations of the descriptor are always computed relative to the edge normal vector at the landmark point which The next important step is hand tracking and feature could be obtained by interpolation of neighboring landmarks. extraction. Tracking means finding frame to frame There are a two main advantages of the SIFT feature correspondence of the segmented hand image to understand descriptor [5]. The first advantage is that SIFT descriptors the hand movement. Following are some of the techniques for encode the internal gradient information of a patch around the hand tracking. landmark. The SIFT descriptors have a more discriminative 1) Template based tracking likelihood model which is distinctive enough to differentiate If images are acquired frequently enough hand can be between landmarks. tracked. It uses correlation based template matching. By The second advantage of the SIFT descriptors is that they are comparing and correlating hand in different pictures it could more stable to changes that occur due to changes of pose, that be tracked. can occur when dealing with hands. 2) Optimal estimation technique Hands are tracked from multiple cameras to obtain a 3D D. Gesture Determination hand image. In order to test any comparison metric devised it is 3) Tracking based on mean shift algorithm important to have a constant set of easily reproducible To characterize the object of interest it uses color gestures. It is also important to ensure that the gestures are not distribution and spatial gradient. Mean shift algorithm is used chosen to be as dissimilar as possible. Sign language gestures to track skin color area of human hand. are an excellent test, but sign language normally involves both Two types of features are there first one is global statistical hands with one hand regularly occluding the other. However, features such as centre of gravity and second one is contour there is an American one-handed sign language alphabet, based feature that is local feature that includes fingertips and which, with slight modification, can be used. IV. EXPERIMENTS RESULTS V. CONCLUSION In our preliminary experiments, we have obtained some We have proposed an efficient 2D Hand Detection for promising results for different gesture recognition. Gestures Recognition. The approach is definitely robust, Experimental results are given to demonstrate the viability of simple, and easy and fast to implement compared to other the proposed Hand Gesture Recognition method. algorithms. It provides a practical solution to the reconstruction problem. Future work includes applying the 3D model to hand animation and recognition, and using robust multi-view hand alignment to automate the reconstruction. Figure 4. Gesture Determination REFERENCES [1] M. Tang. “Hand Gesture Recognition Using Microsoft’s Kinect.” Paper written for CS229, March 16, 2011. [2] A. Zafrulla, H. Brashear, H. Hamilton, T. Starner. “A novel approach to American Sign Language (ASL) Phrase Verification using Reversed Signing.” Computer Vision and Pattern Recognition Workshops, 2010. [3] R. Lockton. “Hand Gesture Recognition Using Computer Vision.” http://research.microsoft.com/en-us/um/people/awf/bmvc02/project.pdf. [4] L. Bretzner, I. Laptev, T Lindeberg. “Hand Gesture Recognition using Multi-Scale Colour Features, Hierarchical Models and Partical Filtering.” Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition, 2002. [5] D. Zhou, D. Petrovska-Delacrétaz and B. Dorizzi. “Automatic Landmark Location with a Combined Active Shape Model”, 978-1-4244-5020- 6/09/$25.00 ©2009 IEEE. [6] Y. Fang, J. Cheng, K. Wang and H. Lu, “Hand Gesture Recognition Using Fast Multi-scale Analysis”, Proc. of the Fourth International Conference on Image and Graphics, pp 694-698, 2007. [7] X. Liu and K. Fujimura, “ Hand Gesture Recognition using Depth Data”, Proc. of the Sixth IEEE International conference on automatic Face and Gesture Recognition, pp. 529-534, 2004. [8] S. Mitra, T. Acharya “Gesture recognition: a survey”, IEEE Trans Syst Man Cybern Part C Appl Rev 37(3):311–324 (2007).